Stb-tester Manual

Automated Testing for Set-Top Boxes and OTT devices.

Contents:

Getting Started

• The Stb-tester Portal
  • Sign in
  • Select your device
  • Live video
    • If the video doesn’t play
  • Manual control
  • Run a test
  • View test results
  • The test-pack git repository
  • Next steps
• Hardware Setup
  • Physical connections
  • HDMI CEC control
  • View your device in the Portal
  • Troubleshooting
• Device Configuration
  • Apple TV
    • Remote control
    • Key names
    • Power on / power off
    • Network control
    • Apple TV settings
  • Amazon Fire TV
    • Remote control
    • Key names
    • Waking from sleep
    • Inter-press delay
    • Capturing logs
  • NVidia Shield
    • Remote control
    • Key names
    • Capturing logs
  • PlayStation
    • Remote control
  • Roku
    • Roku HTTP remote control
      • Key names
      • Reliability
    • Python API
    • Capturing logs
• IDE Setup
  • VS Code
    • Install Python 3.10
    • Install Git
    • Install VS Code
    • Install VS Code extensions
    • Clone the test-pack repository
    • Configure VS Code and venv
    • Test that it’s all working
    • Troubleshooting
  • PyCharm
    • IntelliSense for stbt APIs
    • Sync the code from your IDE to the Stb-tester Portal
    • Run a test from your IDE
    • Troubleshooting
• Writing your first test
• Object Repository
  • What is a Page Object?
  • What is a Frame Object?
  • Creating new Page Objects
    • Save a screenshot
    • Create a class that derives from stbt.FrameObject
    • Define a property called is_visible
    • Define properties to extract information from the frame
    • View your Page Object in the Stb-tester Portal
• Continuous Integration
  • Set up your source-code repository
  • Jenkins
    • Set up your Jenkins server
    • Set up your Jenkins job
  • Bamboo
    • Set up your Bamboo server
    • Set up your Bamboo job
  • CI for your test-pack
• Troubleshooting
  • My Stb-tester Node is offline
  • Live video doesn’t play in the web portal
  • The set-top box intermittently ignores infrared keypresses
  • My IDE can’t find the stbt APIs
• Frequently Asked Questions
  • Installing third-party packages
  • Passing parameters to test scripts
    • Static configuration
    • Run-time parameters
    • The current node_id
    • Passwords and other secrets
  • OCR
    • Installing language packs
• How to ask for support
  • Issues with image processing APIs
  • Other issues

Reference

• Python API Overview
  • Example
  • Controlling the system-under-test
    • Remote control
    • Network-based protocols
  • Verifying the system-under-test’s behaviour
    • Searching for an image
    • Searching for text using OCR (optical character recognition)
    • Searching for motion
    • Miscellaneous video APIs
    • Audio APIs
  • Custom image processing
  • Logging
  • Metrics
  • Utilities
  • Exceptions
• Python API Reference
  • adb()
  • AdbDevice
    • AdbDevice.adb()
    • AdbDevice.devices()
    • AdbDevice.get_frame()
    • AdbDevice.press()
    • AdbDevice.swipe()
    • AdbDevice.tap()
    • AdbDevice.logcat()
  • AdbError
  • AppleTV
    • AppleTV.launch_app()
    • AppleTV.list_apps()
    • AppleTV.set_text()
  • apply_ocr_corrections()
  • as_precondition()
  • audio_chunks()
  • AudioChunk
  • BGRDiff
  • Color
  • color_diff()
  • ConfigurationError
  • ConfirmMethod
    • ConfirmMethod.NONE
    • ConfirmMethod.ABSDIFF
    • ConfirmMethod.NORMED_ABSDIFF
  • crop()
  • detect_motion()
  • detect_pages()
  • Differ
  • Direction
    • Direction.HORIZONTAL
    • Direction.VERTICAL
  • draw_text()
  • find_file()
  • find_regions_by_color()
  • find_selection_from_background()
  • Frame
  • FrameObject
    • FrameObject._fields
    • FrameObject.__bool__()
    • FrameObject.__eq__()
    • FrameObject.__hash__()
    • FrameObject.__init__()
    • FrameObject.__repr__()
    • FrameObject.refresh()
  • frames()
  • get_config()
  • get_frame()
  • get_rms_volume()
  • GrayscaleDiff
  • Grid
    • Grid.Cell
    • Grid.get()
  • Image
  • is_screen_black()
  • Keyboard
    • Keyboard.Key
    • Keyboard.add_key()
    • Keyboard.find_key()
    • Keyboard.find_keys()
    • Keyboard.add_transition()
    • Keyboard.add_edgelist()
    • Keyboard.add_grid()
    • Keyboard.enter_text()
    • Keyboard.navigate_to()
  • Keypress
    • Keypress.key
    • Keypress.start_time
    • Keypress.end_time
    • Keypress.frame_before
  • last_keypress()
  • Learning
    • Learning.NONE
    • Learning.TEMPORARY
    • Learning.PERSISTENT
  • load_image()
  • load_mask()
  • Mask
    • Mask.from_alpha_channel()
    • Mask.to_array()
  • match()
  • match_all()
  • match_text()
  • MatchMethod
    • MatchMethod.SQDIFF
    • MatchMethod.SQDIFF_NORMED
    • MatchMethod.CCORR_NORMED
    • MatchMethod.CCOEFF_NORMED
  • MatchParameters
  • MatchResult
  • MatchTimeout
  • MotionResult
  • MotionTimeout
  • MultiPress
    • MultiPress.enter_text()
  • navigate_1d()
  • navigate_grid()
  • NavigationFailed
  • NavigationState
    • NavigationState.frame
    • NavigationState.last_page
    • NavigationState.last_key
  • ocr()
  • ocr_eq()
  • OcrEngine
    • OcrEngine.TESSERACT
    • OcrEngine.LSTM
    • OcrEngine.TESSERACT_AND_LSTM
    • OcrEngine.DEFAULT
  • OcrMode
    • OcrMode.ORIENTATION_AND_SCRIPT_DETECTION_ONLY
    • OcrMode.PAGE_SEGMENTATION_WITH_OSD
    • OcrMode.PAGE_SEGMENTATION_WITHOUT_OSD_OR_OCR
    • OcrMode.PAGE_SEGMENTATION_WITHOUT_OSD
    • OcrMode.SINGLE_COLUMN_OF_TEXT_OF_VARIABLE_SIZES
    • OcrMode.SINGLE_UNIFORM_BLOCK_OF_VERTICALLY_ALIGNED_TEXT
    • OcrMode.SINGLE_UNIFORM_BLOCK_OF_TEXT
    • OcrMode.SINGLE_LINE
    • OcrMode.SINGLE_WORD
    • OcrMode.SINGLE_WORD_IN_A_CIRCLE
    • OcrMode.SINGLE_CHARACTER
    • OcrMode.SPARSE_TEXT
    • OcrMode.SPARSE_TEXT_WITH_OSD
    • OcrMode.RAW_LINE
  • pdu()
  • PDU
    • PDU.power_on()
    • PDU.power_off()
    • PDU.set()
    • PDU.get()
  • Position
  • PreconditionError
  • press()
  • press_and_wait()
  • pressing()
  • press_until_match()
  • Counter
    • Counter.inc()
  • Gauge
    • Gauge.set()
  • Histogram
    • Histogram.log()
  • Region
    • Region.x
    • Region.y
    • Region.right
    • Region.bottom
    • Region.width
    • Region.height
    • Region.center
    • Region.from_extents()
    • Region.bounding_box()
    • Region.intersect()
    • Region.to_slice()
    • Region.contains()
    • Region.translate()
    • Region.extend()
    • Region.replace()
    • Region.dilate()
    • Region.erode()
    • Region.above()
    • Region.below()
    • Region.right_of()
    • Region.left_of()
  • RmsVolumeResult
    • RmsVolumeResult.dBov()
  • Roku
    • Roku.from_config()
    • Roku.save_logs()
    • Roku.query_apps()
    • Roku.launch_app()
  • segment()
  • set_global_ocr_corrections()
  • Size
  • stop_job()
  • TextMatchResult
  • Transition
  • TransitionStatus
    • TransitionStatus.START_TIMEOUT
    • TransitionStatus.STABLE_TIMEOUT
    • TransitionStatus.COMPLETE
  • UITestFailure
  • VolumeChangeDirection
    • VolumeChangeDirection.LOUDER
    • VolumeChangeDirection.QUIETER
  • VolumeChangeTimeout
  • wait_for_content_start()
  • wait_for_match()
  • wait_for_motion()
  • wait_for_transition_to_end()
  • wait_for_volume_change()
  • wait_until()
• Python API Release Notes
  • v34
  • v33
  • v32
• HTTP REST API v2
  • Authentication
  • Rate limits
  • Screenshots
    • Grab screenshot
    • Grab screenshot thumbnail
  • Running tests
    • List testcases
    • Run tests
    • Inspect a test job
    • Inspect a node’s current test job
    • Wait for a job to finish
    • Wait for a node’s current job to finish
    • Stop a job in progress
    • Stop a node’s current job
  • Test results
    • Get list of test results
    • Get list of test results (CSV)
    • Get list of test results (XML)
    • Get detailed information about a test run
    • Get log output from test run
    • Get screenshot saved at end of test run
    • List artifacts produced during a test run
    • Get an artifact produced during a test run
  • Test farm node information
    • Get status of all nodes
    • Get a node’s current status
  • Reservations
    • Reserve nodes for exclusive use
    • Extend a reservation
    • End a reservation
    • Interrupt a reservation
• stbt_rig Command-Line Tool
  • Installation
  • Authentication
  • Global options
  • stbt_rig snapshot
  • stbt_rig run
    • Interactive mode
    • Jenkins integration
    • Bamboo integration
  • stbt_rig download
  • stbt_rig screenshot
  • stbt_rig setup
• Configuration Reference
  • Configuration files
  • .stbt.conf
  • Node-specific configuration files
  • Reading configuration values in your test scripts
  • Storing passwords and other secrets
  • Automatic configuration keys
  • Customising the test-run environment
    • Caching
    • How the setup script is run
    • Reproducible builds
    • Example
• Remote Control Configuration
  • Infrared control
  • HDMI CEC control
  • USB HID control
  • Roku HTTP control
  • Default remote control
  • Remote control image
  • Key mappings
• User Interface Reference
  • Test farm
  • Live video formats
  • Admin page
    • Location data
  • Running tests
    • Tags
    • Job category
    • Soak-testing options
      • Run once
      • Run forever (until stopped manually)
      • Run forever in random order (until stopped manually)
  • Manual control
    • Record video
  • Test results
    • Filter
    • REST API
    • Further reading
  • Save screenshot to Object Repository
• User Access Control
  • Granting access to the portal
    • Read, Write, or Admin access
    • Manual control
  • Roles
• Power Distribution Units
  • Configuration files
    • Common configuration keys
    • SNMP configuration keys
  • Network configuration
  • Controlling the PDU from Python scripts
• Lint checks in pull requests
  • About pylint
  • Lint checks in your IDE
  • Lint checks in pull requests
  • Disabling Pylint warnings
• Interactive log viewer
• Controlling Apple TV with pyatv
  • Usage
  • Pairing
  • Configuration
    • IP address
    • Apple TV settings
  • Installation
• Capturing logs from the device-under-test
  • Android log capture
  • Roku log capture

User Guides

• Investigating intermittent bugs
  • Case study: BT multicast IPTV
  • How many test-runs do you need?
    • Chi-squared test
    • Confidence intervals
    • Validity of these statistical tests
  • Video: Checking test results
  • Case study: Using preconditions to discard false positives
  • Test failures vs errors
    • Test failures
    • Test errors
    • Preconditions
• How to structure your test scripts for re-use across multiple platforms
  • Directory structure
  • Configuration files
  • Launching your app
  • Page Objects
  • Page Objects directory structure
  • Reference images
  • Object Repository screenshots
  • Minor differences in app appearance and behaviour
  • Major differences in app behaviour
  • What are those __init__.py files?
  • Rationale
  • Questions?
• Pytest
  • Assertion introspection
  • Fixtures
  • Older Stb-tester versions
  • Notes for pytest experts
• Regions and Masks
  • Regions
  • Masks
• Finding GUI elements by color
• Using segmentation to find GUI elements
• Testing on-screen keyboards
  • Demo
  • Modelling the keyboard
  • Modes
  • Identifying the currently selected key
    • Tip: Debugging your Page Object
  • Navigating the keyboard
    • Key.name versus Key.text
  • Common mistake: Using an outdated page instance
  • Simple keyboards (no modes)
  • Shift modes
• Selenium for second-screen login flows
  • Recipe
  • Demo
  • Details
  • References and further reading