“No-reference” means that you don’t have to provide a “reference” or “original” video to compare the content against — the model works with any live content. “Perceptual” means that the model tries to give a rating that would match a human’s perception of the video quality.

UVQ returns a “Mean Opinion Score” (MOS), which is a number between 1 and 5:

The Stb-tester Node has a decent GPU, so our UVQ implementation runs in near real-time, allowing you to sample about 1 second of video out of every 5 seconds.

To learn more:

• UVQ: Measuring YouTube’s Perceptual Video Quality on the Google Research blog has high-level information about the algorithm.
• stbt.UVQ API documentation in the Stb-tester manual.

We have supported ADB from your test scripts for many years, but only over the network. The new features are:

• ADB over USB: This is more reliable than Network ADB, and doesn’t require any configuration (with Network ADB you need to configure the device-under-test’s IP address). Simply use a USB cable to connect the Android device to the Stb-tester Node.

• ADB for remote control from the interactive web interface of the Stb-tester Portal. (Previously ADB was only accessible from your Python test scripts.)

We still recommend using infrared, HDMI CEC, or Bluetooth for remote control when possible, as these are more representative of real-world usage. Using ADB for other purposes (such as collecting logs with logcat) is, of course, a good idea, and now it’s more convenient if your Android device supports USB.

For configuration instructions see Remote Control Configuration: ADB control.

Find us at stand 14.B57 in the Future Tech hall. Drop by to see Stb-tester’s latest innovations in automated testing: how our private, on-device AI models can be adopted incrementally in your existing test scripts; our new Bluetooth remote-control capabilities; and more.

Whether you’re a broadcaster, streaming operator, middleware developer or device manufacturer, we’d love to show you what’s new and hear about your testing challenges. Contact us at ibc@stb-tester.com or Book a meeting.

About Stb-tester

Stb-tester is a best-in-class test automation tool for testing TV & streaming apps on real devices such as Android TV, Apple TV, Fire TV, PlayStation, Roku, XBox, and traditional Set-Top Boxes. Founded in 2013, with uninterrupted year-on-year growth, Stb-tester is used by leading broadcasters, ISPs, and streaming operators worldwide.

Local training & inference

Stb-tester’s AI model trains & runs directly on your Stb-tester Node hardware — it isn’t sending each frame of video to the cloud for processing. This allows privacy and low latency (the model runs at about 20 frames per second).

There is a new training interface in the Stb-tester Portal, that allows very quickly collecting the necessary training screenshots, and verifying the behaviour of the model interactively.

How to use it in your Python test scripts

The stbt.FrameObject base class for your Page Objects has a default implementation of the “is_visible” property. Before, each of your Page Objects had to implement this property, perhaps using stbt.match like this:

 class Settings(stbt.FrameObject):
-    @property
-    def is_visible(self):
-        return bool(stbt.match("Settings.png", frame=self._frame))

Now, if you remove the is_visible property from your Page Object, it will use the default implementation, which asks the AI model if the current page is visible:

class Settings(stbt.FrameObject):
    pass

Class names are mapped to the AI labels as “device or application name” slash “Python class name”, for example “appletv/Settings”. The application name comes from the directory layout of your Python code, like this:

tests/{app_name}/pages/{module_name}/{module_name}.py

If you are using Stb-tester’s standard directory layout, this mapping works automatically. (We also support a few other directory layouts used by existing customers with large code-bases, to ease adoption of new Object Repository and AI features.)

This is Stb-tester’s first AI model, and it only applies to the “is_visible” property. Stay tuned for more AI features to come!

Your AI model is version-controlled

The training screenshots are stored in your test-pack git repository under the folder ai/is_visible/training/, so whenever you run a test it will use the model that was trained from that specific git commit.

This way, the behaviour of your tests continues to be version-controlled and reproducible — which, in our opinion, are fundamental requirements for any serious test automation.

Try it out!

You can adopt this new AI model incrementally. Use it for new development —any new Page Objects you write— or if you have an existing Page Object that is unreliable, or is taking a lot of maintenance effort, then try replacing it with AI!

We expect this new AI feature to speed up your test development and simplify your Page Object code significantly, while integrating seamlessly with your existing test scripts and emphasizing a “code first” approach of managing test automation as code. We plan to iterate quickly on ever more powerful and general AI models, so please send your feedback to support@stb-tester.com.

Here’s a short demo signing into the BBC iPlayer app. We use Stb-tester’s ocr API to read the sign-in code from the TV screen, and then we enter it into the browser using Selenium.

For detailed instructions see Selenium for second-screen login flows in the Stb-tester manual.

We have always recommended specifying the device type (e.g. “appletv”) in the Node-specific config file of each Stb-tester Node, in the device_under_test.device_type configuration key. This was a free-form field that you can read in your test scripts to make decisions such as which reference images to use, OCR language, etc.

Today we are broadening that recommendation to an entire configuration section called “device_profile”. This section is treated specially in the following ways:

1. When you save a screenshot to the Object Repository, Stb-tester will record the entire device_profile config section (copied from the Node-specific config file of the Node where the screenshot came from) as well as the device_under_test.device_type and ocr.lang config values. This allows the Object Repository to use the correct OCR language for that screenshot when testing your Page Objects; and your Page Object properties can read other values from this section.
2. For convenience, the Stb-tester Portal web interface can show the device_type or selected values from the device_profile configuration of each Node in the Nodes menu, the “video wall” or “virtual NOC” page, and the admin page. This is controlled by the new test_pack.node_label_format setting.

We have also changed the way that the node.friendly_name configuration is displayed in the Stb-tester Portal web interface:

3. The Node’s “friendly name” is now taken from the config files on the current git branch — previously the web interface always showed the friendly name from the main branch.
4. Similarly, editing the Node’s “friendly name” in the admin page will write the new value to the config files on the current git branch, instead of the main branch.

Changes 3 & 4 were made for consistency with the new node_label_format and with the existing behaviour of stbt.get_config("node", "friendly_name") when running a test script. This only affects the web interface; the friendly_name value in the response of the /api/v2/nodes REST API endpoint is always taken from the main branch (no change in behaviour there, as it’s a public API).

Motivation: Configuration for the Object Repository

When you run a test script against a real device, you might need to configure certain settings differently depending on the device. For example, if you are testing devices in different territories, you need to configure the language for OCR. This is done through the Node-specific config files, which are files named after the serial number of each Stb-tester Node, like this:

[device_profile]
name = Roku Mexico
[ocr]
lang = spa

When you run a test, the configuration is loaded to match the Node that is running the test. This is true for Stb-tester APIs such as stbt.ocr which reads the “lang” setting, and it’s true for configuration that you read explicitly by calling stbt.get_config.

However, when you are testing your Page Object code in the Object Repository, the code isn’t running on any particular Node; instead of a live device-under-test the Page Object code is tested against a screenshot. You want the configuration (such as OCR language) to match the device the screenshot came from. When you save a screenshot to the Object Repository, Stb-tester automatically records the following configuration:

• ocr.lang
• device_under_test.device_type
• The entire device_profile section

This configuration is saved into a “.conf” file next to the screenshot (for example selftest/screenshots/roku/Menu/screenshot.png.conf).

We don’t copy all of the configuration, only the above values, to minimize maintenance overhead. The Object Repository will continue to read common configuration (in .stbt.conf) from the current values on the selected branch; whereas the above values are frozen in time when you save the screenshot.

To recap:

When you run a test, calling stbt.get_config will read from these files on the git branch that you are running the test from:

• .stbt.conf (global config)
• config/test-farm/${node_id}.conf (the Node-specific config file for the Node running the test)

In the Object Repository, calling stbt.get_config will read from these files instead:

• .stbt.conf (global config)
• selftest/screenshots/${path_to_screenshot_png}.conf (recorded from the Node-specific config file at the time the screenshot was taken)

Recommendations

Any configuration that varies between different devices and needs to be read from Page Object properties, should be placed in the device_profile section of the Node-specific config file of each Node.

Note that Page Object properties should only read information from the screen (a video-frame or screenshot). They should not interact with the device-under-test, nor read information from the network. This device_profile section shouldn’t contain details about a single device (such as the device’s IP address) — that would belong in the device_under_test section. Avoiding high-cardinality values (IP addresess, serial numbers) in the device_profile section will help the Object Repository generate its data more efficiently, too.

Optionally, use the new test_pack.node_label_format setting in your global .stbt.conf file to display the device type (or any other device-specific configuration) in the Stb-tester Portal web interface.

We recommend using a pytest fixture to start and stop log capture. Pytest is the test-runner used by Stb-tester since v33; a fixture allows you to run code before and after every test-case. (See pytest’s documentation on fixtures.)

For detailed instructions see the new chapter in the Stb-tester manual: Capturing logs from the device-under-test.

Simply click the “Record video” button underneath the live video on the “Manual control” tab of the Stb-tester Portal.

Once it has started recording you can use the remote control as usual. When you are done, click “Stop Recording”, and it will provide a link which you can copy-paste into whichever system you’re using to manage your manual tests.

The link includes the video recording and the logs from the set-top-box itself, which are synchronised with the video. To configure log capture see Configuration Reference: Capturing logs from the device-under-test in the Stb-tester manual.

This new feature has been highly requested by our users. It serves a dual purpose:

• Easily capturing and sharing video recordings of UAT/discovery sessions: helps product owners to understand user behaviour and identify areas for optimisation.
• Automatically capturing ADB logs during manual testing sessions: saves testers’ time and enhances the testing experience.

Background

The Object Repository works by checking your Python code against canned screenshots of the device-under-test, instead of a live video feed. This provides a much faster turn-around time for your developers’ edit-test-debug cycle when writing or modifying your “Page Object” Python code. But if you had many of these canned screenshots (Stb-tester calls them “selftest screenshots”) the cycle was still too slow.

The Object Repository checks your Page Object code against each screenshot by running the code on an Stb-tester Node. This ensures that the behaviour is the same as when you run the full test scripts, and it simplifies deployment of the Object Repository tool (the code runs in the same Python environment, with the same Python libraries available, and the same network environment, as your real test scripts).

The performance improvements

Our new performance improvements come from the following changes:

1. The Object Repository only checks the screenshots that you select, on demand. (Previously, it would check your selected Page Object against all the selftest screenshots in your test-pack, and then show you the data for the selected ones.)

2. By default, the Object Repository selects the screenshots in selftest/screenshots/{appname}/{pagename}/**.png.
   To check other screenshots, click the “Select Screenshots” button.

3. The work of checking each screenshot is farmed out to all idle Stb-tester Nodes, in parallel. (Previously, all screenshots were processed on a single Node.)

Since founding our company 10 years ago, we have grown from a services-focused company in the first 2 years, to a primarily product-based company since 2016 with uninterrupted year-on-year sales growth and customers in 4 continents. In 2023 we had our best year yet, with sales up 50% compared to the previous year, and headcount up 25%. Now barely one quarter into 2024, sales are already on trajectory to exceed 2023’s growth, all the while we continue to release new features such as our latest “Smart Navigation” machine learning APIs.

Backward compatibility

Throughout all our new hardware models, architecture changes, and new APIs, we have maintained compatibility with the test scripts you have written with our older hardware & APIs. Backward compatibility is one of our core company values! We respect the time and effort you have invested into developing your test scripts. We have customers who started with our first “Stb-tester ONE” hardware appliance, migrated from the single-appliance model to our current cloud portal with distributed “Stb-tester Node” hardware, and are still running the same test scripts.

Thank you

We are very thankful to YouView who supported the original development of Stb-tester, and to you, our customers, for your business and feedback throughout the years. Here’s to another 10 years of mutually fruitful partnership!

About Stb-tester

Stb-tester is an automation tool for testing TV & video-streaming apps on real devices such as Android TV, Apple TV, Fire TV, PlayStation, Roku, XBox, and traditional Set-Top Boxes.

Key features:

• Easy to use: There’s very little code for the user (you!) to write.
• Widely applicable: It’s designed to handle all sorts of menus — lists vs. grids, wrapping vs. non-wrapping, fixed focus vs. moving focus, etc. To customise it for your UI, you only need to provide a Page Object that reads the name of the currently focused item.
• Robust: Even if the real menu has changed from what we learned the previous time, it will detect this and learn the new structure.

See this screencast for a demo and a thorough explanation:

These were always possible to do with Stb-tester, by navigating the relevant AppleTV screens, but pyatv is much faster because it doesn’t require navigation:

For usage, configuration and installation instructions, see Controlling Apple TV with pyatv in the Stb-tester manual.

For more details, including configuration instructions and API reference documentation, see Power Distribution Units in the Stb-tester manual.

This release also enables Pytest by default. Pytest is a popular Python test-runner that provides advanced functionality such as assertion introspection, and fixtures for test setup & teardown. Pytest support was added in Stb-tester v33, but it was disabled by default; in v34 it is the default (and only) test-runner. Our largest customers have been using the Pytest support with v33 for at least 6 months, so we think it’s time that all our users benefit from Pytest’s features. You shouldn’t see any change in behaviour from your tests, except that some of the output in stbt.log will be different (with additional, useful information). To learn more about Pytest see New features in v33: Pytest support.

This release includes several other minor additions and fixes. For full details see the release notes.

For example, let’s find the location of each “poster” or “tile” in this screenshot:

By default, segment starts from the top of the screen and travels down, finding distinct rows:

frame = stbt.get_frame()
regions = stbt.segment(frame)

Next, we can discard rows that are too small, and then run segment again, but horizontally: Starting from the left of the row, and moving right.

tiles = []
for row in regions:
    if row.height > 100:
        tiles.extend(stbt.segment(frame, region=row,
                                  initial_direction=stbt.Direction.HORIZONTAL))

We can combine both of these steps into a single call to segment, by specifying steps=2. The steps alternate between vertical and horizontal directions:

regions = stbt.segment(frame, steps=2)

Then we can discard regions that are too small to be a tile:

tiles = [r for r in stbt.segment(frame, steps=2)
         if r.height > 100]

Alternately, we could have limited the search region —if we know it— by specifying the region parameter of segment, instead of searching the whole frame.

Note that some of the regions aren’t the same height, because the bottom of the poster is too similar to the background color. If we set narrow=False, then segment will keep the top and bottom coordinates of the row from the previous step:

tiles = [r for r in stbt.segment(frame, steps=2, narrow=False)
         if r.height > 100 and r.width > 40]

Previously, these APIs could take a single rectangular Region to restrict the image-processing to that part of the screen. If you needed to process an area that isn’t a rectangle —for example processing the whole screen except for some parts— then you had to create a black-and-white mask in an image editor:

frame	mask	is_screen_black(frame, mask)
	None	False
		True

Since v33, you can construct such a mask in Python code by adding or subtracting regions. For example, the mask shown above can be created with the following code:

channel_patch = stbt.Region(x=570, y=264, right=715, bottom=450)
info_bar = stbt.Region(x=0, y=520, right=1280, bottom=690)
mask = stbt.Region.ALL - channel_patch - info_bar

To learn more, see Regions and Masks in the Stb-tester manual.

v33 includes many other improvements, including an upgrade to Python 3.10, APIs for interacting with Android and Roku devices, and more. See the release notes for instructions on how to upgrade.

For example, let’s find the current focus in the menu at the top of this screenshot:

We’ll match the logo in the top-left corner to check if the menu is present, and we’ll use find_regions_by_color to find the position of the white underline or “focus”:

Here’s a Page Object that implements this:

import stbt

class Menu(stbt.FrameObject):

    @property
    def is_visible(self):
        return bool (stbt.match("BT SPORT.png", frame=self._frame)
                     and self.underline)

    @property
    def underline(self):
        r = stbt.find_regions_by_color(
            "#fff",
            frame=self._frame,
            region=stbt.Region(x=0, y=80, right=1280, bottom=90),
            min_size=(20, 2))
        if len(r) == 1:
            return r[0]
        else:
            return None

find_regions_by_color takes a color in either of the following formats:

• A string in the CSS hexadecimal color format: White is "#ffffff" or "#fff".
• A tuple of 3 numbers in Blue, Green, Red order (note that this is the opposite of the CSS Red, Green, Blue order). White is (255, 255, 255).

It returns a list of regions. Here we are looking in the narrow band where we expect the white underline to be, so we only expect to find 1 region. If we find any more or any less, it’s probably a different page so we make is_visible return False.

Now we can use ocr to read the text above the underline:

class Menu(stbt.FrameObject):
    ...

    @property
    def focus(self):
        return stbt.ocr(
            frame=self._frame,
            region=self.underline
                       .above()
                       .extend(x=-10, right=10, bottom=-5))

We can test our Page Object in the Object Repository (simulated below). You can see that for each example screenshot, our underline property is finding the correct position (try hovering over the stbt.Region text in the table) and our focus property is reading the correct menu entry.

Menu

	channels.png	football.png
x	<Menu>	<Menu>
x.is_visible	True	True
x.focus	Channels	Football
x.underline	stbt.Region(x=205, y=83, right=285, bottom=85)	stbt.Region(x=323, y=83, right=397, bottom=85)

Here’s a quick example showing one Gherkin “feature” with two “scenarios” — one of them is a “scenario outline” to be run with different parameters or “examples”. They are marked with tags like “@menu” and “@nav”, to enable selecting or deselecting the scenarios to run. The BDD scenarios are on the left, with the Python step definitions on the right. For more details see the pytest-bdd documentation.

Configuration

You can pass additional command-line options to pytest-bdd in your .stbt.conf file like this:

[test_pack]
stbt_version = 33
pytest = --cucumberjson=steps.json

For each test-run, if an artifact called “steps.json” exists, the Stb-tester Portal will show the detailed pass/fail status of each step:

Assertion introspection

When an assert in your test fails, pytest prints additional information about the value that triggered the assertion. Without pytest the output looks like this:

    assert menu.selection == "Channels"
AssertionError

Pytest’s output looks like this:

>       assert menu.selection == "Channels"
E       AssertionError: assert 'Settings' == 'Channels'
E         - Channels
E         + Settings
tests/btsport/demo.py:13: AssertionError

Much more helpful!

To learn more, see pytest’s documentation about asserting.

Fixtures

Pytest’s fixtures allow you to define setup and teardown code that can be re-used in many tests. For example, the following fixture captures logs from the device-under-test while the test is running:

@pytest.fixture
def logs():
    start_capturing_logs("dut.log")
    try:
        yield
    finally:
        stop_capturing_logs()

(I’ll leave the implementation of start_capturing_logs() and stop_capturing_logs() up to you. 😁)

To use this fixture in any test, just pass it as an argument to the test:

def test_menu_navigation(logs):
    page = launch_my_app()
    page = page.navigate_to("Settings")
    assert page.selection == "Settings"

In the fixture, code before the yield will run before the test; code in the finally block after the yield will run after the test (even if the test raised an exception).

To learn more, see About fixtures in the pytest documentation.

Configuration

In Stb-tester v33, using pytest as the test-runner is optional. From Stb-tester v34, pytest is the default (and only) test-runner.

To enable pytest on v33, use the following configuration in your test-pack’s .stbt.conf file (you don’t need this on v34 or newer):

[test_pack]
stbt_version = 33
pytest = true

Notes for pytest experts

We run each test-case in a separate pytest process. This means that there’s no difference between “session” fixtures and “function” fixtures — each test function is run in a separate session.

Support for parametrized tests is limited: All parametrized variants are run as a single test-run, so you get a single test-result (a single video, single logfile, and single pass/fail outcome) for all of them.

Both of these limitations may be removed in a future Stb-tester release.

Previously, we would run Pylint in the cloud. The advantage of running lint on the Stb-tester Nodes is that we lint your code using exactly the same environment that we use to run your code when you run a test. This environment includes any third-party libraries that you install in your setup script, so Pylint will be able to check your code when you call those libraries. For our customers who install in-house libraries from a server on a private network, this enables linting that isn’t possible on a cloud-based runner.

We run Pylint in the background on any idle Node. If you start running a test on a Node that was running Pylint, Pylint will be transparently re-scheduled onto a different Node so that it doesn’t interfere with your tests. All of this is completely automatic and requires no configuration.

Starting today, we are also reporting the lint results using GitHub’s “checks” API, so the lint warnings appear directly in your pull request:

To learn more about Pylint —including how to configure or disable specific Pylint checks— see Static analysis for your test scripts: Catching common mistakes before you run the tests!

The “manual control” role allows users to control devices-under-test remotely, and see live video from the devices, without access to the test scripts in the test-pack git repository. This allows you to grant access to a third party without letting them see the test scripts you have developed which are your own intellectual property.

The Stb-tester Portal uses GitHub OAuth for single-sign-on: Any users with read, write, or admin access to the test-pack git repository have access to the Stb-tester Portal. Users with the new “manual control” role still need a GitHub account for authentication, but they don’t need access to the git repository.

To enable this feature, contact support@stb-tester.com.

To test your app you need a mechanism to control the Nvidia Shield and a way to get feedback to check that your app is behaving as it should.

Stb-tester sends CEC commands (over the HDMI connection) to control the Nvidia Shield, and captures the video over the same HDMI connection to verify your app’s behaviour. CEC is the same technology that allows you to use your TV’s remote to control the Nvidia Shield.

Reliability

The Shield’s CEC implementation is very reliable: We ran a soak test for 32 hours and there were 0 missed keypress or “double” keypress in 73,000 keypresses. The test script used stbt.press to navigate right & left on the Shield’s home screen, and after every keypress it checked that the selection had moved to the expected position.

Setup

You will need our USB CEC adapter. If we didn’t ship one with your Stb-tester device, contact support@stb-tester.com. Plug it in as per the diagram below — the Stb-tester Node will automatically use the CEC adapter if there is no infrared transmitter connected.

CEC is enabled by default on the Nvidia Shield.

Key names

In your test scripts you can use the following key names:

• "KEY_POWER"
• "KEY_UP"
• "KEY_DOWN"
• "KEY_LEFT"
• "KEY_RIGHT"
• "KEY_OK"
• "KEY_BACK"
• "KEY_HOME"
• "KEY_FASTFORWARD"
• "KEY_PLAY"
• "KEY_PAUSE"
• "KEY_REWIND"
• "KEY_VOLUMEUP"
• "KEY_VOLUMEDOWN"

"KEY_PLAY" and "KEY_PAUSE" work as expected (tested in the YouTube app). There’s no equivalent of the single “play/pause” button on the remote control (that is, CEC’s "KEY_PLAY" doesn’t pause and "KEY_PAUSE" doesn’t play) so in the remote control image for the Stb-tester Portal’s “manual control” clicking the left half of the button sends "KEY_PLAY" and the right half sends "KEY_PAUSE".

"KEY_FASTFORWARD" and "KEY_REWIND" work as expected (tested in the YouTube app). With long-hold the fast-forwarding/rewinding isn’t as fast as with the real remote control, but it does work.

There’s no CEC equivalent to the Settings button (we tried all 256 CEC key codes). However, you can still get to the same Settings menu by navigating to the gear icon on the home screen using "KEY_UP", "KEY_RIGHT" and "KEY_OK".

The Stb-tester Portal uses TLS certificates issued by Let’s Encrypt. Due to certain technical decisions made by Let’s Encrypt, software using older versions of OpenSSL (before 1.1.0) can get “certificate expired” errors when trying to access the Stb-tester Portal after 30 September 2021. Specifically RHEL/CentOS 7 are affected, as they use OpenSSL 1.0.2k.

The fix requires 2 steps:

1. If you are using RHEL/CentOS 7, upgrade your ca-certificates package. This issue was fixed in ca-certificates version 2021.2.50–72.

2. If you are using the Python requests library (or stbt_rig.py) then you need to tell requests to use the operating system’s CA (Certificate Authority) bundle, mentioned above, instead of the CA bundle that’s embedded into the requests library itself. To do this, set the environment variable REQUESTS_CA_BUNDLE before running stbt_rig.py, like this:

   export REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
   

We have other major new features in the works, but the ones in this article are available now to all our customers.

Prometheus & Grafana integration

Prometheus is an open-source time-series database used for monitoring and alerting. Grafana is an open-source dashboard and visualisation tool that integrates with Prometheus.

Using our new stbt.prometheus APIs, your test-scripts can record measurements for ingestion into a Prometheus database. On your Stb-tester Portal we can host a Prometheus & Grafana instance for you; or you can ingest the data into your own Prometheus instance.

Example use-cases include:

• Counters: The number of times the “buffering” indicator or “loading” spinner has appeared; the number of black frames or glitches seen; the number of tests that failed, or the number of VoD assets that failed to play; etc.
• Gauges: Measuring the temperature or memory usage on your device-under-test.
• Histograms: Performance measurements such as channel-change duration, app launch duration, or time for VoD content to start playing.

Since your tests are written in Python, performance measurements are as simple as calling time.time() before and after the action you want to measure.

Bluetooth & RF4CE remote controls

The Stb-tester press() API now supports RedRat-X hardware, to control the device-under-test via Bluetooth or RF4CE (ZigBee).

RedRat-X hardware must be purchased separately. Contact RedRat to see if your particular remote control is supported. Typically there is a one-off onboarding fee to support new remote controls.

Interactive log viewer

In the Stb-tester Portal’s test-results view, click on the timestamp of any log line to seek the video to that exact time:

This log is anything printed by your test-script — for example using Python’s print() function, or the debug information that is automatically printed when you call Stb-tester APIs.

The timestamp at the left of each line is the time when Python printed the line. Timestamps from a specific frame of video are also clickable — for example Frame.time, MatchResult.time and MotionResult.time in the debug output from various Stb-tester APIs. These frame timestamps come from our HDMI video-capture hardware. If you’re confused by the difference, consider this koan:

frame = stbt.get_frame()
time.sleep(10)
print(f"The frame was captured at: {frame.time}")

The output would be something like this:

The log viewer shows timestamps in your local timezone. (In the raw log file, the timestamps are always in UTC.)

We have also added controls to the video player to step forward or backward by 1 frame at a time, or 1 second, or 10 seconds.

Our new log viewer is a very powerful tool for triaging test failures and understanding the behaviour of your device-under-test. It’s also useful for spot-checking performance measurements, to gain confidence in the correctness of your test scripts!

Test-farm administration interface

The new Test-farm administration page in the Stb-tester Portal lists all of your Stb-tester Nodes, with their current firmware version and license expiration dates. It allows you to reboot Stb-tester Nodes, force HDMI re-negotiation to simulate hot-plugging the HDMI cable, and install firmware updates.

Access this page from the user drop-down menu in the top right corner of the Stb-tester Portal:

1. Auto-complete for stbt APIs.
2. API documentation in the IDE.
3. Linting results in the IDE including stbt specific lints.
4. Run tests directly from within the IDE, without having to commit or push.
5. Automatically pushing test-code to the portal for instant feedback in the Object Repository during test script development.

See it in action here:

We have recently revamped our tooling and instructions to make it easy to get started on Windows. There’s now a step-by-step guide (with videos). If you had problems setting up VSCode before, now is a great time to try again.

Check it out here in our manual: Development Environment Setup with VSCode. If you have any difficulty please contact us on support@stb-tester.com and we will help to resolve any issues.

Low latency

The latency improvements are best shown with a video. Here we have 2 browser windows open at the same page. The left window is using WebRTC, and the right window is using Flash:

With WebRTC the latency is under 500ms, and this includes the time to send the “keypress” instruction to the Stb-tester Node. Flash’s latency is around 2 seconds.

NAT traversal

WebRTC uses peer-to-peer communication to stream the video from the Stb-tester Node to your web browser — the video traffic doesn’t go via the Stb-tester Portal. If your PC is on the same network as the Stb-tester Node, the traffic doesn’t leave your local network.

This was also true of our old Flash video implementation; the difference is that WebRTC also has the ability to negotiate a peer-to-peer connection when you are on different networks, using ICE protocols. However, this only works with some firewalls/NATs.

When direct peer-to-peer communication isn’t possible, WebRTC allows tunnelling the video traffic through a TURN server. Contact sales@stb-tester.com if you want this capability.

Note that the Stb-tester Portal has always had a fallback mechanism for viewers on a different network, using JPEG thumbnails at 1 frame per second — see Live video formats in the Stb-tester manual for details.

WebRTC traffic is secured using industry-standard DTLS encryption.

Flash reaches end-of-life soon

Flash is being discontinued by Adobe after December 31 2020, and major browsers will disable their Flash support entirely.

To summarise:

Selecting the video format in the Stb-tester portal

To use WebRTC, click the settings icon underneath the video:

Browser settings

You may need to enable “auto-play” for the Stb-tester Portal. For example on Firefox, click the “permissions” icon next to the URL bar:

By the end of this tutorial we will have implemented a Page Object with an enter_text method, so you can write a test script like this:

page = Search()
page.enter_text("Peppa Pig")

Watch it in action:

Modelling the keyboard

First we will specify a Directed Graph that describes the behaviour of the keyboard under test. A “graph” (in the computer-science sense of the word) consists of “nodes” connected by “edges”. Each key on our keyboard will be a node in our graph, and the possible transitions from each key to its neighbours will be the edges:

Each edge specifies the button that you need to press on the remote control to trigger that transition: KEY_RIGHT, KEY_LEFT, KEY_UP, or KEY_DOWN.

Stb-tester will use this graph to calculate the shortest path to the target. For example, if the current selection is on “a”, to type the letter “p” Stb-tester would press KEY_RIGHT 3 times and KEY_DOWN twice (and then KEY_OK to type the selected letter). Note that there can be more than one shortest path.

To model this keyboard’s behaviour in our Python code, we can use the stbt.Keyboard API to specify each key’s name and region (its position on the screen; we’ll use this later) using Keyboard.add_key, and the transitions between keys using Keyboard.add_transition, like this:

kb = stbt.Keyboard()
kb.add_key(name="a", region=stbt.Region(x=125, y=175, width=50, height=50))
kb.add_key(name="b", region=stbt.Region(x=175, y=175, width=50, height=50))
kb.add_transition("a", "b", "KEY_DOWN")
# ...and so on for all the other keys...

Hold on, hold on. This keyboard is laid out in a regular grid, so instead of typing each key one by one, let’s use Keyboard.add_grid. Our keyboard has 3 different grids, shown below in different colours:

We specify all these keys like this:

kb = stbt.Keyboard()
kb.add_grid(stbt.Grid(region=stbt.Region(x=145, y=125, right=410, bottom=160),
                      data=[["abc", "ABC", "#+-"]]))
kb.add_grid(stbt.Grid(region=stbt.Region(x=125, y=175, right=425, bottom=475),
                      data=["abcdef",
                            "ghijkl",
                            "mnopqr",
                            "stuvwx",
                            "yz1234",
                            "567890"]))
kb.add_grid(stbt.Grid(region=stbt.Region(x=125, y=480, right=425, bottom=520),
                      data=[[" ", "DELETE", "CLEAR"]]))

Much easier, isn’t it! Keyboard.add_grid is only suitable if all the cells in the grid are the same size. You don’t need to be super-precise with the region coordinates — just make sure the centre of each key is inside the right cell.

        data=[
            ["a", "b", "c", "d", "e", "f"],
            ["g", "h", "i", "j", "k", "l"],
            ...etc...
        ]

Keyboard.add_grid will add all the keys and the transitions between them (within the grid). It won’t add transitions that go outside of the grid, so we need to add those explicitly, like this:

# abc ABC #+-
# ↕ ↕ ↕ ↕ ↕ ↕
# a b c d e f
kb.add_transition("a", "abc", "KEY_UP")
kb.add_transition("b", "abc", "KEY_UP")
kb.add_transition("c", "ABC", "KEY_UP")
kb.add_transition("d", "ABC", "KEY_UP")
kb.add_transition("e", "#+-", "KEY_UP")
kb.add_transition("f", "#+-", "KEY_UP")

# 5 6 7 8 9 0
# ↕ ↕ ↕ ↕ ↕ ↕
# SPC DEL CLR
kb.add_transition("5", " ", "KEY_DOWN")
kb.add_transition("6", " ", "KEY_DOWN")
kb.add_transition("7", "DELETE", "KEY_DOWN")
kb.add_transition("8", "DELETE", "KEY_DOWN")
kb.add_transition("9", "CLEAR", "KEY_DOWN")
kb.add_transition("0", "CLEAR", "KEY_DOWN")

Note that, by default, Keyboard.add_transition adds the opposite transition automatically, for example KEY_UP for KEY_DOWN or KEY_LEFT for KEY_RIGHT.

You may have noticed that some keys have two possible transitions for the same remote-control button — for example pressing KEY_UP from “SPACE” can land on “5” or on “6”. This reflects the keyboard-under-test’s real behaviour: It remembers which key you came from before navigating down onto “SPACE”, and it returns to the same column when you go back up. stbt.Keyboard doesn’t keep track of this state, so we just accept that both of those two keys (“5” and “6”) are valid targets.

Modes

This keyboard has three different modes: Lowercase, uppercase, and symbols. It’s best to think of each mode as an entirely different keyboard, with transitions that change between them: Pressing KEY_OK on one of the mode keys (like “ABC”) will go to that mode.

Some keys might appear in more than one mode. It’s important to model these as different keys — even though they look the same, they are different because they are connected to different keys. For example there is a “SPACE” key in all of the modes, but pressing KEY_UP from it will go to a totally different key:

The same is true for the “DELETE” and “CLEAR” keys, the mode keys (“abc”, “ABC”, and “#+-“), and the number keys.

To tell these apart in our model we specify mode when we call Keyboard.add_key, Keyboard.add_transition, or Keyboard.add_grid, like this:

top_grid = stbt.Grid(region=stbt.Region(x=145, y=125, right=410, bottom=160),
                     data=[["abc", "ABC", "#+-"]])
bottom_grid = stbt.Grid(region=stbt.Region(x=125, y=480, right=425, bottom=520),
                        data=[[" ", "DELETE", "CLEAR"]])
middle_region = stbt.Region(x=125, y=175, right=425, bottom=475)
middle_grids = {
    "lowercase": stbt.Grid(region=middle_region,
                           data=["abcdef",
                                 "ghijkl",
                                 "mnopqr",
                                 "stuvwx",
                                 "yz1234",
                                 "567890"]),
    "uppercase": stbt.Grid(region=middle_region,
                           data=["ABCDEF",
                                 "GHIJKL",
                                 "MNOPQR",
                                 "STUVWX",
                                 "YZ1234",
                                 "567890"]),
    "symbols": stbt.Grid(region=middle_region,
                         data=["!@#$%&",
                               "~*\\/?^",
                               "_`;:|=",
                               "éñ[]{}",
                               "çü.,+-",
                               "<>()'\""]),
}

kb = stbt.Keyboard()
for mode in ["lowercase", "uppercase", "symbols"]:
    kb.add_grid(top_grid, mode=mode)
    kb.add_grid(bottom_grid, mode=mode)
    g = middle_grids[mode]
    kb.add_grid(g, mode=mode)

    # Transitions between grids:
    #
    # abc ABC #+-  (top grid)
    # ↕ ↕ ↕ ↕ ↕ ↕
    # a b c d e f  (first row of middle grid)
    kb.add_transition(g[0, 0].data, "abc", "KEY_UP", mode=mode)
    kb.add_transition(g[1, 0].data, "abc", "KEY_UP", mode=mode)
    kb.add_transition(g[2, 0].data, "ABC", "KEY_UP", mode=mode)
    kb.add_transition(g[3, 0].data, "ABC", "KEY_UP", mode=mode)
    kb.add_transition(g[4, 0].data, "#+-", "KEY_UP", mode=mode)
    kb.add_transition(g[5, 0].data, "#+-", "KEY_UP", mode=mode)

    # 5 6 7 8 9 0  (last row of middle grid)
    # ↕ ↕ ↕ ↕ ↕ ↕
    # SPC DEL CLR  (bottom grid)
    kb.add_transition(g[0, 5].data, " ", "KEY_DOWN", mode=mode)
    kb.add_transition(g[1, 5].data, " ", "KEY_DOWN", mode=mode)
    kb.add_transition(g[2, 5].data, "DELETE", "KEY_DOWN", mode=mode)
    kb.add_transition(g[3, 5].data, "DELETE", "KEY_DOWN", mode=mode)
    kb.add_transition(g[4, 5].data, "CLEAR", "KEY_DOWN", mode=mode)
    kb.add_transition(g[5, 5].data, "CLEAR", "KEY_DOWN", mode=mode)

Now we just need to add the transitions between modes: If we’re in lowercase mode with the selection on “ABC”, pressing KEY_OK takes us to uppercase mode with the selection still on “ABC” (see Figure 4) — and so on for the other mode keys:

for source_mode in ["lowercase", "uppercase", "symbols"]:
    for name, target_mode in [("abc", "lowercase"),
                              ("ABC", "uppercase"),
                              ("#+-", "symbols")]:
        kb.add_transition(kb.find_key(name=name, mode=source_mode),
                          kb.find_key(name=name, mode=target_mode),
                          "KEY_OK")

Note that in this keyboard we can identify a key unambiguously by its name + mode. Some keyboards might have the same key twice in two different places in the same mode (for example two “shift” keys) — in that case you would model this as two separate keys with the same name & mode, but different region.

This keyboard has another way of changing modes: Pressing KEY_PLAY cycles through the modes. For example from “a” to “A” to “!” and back to “a”; or from “b” to “B” to “@”, etc.

To model this we need to add each a transition from each and every key in the keyboard. We can use Keyboard.find_keys to loop over the keys we have already added to the model, and Keyboard.find_key (singular) to find the corresponding target for each transition, like this:

for source_mode, target_mode in [("lowercase", "uppercase"),
                                 ("uppercase", "symbols"),
                                 ("symbols", "lowercase")]:
    for key in kb.find_keys(mode=source_mode):
        target = kb.find_key(region=key.region, mode=target_mode)
        kb.add_transition(key, target, "KEY_PLAY")

Identifying the currently selected key

We have modelled the keyboard’s behaviour. Now, to use that model we need to understand the current state of the device under test: Which key is currently selected?

With Stb-tester, the way we extract this information from the screen is to write a Page Object (a Python class) that does the necessary image-processing. Our Page Object class will have two properties:

• is_visible: Returns True if the keyboard is visible and focused.
• selection: Returns the currently selected key.

We can answer both of these questions (Is the keyboard visible? And which key is selected?) with stbt.find_selection_from_background. Let’s start with a simple example that only understands the lowercase keyboard:

class Search(stbt.FrameObject):
    """The YouTube search keyboard on Apple TV."""

    @property
    def is_visible(self):
        return bool(self.selection)

    @property
    def selection(self):
        match = stbt.find_selection_from_background(
            "lowercase-background.png",
            max_size=(115, 70),
            frame=self._frame,
            mask=stbt.Region(x=125, y=125, right=425, bottom=520))
        if match:
            return kb.find_key(region=match.region, mode="lowercase")
        else:
            return None

stbt.find_selection_from_background compares the video frame (captured from the device under test) against the specified reference image (“lowercase-background.png”). This reference image is a screenshot of the keyboard without any selection. Thus, any differences between the frame (which does show a white rectangle around the selected key) and the reference image are going to tell us where the selection is. If the differences span a larger region than the size of the biggest key (max_size above), then it means that we’re looking at a different screen — not the keyboard.

You may need to create this selection-less image by merging two different screenshots together. This video shows how to do it in the GNU Image Manipulation Program, a free open-source cross-platform image editor:

Step by step instructions:

1. Open both screenshots. They must have the selection on different, non-overlapping keys.
2. Drag one of the open image tabs onto the other tab and drop it into the Layers window so that it’s above the existing layer.
3. Use the rectangle selection tool to select the part of the image that contains the selection.
4. Choose Edit > Clear (or press Delete on your keyboard). The layer underneath will show through the deleted region, showing the same key but without the white rectangle.
5. Use File > Export As… to save the image in PNG format to your test-pack.
6. Don’t forget to commit the image to git! (Until you do, your IDE will show a lint error underneath the filename to remind you.)

Now, to recognize all three modes, we need to create similar (selection-less) reference images for the other modes: “uppercase-background.png” and “symbols-background.png”. Finally, we update our Search.selection property so that it looks like this:

    @property
    def selection(self):
        for mode in ["lowercase", "uppercase", "symbols"]:
            match = stbt.find_selection_from_background(
                mode + "-background.png",
                max_size=(115, 70),
                frame=self._frame,
                mask=stbt.Region(x=125, y=125, right=425, bottom=520))
            if match:
                return kb.find_key(region=match.region, mode=mode)

        return None

Tip

You can visualise & debug your Page Object’s properties in the Object Repository tab of your Stb-tester Portal:

To learn more about Page Objects see Object Repository in the Stb-tester manual, and the stbt.FrameObject API reference documentation.

Navigating the keyboard

Now we have all the pieces we need to navigate the on-screen keyboard:

• A way to tell which button is currently selected.
• A graph that tells us the shortest path from each key to any other key.

We’ll add a method to our Search Page Object called enter_text. This will take a text parameter, and it will type the given text into the on-screen keyboard by calling Keyboard.enter_text:

class Search(stbt.FrameObject):
    ...

    def enter_text(self, text):
        return kb.enter_text(text, page=self)

Keyboard.enter_text will use the selection property of its page parameter to see which button is currently selected. Then it will loop over each letter in text: Find a key in our model that matches that letter, navigate to it, and press KEY_OK to type the letter.

kb.add_key(name="@gmail.com", text="@gmail.com",
           region=stbt.Region(...), mode=...)

kb.add_grid(stbt.Grid(
    region=...,
    data=[
        [{"name": "@gmail.com", "text": "@gmail.com"}, {"name": "another key", ...}],
        ...
    ]))

We can also navigate to a single key using Keyboard.navigate_to. For example, here is the implementation of a clear method that we can provide so that our test scripts can clear any text that has been entered into the Search page:

class Search(stbt.FrameObject):
    ...

    def clear(self):
        kb.navigate_to("CLEAR", page=self)
        stbt.press_and_wait("KEY_OK")
        return self.refresh()

Some keyboards have an explicit “SEARCH” button that you have to press after typing the text. For those keyboards, our Page Object’s enter_text method would look like this:

    def enter_text(self, text):
        page = self
        page = kb.enter_text(text, page)
        page = kb.navigate_to("SEARCH", page)
        stbt.press_and_wait("KEY_OK")
        return page.refresh()

Common mistake: Using an outdated page instance

Keyboard.enter_text and Keyboard.navigate_to take a Page Object instance in their page parameter. This instance has a selection property that reflects the position of the selection at the time the instance was created. If this instance is out of date (because the selection has moved since that time), then stbt.Keyboard will calculate a path from the wrong start position to your target node.

This is because Stb-tester’s Page Objects are immutable: An instance of the Page Object reflects the state of the device-under-test at the time the instance was created.

The following code won’t work:

    # EXAMPLE OF BAD CODE -- DON'T COPY
    def enter_text(self, text):
        kb.enter_text(text, page=self)
        kb.navigate_to("SEARCH", page=self)  # <-- self.selection is outdated!
        stbt.press_and_wait("KEY_OK")

To get the latest state, you can create a new instance of the Page Object like this:

page = Search()

Or like this:

page = self.refresh()

(where self is an instance of our Search Page Object.)

For this purpose, Keyboard.enter_text and Keyboard.navigate_to return a new page instance that reflects the state of the device-under-test after the text has been entered (or the navigation completed). We can use their return value instead of calling self.refresh(). Here’s the corrected example:

    # FIXED EXAMPLE
    def enter_text(self, text):
        page = self
        page = kb.enter_text(text, page)
        page = kb.navigate_to("SEARCH", page)
        stbt.press_and_wait("KEY_OK")
        return page.refresh()

Note that we have made our method return an updated page instance. This is consistent with Keyboard.enter_text’s behaviour, and it allows any testcases that call our enter_text method to use the same pattern.

Simple keyboards (no modes)

Many on-screen keyboards don’t have modes — there’s just a single, uppercase or lowercase keyboard. Or maybe your keyboard does have several modes, but you don’t care to test them — you just want your test script to type in a search term, and the case doesn’t matter. (Typically the different modes are only really necessary for login keyboards where you type a password.)

In this case, you don’t need to specify mode when you call Keyboard.add_key, Keyboard.add_grid, or Keyboard.add_transition. The key’s name alone, or the name + region, will be enough to identify a key unambiguously. You will need to make your enter_text method convert to lowercase or uppercase, as appropriate, like this:

class Search(stbt.FrameObject):
    ...

    def enter_text(self, text):
        return kb.enter_text(text.lower(), page=self)

Shift modes

If your keyboard has a “shift” mode, where pressing KEY_OK on an uppercase letter types the letter and changes to the lowercase mode, you can model this by specifying a transition from every key in the uppercase mode, to the same key (by region) in the lowercase mode. The code would look somewhat similar to the example earlier in this article for changing modes by pressing KEY_PLAY.

See the full code from this tutorial here.

The are several other new APIs and many fixes and quality-of-life improvements. Upgrading should be painless for most users; the only backwards-incompatible change that will affect most users is with stbt.Keyboard. A detailed tutorial explaining the new API is available here: Testing on-screen keyboards with Stb-tester.

For more details see the v32 release notes.

“STB-Tester has made it possible for us to provide test facilities for colleagues in India, as well as in Belgium, who are stuck at home because of the coronavirus crisis. I can't stress enough how crucial this STB-Tester infrastructure is for us nowadays. Our colleagues rely on it for being able to continue their work.”

In the Stb-tester portal you can see a live video stream from each set-top box and you can press buttons on the remote control. All you need is a web browser. This doesn’t require a VPN nor a complex firewall setup.

Stb-tester stores an HD video recording of each test that is executed. This aids clear communication: a defect is much easier to explain if you can share a video demonstrating it.

The set-top boxes themselves can also be located anywhere: Some in the test-lab at your office, some at home, on different networks. This allows testing the set-top box’s behaviour on different real-world network conditions. The physical setup is very easy: Just connect an Stb-tester Node to the network (so that it can reach the Stb-tester Portal) and to the device that you will be testing.

For more details see Announcing the Stb-tester Cloud Platform.

If you use GitHub Enterprise, the Stb-tester Portal can integrate with a test-pack (git repository of test scripts) hosted on your GitHub Enterprise instance to provide the following features (also available for private git repositories hosted on github.com):

• Transparently sync code from GitHub Enterprise to the Stb-tester Portal and to the Stb-tester Nodes that execute the tests.
• Transparently sync code from your IDE during test development.
• Record the exact version of the test scripts that were used in each test-run.
• Run tests from any git branch.
• Interactive test-development tools on the Stb-tester Portal, such as the Object Repository.

Furthermore, using GitHub Enterprise allows you to log into the Stb-tester Portal using your organisation’s existing Single Sign-On mechanism — any users that are authorised to access the test-pack git repository, will automatically be authorised to access the Stb-tester Portal. You don’t need to manually manage a separate set of user accounts or permissions.

Clicking the “Run Test” button will automatically perform the following steps:

• Sync your code (including local uncommitted changes) to the Stb-tester Portal. This uses a temporary git branch so it doesn’t change your local workspace and it doesn’t affect other users.
• Run the test remotely on your chosen Stb-tester Node.
• Show the test’s output in VS Code’s Output window.

This is useful during test-development, when you are writing or debugging the test. It greatly speeds up the development cycle because you don’t need to sync the code with a git commit & git push; and you don’t need to switch to the Stb-tester Portal in your browser to select the test or view the logs.

After you are satisfied with your changes, you still need to do a git commit & git push to share your new test with the rest of your team — see our recommended Development Workflow.

To set up this functionality in VS Code, see IDE Configuration in the Stb-tester manual.

Stb-tester provides a Pylint plugin that teaches Pylint about some mistakes that are specific to Stb-tester APIs. For example, the frame parameter of stbt.match is usually optional, but it’s mandatory when called inside a FrameObject property:

Another common mistake is using a reference image that doesn’t exist on disk, or that hasn’t been committed to git:

Some Stb-tester APIs raise an exception on failure (for example stbt.wait_for_match) but others don’t (for example stbt.match and stbt.wait_until, among others). This can be confusing, so we report an error if you forget to use the return value from the latter group of APIs:

Disabling Pylint warnings

In some rare cases you might want to ignore the return value for a valid reason. You can disable the warning for that specific line, by adding a comment like this:

Some of Pylint’s built-in checks are related to coding style, not real problems. If they get annoying you can disable specific checks by listing them in the .pylintrc file in the root of your test-pack. For example:

[MESSAGES CONTROL]
disable=
  line-too-long,
  missing-docstring,

The name of each checker (such as “line-too-long”) is shown after the error message in your IDE.

IDE Configuration

To set up your IDE to catch all of these errors, see IDE Configuration in the Stb-tester manual.

If you’re curious about the full list of checks that Stb-tester adds to Pylint, see the source code of our Pylint plugin here.

In our test scripts we often need to look for something by navigating through a menu. For example, let’s say we’re looking for an app on the Apple TV’s “running apps” view, so that we can quit it:

We have a Page Object called AppSwitcher that knows how to read the app name from this view (circled in red above). The following Python code presses the LEFT button on the remote control until it finds the desired app (up to 20 times) and then it kills the app by double-clicking the UP button:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
def kill_app(app_name):
    double_click("KEY_TV")  # Open the "running apps" view
    page = wait_until(appletv.AppSwitcher)
    assert page, 'Failed to open the "running apps" view'

    for _ in range(20):
        if page.title == app_name:
            # Kill the app by double-clicking KEY_UP:
            double_click("KEY_UP")
            return
        stbt.press_and_wait("KEY_LEFT")
        page = page.refresh()

    assert False, "Didn't find app '%s'" % (app_name,)

Once it finds & kills the desired app, this code returns (on line 10) so it doesn’t keep on pressing LEFT looking for other apps. The assert False statement on line 14 only runs if you didn’t find the target app and didn’t return early from the loop.

But what if you want to exit the loop without returning from the function?

Sometimes it isn’t convenient to put each for loop into its own function. Instead of return we can use break, and we can put our error-handling in an else clause.

Here’s the official Python documentation on the for statement:

When the items are exhausted, the suite in the else clause, if present, is executed, and the loop terminates.

A break statement executed in the first suite terminates the loop without executing the else clause’s suite.

Our previous example would look like this:

1
2
3
4
5
6
7
8
9
10
11
    for _ in range(20):
        if page.title == app_name:
            # Kill the app by double-clicking KEY_UP:
            double_click("KEY_UP")
            break
        stbt.press_and_wait("KEY_LEFT")
        page = page.refresh()
    else:
        assert False, "Didn't find app '%s'" % (app_name,)

    # Here I can do other stuff before returning

Note that assert False on line 9 only runs if we didn’t break from the loop.

Line 11 always gets run — it doesn’t matter if the loop ran all 20 times, or if it exited early due to the break statement.

An example of retry logic: Handling an unknown number of dialogs

When testing playback on a video-streaming app (specifically, BBC iPlayer) we needed to handle a series of dialogs before the content starts playing — anywhere between 0 and 4 dialogs:

Our test script to handle this is shown below. If we see the “Resume” dialog above, we’ll select “Start from the beginning”. We loop (retry) several times so that we can handle any other dialogs that might appear, and we break from the loop when we detect that content playback has started.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
def press_ok_to_start_content():
    stbt.press("KEY_OK")

    # Handle any dialogs that may appear:
    for _ in range(5):
        page = wait_until(lambda: Content() or Dialog())
        assert page, "Failed to detect content start or dialog"
        print("Page is a %s" % (type(page),))

        if isinstance(page, Dialog):

            if page.selection in [
                    "I am aged 16 or older",
                    "I understand, continue",
                    "Start from the beginning",
                    "Watch from Start",
                    "Watch Live",
                    "I have a TV Licence. Watch now.",
                    "Continue without setting up lock"]:

                stbt.press_and_wait("KEY_OK")

            elif page.selection in [
                    "Resume",
                    "Cancel",
                    "I don't have a TV Licence.",
                    "Set up Parental Guidance lock"]:

                stbt.press_and_wait("KEY_DOWN")

            else:
                assert False, "Unknown dialog %r" % (page,)

        else:  # Content
            break
    else:
        assert False, "Didn't get to content after dismissing 5 dialogs"

    # Check playback:
    assert stbt.wait_for_motion(mask="content-spinner-mask.png")

(Note: Content and Dialog are Page Objects that know how to recognize the content playback screen and the dialog screens, respectively. Dialog has a property called selection that reads the text of the selected button — for example “Resume” in the screenshot above).

Why not use a while loop?

We could have written the above loop like this:

1
2
3
4
5
6
7
8
9
10
11
12
    page = wait_until(lambda: Content() or Dialog())
    assert page, "Failed to detect content start or dialog"

    # Handle any dialogs that may appear:
    while isinstance(page, Dialog):
        # ... Dialog-handling code here (not shown), and then:

        page = wait_until(lambda: Content() or Dialog())
        assert page, "Failed to detect content start or dialog"

    # Check playback:
    assert stbt.wait_for_motion(mask="content-spinner-mask.png")

But Python’s while loops have 2 problems:

1. We have to repeat the code that updates page twice: In lines 1-2 and lines 8-9. It’s more boilerplate.

2. Bugs in the test script or in the system-under-test can lead to an infinite loop. We would have to add explicit timeout checks inside the loop — more boilerplate.

The for…else pattern allows us to add assertions to our retry logic, to handle the error case with a minimal amount of extra code.

Installing this package will enable your IDE’s “IntelliSense” or “Code Insight” features such as code completion, type inference, inline documentation, and linting.

For installation instructions see IDE Configuration in the Stb-tester manual.

The page looks like this:

My Page Object’s search_text property reads the search term that the user has typed using the on-screen keyboard. First we look for the magnifying-glass icon, to determine the region where the text should appear:

Then we read the text to the right of the icon. Here is the code of my Page Object property that does this:

class Search(stbt.FrameObject):
    # 'is_visible' and other properties not shown

    @property
    def search_text(self):
        icon = stbt.match("search-icon.png", frame=self._frame)
        if icon:
            return stbt.ocr(
                frame=self._frame,
                region=icon.region.right_of(width=240))

One of my tests failed because the text didn’t match the value I was expecting. First we save and commit a screenshot under selftest/screenshots/. (I recommend that you organise the screenshots in subfolders according to device, application, and Page Object, for example selftest/screenshots/appletv/youtube/keyboard/peppa pig.png.)

Then we can debug the Page Object against this screenshot in Stb-tester’s Object Repository:

In the Object Repository, your Page Object’s properties are run against the committed screenshots, instead of a live video-frame from the device under test. Here we can see that the Page Object’s search_text property is reading “Denna piq” instead of “peppa pig”. To investigate, let’s click on the property’s drop-down menu and select Debug:

This will show detailed debug for each of the Stb-tester APIs that are called to evaluate this property (stbt.match to find the magnifying-glass icon and stbt.ocr to read the text). Now the problem is obvious: The region that we are giving to stbt.ocr is chopping off the bottom of the text!

The solution is to extend the region downward by a few pixels so that it includes the descenders in the text:

     @property
     def search_text(self):
         icon = stbt.match("search-icon.png", frame=self._frame)
         if icon:
             return stbt.ocr(
                 frame=self._frame,
-                region=icon.region.right_of(width=240))
+                region=icon.region.right_of(width=240).extend(bottom=5))

Now we can see in the object repository that search_text is correct:

Tip: When you’re trying out code changes to debug an issue like this, you can use stbt_rig.py snapshot to sync the code from your IDE to the Stb-tester Portal every time you press “Save”. See IDE Configuration in the Stb-tester manual.

To learn more about Page Objects, see Object Repository in the Stb-tester manual.

Note that there is no immediate pressure to migrate existing Python 2 test-packs. Python 2 test-scripts will continue to work, even after Python 2’s official sunset date in January 2020.

• Coordinate across your team so that nobody is doing new development with Python 2 until the porting to Python 3 is complete.

• Check out a new git branch called “python3”.

• Delete all Python files that you aren’t using, and delete any “dead code” that you aren’t using inside the remaining files. All the test-scripts are version-controlled with git, so there is no need to keep unused code – you can always get it back from the git history if you really need to.

  Commit this deletion to git.

• Update .stbt.conf to contain the following in the [test_pack] section, and commit the change:

  [test_pack]
  stbt_version = 31
  python_version = 3
  

• If you install any third-party packages using the config/setup/setup script:

  • Change apt install python-<package-name> to apt install python3-<package-name>.
  • Change pip install <package-name> to pip3 install <package-name>.

  …and commit the changes.

• Run 2to3 to translate all your Python files to Python 3. This is a script that comes with Python. The name of the script might be 2to3 or 2to3-2.7 depending on your OS. Run it like this:

  2to3 --write tests/*.py
  

  If you have Python files in sub-directories under tests/ you will need to specify them too. You can get the full list of files by running this:

  git ls-files "tests/*.py"
  

  Make sure that you only run each file once through 2to3.

• Inspect those changes manually, and make sure you understand them. Commit the changes to git.

• Push your branch, and create a pull request. Stb-tester automatically runs a static analysis of the code in your pull request. This means that you can check that your tests are valid Python 3 code, without even having to run the tests!

  Stb-tester shows the result of the static analysis as a green tick or red cross on your pull request. Click the red cross to see a log with details of the failures.

• Fix the errors, commit, push, and check the new static analysis results in the pull request. Repeat until the static analysis is green.

  Some of the static analysis checks are about coding style, not actual errors. You can disable these checks if you want, by adding them to .pylintrc in your test-pack (the name of each check is shown in the static analysis log).

• Finally run the tests to catch any remaining problems that aren’t caught by the static analysis (if any).

• Merge the pull request to master.

For more details about pull requests, see Development Workflow in the Stb-tester manual.

You must specify which Python version to use in your test-pack’s .stbt.conf file like this:

[test_pack]
stbt_version = 31
python_version = 3

Valid values are “2” and “3”. The tests are run in an Ubuntu 18.04 environment, so you get Python 2.7 or 3.6, specifically.

We recommend Python 3 for all new test-packs, but there is no immediate pressure to migrate existing Python 2 test-packs. Python 2 test-scripts will continue to work, even after Python 2’s official sunset date in January 2020. To port an existing test-pack see Porting a test-pack to Python 3.

This release also contains new APIs that make it much easier to test on-screen keyboards. We’ll publish tutorials on this blog soon; in the meantime see the stbt.Keyboard reference documentation.

As always, changes to the stbt core Python API are version-controlled. You can specify the version you want to use in your .stbt.conf file. This allows you to upgrade in a controlled manner, and to test the upgrade on a branch first.

This release contains many other improvements and bug-fixes. For full details see the v31 release notes.

Matching buttons with unknown text

Take this keyboard for example:

We want to find the currently selected key. The selection is green, unlike the rest of the keys, but each key looks different – they have a different letter on the inside. With transparency we can have a single reference image that will match this selection no matter which key is selected.

In the video below we use the GNU image manipulation program to construct a suitable reference image:

Steps:

1. Open the screenshot.
2. Use the crop tool to crop to the area you want to match.
3. Right-click on the layer in the “Layers” pane and click “Add Alpha Channel”.
4. Use the selection tool to select the area that you want to be transparent.
5. Press the “delete” key on your keyboard.
6. Use “File > Export” to save the image as a PNG to your test-pack.

Here’s the reference image we saved:

The middle of the key may look like a white box in your web browser – but it is in fact transparent.

We can now use this image in our test scripts directly:

matched = stbt.match("keyboard-key.png")
assert matched
print "Selected key %s in region" % (
    stbt.ocr(region=matched.region), matched.region)

More realistically we’d use it as part of a Page Object to find the selected key:

class KeyboardKey(stbt.FrameObject):
    @property
    def is_visible(self):
        return bool(self.region)

    @property
    def region(self):
        m = stbt.match("keyboard-key.png", frame=self._frame)
        return m and m.region

    @property
    def label(self):
        return stbt.ocr(region=self.region, frame=self._frame)

You can use the KeyboardKey Page Object like this:

key = KeyboardKey()
print "Selected letter %s in region %s" % (key.label, key.region)

Finding outline selections on carousels

On the Roku’s Home page, the focused item is shown by a white outline:

How to find which item is currently focused? Create a reference image as before:

This will match the currently selected tile, independent of what the tile contains.

We can now use this in our Page Objects:

class CarouselSelection(stbt.FrameObject):
    ORIGIN = (117, 149)

    @property
    def is_visible(self):
        return bool(self.region)

    @property
    def region(self):
        m = stbt.match("roku-outline.png", frame=self._frame)
        return m and m.region

Determining which page we’re on and checking background details

A current fashion in set-top box UIs is the “clean look” – UIs showing as little as possible so as to not distract from the content. Here’s an example:

Most of the pixels here are going to change depending on what content is available and what is currently selected. That doesn’t mean there’s nothing to match on. Using transparency we can invert our idea of matching: instead of looking for a part of the frame that is unique to this page, we start with an example frame and delete the areas that contain dynamic content.

This can be useful for:

• Identifying what page you’re currently viewing – typically as a part of the implementation of the is_visible property on FrameObject classes.
• Checking for regressions/changes in UI layout.

We end up with a template like this:

This captures the blue background and the light-blue box surrounding the text on the right hand side. There are very few features in this image that we’re matching against. This template will only match against frames that are similarly featureless in these areas. In practice this may be enough to identify this page, but it can also be used as a “first pass” to discard non-matches before doing more expensive checks involving OCR.

This type of matching is very fast because the template is the same size as the frame so we don’t need to search all the locations in the frame to find a match.

Example

Here in our is_visible implementation we use the transparent template to check the page before calling OCR to read the title.

class EntertainmentGrid(stbt.FrameObject):
    @property
    def is_visible(self):
        return (self.match("entertainment-grid.png", frame=self._frame) and
                self.title == "Entertainment")

    @property
    def title():
        return stbt.ocr(region=stbt.Region(90, 50, 193, 38), frame=self._frame)

This is great for test-script maintenance. Full page transparent templates are easy to view and to understand, and when your UI changes they’re easy to update too.

The run-time environment of the test scripts has also been upgraded from Ubuntu 14.04 to Ubuntu 18.04. This means that any third-party packages that you install with apt install will be upgraded too.

In particular, Tesseract (the engine used by stbt.ocr) has been upgraded to version 4.0. Overall the OCR accuracy should improve slightly, but there may be regressions or differences in behaviour in some cases.

As always, changes to the stbt core Python API and the Ubuntu run-time environment are version-controlled. You can specify the version you want to use in your stbt.conf file. This allows you to upgrade in a controlled manner, and to test the upgrade on a branch first.

This release also contains many other improvements and bug-fixes. For full details see the v30 release notes.

For example:

For instructions see IDE Configuration in the Stb-tester manual.

To test your app you need a mechanism to control the Roku and a way to get feedback to check that your app is behaving as it should. With the other Roku models Stb-tester emulates an infrared remote control, but the Roku Streaming Sticks don’t have an infrared receiver (instead, their remote controls use Wi-Fi Direct).

All Roku models also provide an HTTP-based API for controlling the Roku device, which is used by Roku’s mobile app. Stb-tester now supports this API natively, allowing you to test Roku models that don’t have an infrared receiver (that is, you can now test the Roku Streaming Stick and Streaming Stick+).

Configuration

See Roku HTTP remote control in the Stb-tester manual for configuration & usage instructions.

Reliability

We have tested 60,000 keypresses over 3 days without a single missed keypress (we validate that each and every keypress had the expected effect in the Roku’s UI).

Roku devices never really go to sleep, so you won’t have any issues in waking the device using the HTTP control mechanism.

You can now test the following scenarios related to audio:

• Is audio playing at all?
• How long does it take for the audio content to start playing?
• Does the mute button work?
• Do the volume buttons work?
• Do you hear clicks/bloops when you press the keys to navigate around your UI?
• Is the loudness of advertisements within the permitted levels defined by EBU R128 / ITU-R BS.1770?
• Measure A/V synchronisation (detailed example coming soon).

Furthermore, the video-recording of each test-run now includes the audio track.

We have also added APIs for pressing-and-holding a button on the remote control. For example, you can now test these scenarios:

• Powering off set-top boxes that require you to hold the power button for several seconds.
• How fast does the volume bar change as you hold “Volume Up”?
• How soon does the actual volume change as you hold “Volume Up”?
• Does channel zapping continue to occur while you hold the “Channel Down” button, and does it stop as soon as you release the button?
• Quickly scroll to the end of a carousel or menu.

Another new API is stbt.press_and_wait, which makes it really easy to detect and measure transitions: For example, has the selection finished moving after you press up/down/left/right; or transitions to a new screen, such as going from live TV to the programme guide. This helps with scenarios such as these:

• Frame-accurate performance measurement of animations, for example a menu selection that slides from one entry to the next. Measure the time for the animation to start after the keypress, and then time for the animation to complete.
• Measure how long your Guide takes to appear (time to first frame) and then the time to populate fully.
• Measure the performance of every single keypress when navigating through your UI.
• Faster, more robust navigation.

This release also contains many other improvements and bug-fixes. For full details see the v29 release notes.

1. Make your changes on a branch
2. Test your branch
3. Make a pull request
4. Merge to master

Stb-tester helps you to follow best practices by making it easy to follow this workflow. It stops developers from stepping on each other’s toes, it fosters communication and peer review, and it ensures that all your changes are tracked and version-controlled.

Stb-tester will even run a static analysis of the code in your pull requests, automatically. This can catch common issues like syntax errors, some logic errors, and failure to comply with your team’s coding style.

For a detailed explanation see Development Workflow in the Stb-tester manual.

If your development team has an established workflow feel free to use that instead — but keep in mind that test-script development can benefit from a lighter-weight workflow than the workflows that are appropriate in other contexts.

Bonus tip: Debugging your test scripts can get tedious if you have to make a git commit for every change you want to test. stbt-rig is a command-line tool that can run your local code-changes on an Stb-tester node without having to make git commits or clicking in the web portal. Follow the link above for instructions.

To test your app you need a mechanism to control the Fire TV and a way to get feedback to check that your app is behaving as it should.

Stb-tester sends CEC commands (over the HDMI connection) to control the Fire TV, and captures the video over the same HDMI connection to verify your app’s behaviour. CEC is the same technology that allows you to use your TV’s remote to control the Amazon Fire.

Setup

You will need our USB CEC adapter. If we didn’t ship one with your Stb-tester device, contact support@stb-tester.com. Plug it in as per the diagram below — the Stb-tester Node will automatically use the CEC adapter if there is no infrared transmitter connected.

CEC is enabled by default on the Fire TV. If you had previously disabled it, you can re-enable it by going to Settings > Display & Sounds > HDMI CEC Device Control.

Key names

In your test scripts you can use the following key names:

• "KEY_UP"
• "KEY_DOWN"
• "KEY_LEFT"
• "KEY_RIGHT"
• "KEY_OK"
• "KEY_BACK"
• "KEY_HOME"
• "KEY_OPTIONS"
• "KEY_REWIND"
• "KEY_PLAY"
• "KEY_FASTFORWARD"

Waking from sleep

The Fire TV doesn’t implement the CEC command for “power on”. To work around this, you can stop the Fire TV from going to sleep by running the following ADB commands (this setting isn’t available from the Fire TV’s on-screen settings menus):

1. First enable ADB in Settings > Device > Developer Options > ADB debugging.
2. Go to Settings > Device > About > Network. Note the Fire’s IP address.
3. In your terminal, run adb connect 192.168.1.208 (substituting your Fire’s actual IP address).
4. Again in your terminal, run adb shell settings put secure sleep_timeout 0

This setting persists after power-cycling the Fire TV.

For more help on steps 1-3 see Amazon’s developer documentation Connect to Fire TV Through ADB.

Inter-press delay

If you send several keypresses too quickly, the Fire TV doesn’t always send the ACK (acknowledgement) specified by the CEC protocol. When this happens, Stb-tester’s stbt.press function will raise an exception, to avoid false positives in your tests.

To prevent this from happening you can enforce a minimum delay of 3 seconds between keypresses, by specifying interpress_delay_secs in your test-pack’s .stbt.conf file:

[press]
interpress_delay_secs = 3

The Stb-tester Platform now supports Atlassian’s Bamboo Server for Continuous Integration (CI).

Many of our customers already use Bamboo for continuous integration, deployment, and delivery. Now, it will only take a few minutes to get your Stb-tester end-to-end tests running from your Bamboo jobs.

For full instructions see Continuous Integration in the Stb-tester manual.

This is in addition to Stb-tester’s existing first-class support for Jenkins. And integrating with other systems is also possible using our HTTP REST API directly.

Start by pressing the Screenshot button underneath the live video view in the Stb-tester portal (as usual). This will take a lossless screenshot from the system-under-test as a PNG file.

From here you can drag on the image to crop it: Select the part that you want to match in your test-case, then press the Download button to save it to your test-pack.

For more details see Reference images in the Stb-tester manual.

Changes to stb-tester’s core Python API are version-controlled. You can specify the version you want to use in your .stbt.conf file. See Advanced Configuration: test_pack.stbt_version in our manual.

We generally expect that upgrading to new versions will be safe; we strive to maintain backwards compatibility. But there may be some edge cases that affect some users, so this mechanism allows you to upgrade in a controlled manner, and to test the upgrade on a branch first.

Version 28 includes support for multi-threading in your test scripts; better OCR performance for text on translucent overlays; higher-quality and higher-framerate video recordings of each test-run; and many API improvements. For details see the v28 release notes here.

The Stb-tester cloud platform is the result of a year’s work on a new client/server architecture and new node hardware, based on customer requirements and feedback. It enables new use-cases such as remote testing and monitoring, or testing real consumer network conditions from a variety of locations.

The Stb-tester platform consists of:

• A cloud-based portal, which provides the user interface and stores your test results; and
• One or more Stb-tester nodes — physical hardware that sits next to your “devices under test” (DUTs). An Stb-tester node captures video from your DUT, and executes your automated test scripts.
  • The Stb-tester HDMI Node captures video via HDMI for testing a Set-Top Box, Roku, Apple TV, PlayStation 4, XBox One, or similar devices.

The Stb-tester cloud platform has seen tremendous adoption since we released it in April 2017. Our largest customers have upgraded to the new cloud platform and have expanded their test farms. We have also acquired dozens of new customers across 3 continents, including large deployments in France and Belgium.

Designed to scale

The Stb-tester platform was designed with scalability in mind. Tests are executed on the Stb-tester nodes, allowing low latency and easy horizontal scaling to many devices.

The Stb-tester nodes are small and draw around 16W at full load, so you can fit several nodes, plus your set-top boxes, on a single rack shelf.

The test results from all your devices are aggregated in the portal, where you can quickly filter to compare results across different devices and software versions.

Simple, fair pricing

Our pricing is a yearly license fee per Stb-tester node (1 node = 1 simultaneous device-under-test). It is cheap to start with a single node, and you can scale up by adding more nodes. Significant discounts apply for large test farms.

Compared to our previous product, our pricing has changed to a rolling license model, which significantly reduces the up-front cost in the first year (total cost of ownership over 3 years is about the same as our previous pricing model). The minimum license period is 1 year. There are no cancellation charges if you choose not to renew.

Customers who prefer an up-front capital expenditure can buy a 5-year license at a discount. A separate “Enterprise” pricing model is available for test farms of 100 units or more.

Easy to set up and manage

Just connect your Stb-tester node to the network and to the device-under-test, and you’re ready to go. From any web browser you can see live video from the device, control it manually, and run automated tests.

The Stb-tester node only needs outbound access to the Stb-tester portal. You don’t need to open holes in your firewall.

All the configuration of the test-farm is centralized. The Stb-tester nodes are stateless, so you can swap hardware in or out easily. Furthermore, we manage the storage, backups, monitoring, and software updates automatically.

Our hardware is rock solid — you can run soak tests for days or weeks without having to worry about the stability of your video-capture hardware, drivers, or anything else.

All this makes the Stb-tester platform ideal for teams to “self serve” without needing a dedicated sysadmin or test infrastructure team.

Secure

Our platform was built with security as a core feature. Your portal and test-results database are hosted in a private AWS instance per customer — they are not multi-tenant. Test artifacts (logs, screenshots, and videos) are stored securely in S3. Traffic is secured with industry standard HTTPS encryption, authenticated with client certificates (the nodes) or OAuth (the human users). All test artifacts are encrypted at rest.

Access control is enforced per user, with the ability to grant read, write, or admin access. Currently we support GitHub’s single sign-on for user account management.

The Stb-tester cloud platform is trusted by large operators and OEMs, including Dish, Telstra, Walmart, and Sony.

Modern tooling

The Stb-tester platform integrates with the development tools you are already using: GitHub for source control, pull requests and code review; Jenkins and Bamboo for Continuous Integration; IDEs such as PyCharm and Visual Studio Code for test-development. We also provide a REST API to enable you to develop custom integrations.

Additional integrations will be provided out-of-the-box based on customer demand.

Backwards compatible

The new hardware is 100% compatible with test scripts written for our previous product. Backwards compatibility is one of our core company values.

Request a demo!

To request a demo, please contact sales@stb-tester.com.

Automated user-interface testing tools for mobile apps tend to test an internal representation of the UI — a hierarchical structure of nodes such as the example shown here for the “Now TV” app. (If you’re familiar with web development, this is similar to a web page’s DOM.) Tools like Appium or Calabash can check that a node with the ID “videoplayer” is present, but they can’t check whether the video player is actually working.

Capturing screenshots over USB

These tools do have the capability to capture screenshots, which you could analyse with Stb-tester’s image-processing APIs. Unfortunately, these screenshots may not include the video plane, depending on the app. For example, you don’t see video content with Netflix, but you do with Now TV:

App	iPhone	Android
Netflix		Returns invalid image data
Now TV

Presumably the difference is due to different DRM technologies used by the two apps.

Note that the white text in the Netflix screenshot is a subtitle, which is layered on top of the video by the Netflix app. Only the video itself is missing from the screenshots.

The frame rate (of frames delivered to the test script) is poor, because the mobile device is encoding and transferring large PNGs. Expect around 2 to 0.2 frames per second depending on the device and the complexity of the image in a particular frame. This might increase the load on the device-under-test, which could affect subtle timing conditions in your app — if your test is trying to reproduce some kind of intermittent defect, its results might not be representative.

The great advantage of this method is that it is simple, it doesn’t require any test hardware, and it is provided by many mobile testing tools. Stb-tester supports this method (we use ADB to capture screenshots from Android devices, and WebDriverAgent for iOS).

Capturing video over HDMI

Apple iPhones & iPads can output video to HDMI using a Lightning Digital AV Adapter. Similarly, some (but not all) Android devices can output HDMI using an MHL cable. See a full list of supported Android devices here.

Once again, it’s up to the app to support this. This time the situation is reversed: Netflix is happy to play the video over HDMI, whereas Now TV brings up a dialog saying “Sorry, we don’t support HDMI streaming to your TV from mobile devices”.

App	iPhone	Android
Netflix
Now TV

Note that the Netflix behaviour is slightly different across platforms. On Android the device’s screen is mirrored to the HDMI output, so UI elements like the playback controls appear on both screens. On iOS the video plays on the HDMI output while the device goes into “second screen remote control” mode: The device’s screen shows a static image plus the playback controls.

When you’re not playing video, all devices mirror the screen to the HDMI output. This works in all apps including the operating system home screen. When the device is in portrait mode you get some letterboxing on the sides (HDMI doesn’t support portrait resolutions).

A big disadvantage of this method is that it doesn’t leave a USB connection available, so the PC that is running the tests will have to find another way to send input events (simulated taps & swipes) to the mobile device. The Apple adapter and Android MHL cables all have an additional USB connector, but this is used exclusively to power the mobile device. On Android devices you can work around this by enabling ADB over TCP/IP, but this requires a USB connection to set up in the first place so it will require manual intervention if the device reboots or loses WiFi connection.

Capturing video with a camera

This is the most “black box” approach of all, so it works with any device and any app. Just point a camera at the device’s screen!

Our product, the Stb-tester CAMERA, performs automatic calibration on each video frame in real time so your test script sees only the mobile device’s screen, regardless of the camera’s position.

We have made the Stb-tester CAMERA extremely straight-forward to install and configure. Nevertheless, it does have some disadvantages: There is more hardware and it takes up more space. You need to cover the mobile device and camera with a box to eliminate reflections on the screen. The analogue nature of the process introduces some lossiness, and in particular some motion blur.

Summary of video-capture methods

The following table summarises the pros & cons of the various methods for capturing video frames from iOS & Android devices:

Our recommendations

If you don’t need to verify video playback and you just want to test user interaction within an app, we recommend a tool like Appium, which is tailored specifically for mobile UI testing. Services like Sauce Labs and Testdroid can even host the mobile devices for you, so you don’t have to deal with dozens of physical devices.

If you’re already using Stb-tester to test a set-top box (using HDMI video-capture) and you want to test the interaction between the set-top box and a “second screen” mobile app (remote control, TV guide, etc): Use USB screenshot capture from your existing Stb-tester scripts.

If you really do need a black-box view of your device’s screen (to verify video playback on any app & device, especially useful for monitoring teams): Use the Stb-tester CAMERA.

Set-Top Box

PS4

Stb-tester uses HDMI video-capture for feedback, and –traditionally– infrared to control set-top boxes. But the PS4 doesn’t have an infrared receiver. Since stb-tester v27 we can now control the PS4 using CEC commands over the HDMI connection.

We tried several control mechanisms, and CEC was the only one that worked consistently and reliably. The PS4 allows you to plug in a USB keyboard, so we considered using a Flirc Infrared-to-USB-keyboard dongle. However the USB keyboard input only works with some apps, and it doesn’t work at the PS4 login screen. Sony provides a mobile app with remote control capabilities, but it has similar limitations.

The CEC control mechanism is very reliable: I ran a soak test for 3 days and there wasn’t a single missed keypress in 120,000 keypresses. The test script used CEC commands to navigate right & left through a menu, and after every keypress it checked that the selection had moved to the expected menu item.

Setup

You will need our USB CEC adapter. If we didn’t ship one with your Stb-tester device, contact support@stb-tester.com. Plug it in as per the diagram below — the Stb-tester Node will automatically use the CEC adapter if there is no infrared transmitter connected.

To enable CEC on the PS4, go to Settings > System > Enable HDMI Device Link. On the PS3 it’s System Settings > Control for HDMI.

First the complete solution, for the impatient:

DEVICE=/dev/serial/by-path/platform-3f980000.usb-usb-0:1.3.3.1:1.0-port0
PORT=4953
socat -b 2048 -u "\"$DEVICE\",b115200,crnl,raw,ignoreeof,echo=0" - </dev/null \
    | ts %Y-%m-%dT%H:%M:%S%z \
    | gst-launch-1.0 fdsrc ! text/plain ! queue \
                     ! tcpserversink sync=false host=0.0.0.0 port=$PORT

This uses socat, the Swiss Army Knife of networking; ts from the moreutils package to timestamp each line; and GStreamer as our TCP server.

The above socat command transfers data from the serial device to standard output. The two layers of quoting are necessary so that socat doesn’t treat the “:” in the device path as separators. See the socat manual for details on the other parameters.

Then we pipe socat’s output to ts, which prepends an ISO 8601 timestamp to each line.

Finally, gst-launch runs a GStreamer pipeline that reads the text data from its standard input, starts up a server listening on the specified TCP port, and sends the data to any clients that connect. GStreamer is perhaps overkill for this but it gets the job done, and it supports multiple clients connecting/disconnecting and reading the same data simultaneously.

You’ll want to configure your init system to start the above service automatically on startup. I’ll leave that as an exercise to the reader.

Testing it

You can test this using netcat. This should print your set-top box’s serial output to the console:

nc $serial_server_address 4953

Or using socat:

socat TCP:$serial_server_address:4953 -

Installing the dependencies

Assuming you’re using Raspbian on a Raspberry Pi, here’s how to install these tools:

apt-get update &&
apt-get install \
    gstreamer1.0-plugins-base \
    gstreamer1.0-tools \
    moreutils \
    socat

Specifying the serial device

On a single server you can have multiple versions of the above script serving different serial devices, as long as each one uses a different TCP port.

We use /dev/serial/by-path/... (instead of /dev/ttyUSBx) because it won’t change across reboots. It refers to the physical USB port that your serial adapter is plugged into.

To find the path of your serial device, just run ls /dev/serial/by-path before and after plugging it in.

USB serial adapters

For out-of-the-box Linux support and good reliability, you only have two options:

• The Tripp-Lite Keyspan adapter.
• Anything using FTDI chips (buy direct from FTDI or from reputable vendors, as there are counterfeit chips on the market).

Some of our users care about frame-accurate performance measurements. For example, how many frames of video does it take a menu transition to complete? The Stb-tester HDMI provides the following guarantees:

• Your test scripts can process every frame of video, up to 60 frames per second, using the APIs provided by stb-tester (like match and wait_for_motion) or with custom image-processing using OpenCV.
• Stb-tester reports the timestamp of each frame.
• The timestamp is in wall-clock time (UTC), so you can compare it against log entries from the device-under-test, your head-end systems, etc.
• On the Stb-tester ONE the timestamp is accurate to within 35ms. At 60 frames per second, this is 2 frames of video. On our new Stb-tester HDMI hardware the timestamp is accurate to within 0.9ms (at 60 frames per second a single frame is 16ms, so stb-tester is providing frame-accurate timings by a wide margin).
• There is no long-term drift (we’ve tested every frame over 4 days of continuous test execution).

How we test & validate video-capture timestamps

Our test system consists of an stb-tester ONE capturing the video from a Raspberry Pi’s HDMI output:

The Raspberry Pi is generating a video on the fly. Each frame of video has a time drawn on it in binary — this is the time at which the frame will be sent over HDMI, according to the Raspberry Pi’s MMAL video subsystem. We have open-sourced the code that generates and displays this video: See stb-tester/latency-clock on github.

The receiver (an stb-tester Python script) decodes the time drawn on the frame, and compares it against the time reported by the video-capture process.

The Raspberry Pi uses a GPS-stabilised clock, by using a GPS expansion board. This keeps the Rasperry Pi’s system clock stable to ~20µs (as reported by ppstest). The Raspberry Pi also runs an NTP server; the stb-tester ONE runs an NTP client that synchronises against the Raspberry Pi’s clock, to an accuracy of ~300µs (as reported by ntpq). For instructions see Not quite 5 minute guide to making an NTP Server.

It sounds obvious, but so many organisations fail at test automation because they don’t run the tests often enough to get real value. We have seen teams spend a year (no exaggeration) building elaborate test infrastructure without producing any test results. Run your tests! Don’t spend any more time automating new testcases until you are consistently running the tests you already have!

Costs of automated vs. manual testing

The costs of automated testing are very different to the costs of manual testing. A manual testcase is relatively cheap to write and relatively expensive to run, whereas automated testcases are relatively expensive to write and very cheap to run — almost free.

So what’s going to give you more value for the least effort: Writing more automated tests, or running them more often?

I’m not suggesting that you shouldn’t aim for high test coverage. But pick the low-hanging fruit first!

Run your tests against more builds

If your QA team gets infrequent releases from your development team, and you only run your automated tests against each release, some of your tests will fail because they haven’t kept up with changes to your set-top box’s UI.

If you only run your tests once a month, and you also have to fix the tests each time, then it might be cheaper to stick with manual testing!

To take advantage of automation’s strengths, you want to be running your tests against every change made by the development teams, in a Continuous Integration system (CI). The benefits of CI are well known: Developers can work faster and more productively, with confidence that they aren’t introducing regressions. Bugs are caught earlier, and thus they are easier and cheaper to fix.

Run your tests on more devices

If you need to support several different set-top box models, you will often get more value by extending your existing tests so that they work on all the different models, than by increasing the test coverage for a single device.

If the user journeys are similar across your devices, you can use the Frame Object technique to handle UI variations while using the same test scripts for all your devices.

Run your tests more times: Soak testing for stability

Even the most trivial testcases can sometimes find memory leaks and crashes if you run them long enough. If your set-top box is going to sit idle, why not leave your tests running all night? You’ll be surprised at the bugs you’ll find.

Soak-testing is especially valuable if you’re a set-top box manufacturer or integrator. If you’re developing an app for someone else’s device you’ll be less interested in device instabilities, but a soak test can still find resource leaks in your own app.

Even if you don’t have time to look into every test failure, you can get good value by implementing some simple health-checks to detect crashes, spontaneous reboots, and memory leaks.

And even if you don’t look at the results of every overnight soak, run it anyway! That way you’ll have data if you need to answer “when did this stop working?” or “when did this start getting slower?”.

Run your tests more times: Soak testing for functional coverage

If you run your tests in a random order you will end up exercising user journeys that aren’t explicitly coded into your test scripts, and this can often uncover bugs that you wouldn’t find otherwise. (These bugs tend to be the kind that are really annoying to diagnose if they’re only noticed in the wild!)

What do I do next?

Don’t spend any more time automating new testcases until you are consistently running the testcases you already have.

Don’t let this be a manual process. Set up a Jenkins job (or similar) that runs every night: It deploys the latest software build to your device-under-test, and runs all your automated tests in a random order, for 16 hours (or however long it takes before you get into the office the next morning).

Depending on your development team’s process and tools, deploying the latest software build to your device-under-test could be the hardest part to automate. Don’t give up! On previous projects we have automated the update process of set-top box firmware that requires plugging in a physical USB stick, by using a controllable USB switch.

Once this is in place, the next step (automatically testing every change in Continuous Integration) will be easy!

If you’re a customer of ours and you have an stb-tester ONE, we will be posting you a replacement infrared transmitter free of charge within the next few days. All you have to do is install the v24.7 software update on your stb-tester ONE, unplug the old infrared transmitter, and plug in the new one. The new transmitter is perfectly reliable – we have tested more than 500,000 keypresses without a single missed keypress.

This doesn’t affect our customers with larger test rigs (who are using the RedRat irNetBox instead of our USB infrared transmitters).

Technical summary: Our old infrared transmitter used the FTDI FT232R USB-to-serial chip in bit-bang mode. We proved that the FT232R’s clock is unreliable in bit-bang mode, even the newer “C” revision of the chip. The FTDI FT230X chip doesn’t have such problems.

Read on for details of the defect, the solution, and our test methodology.

Reproducing the problem: Missed keypresses

We became aware that the infrared transmitter we ship with the stb-tester ONE wasn’t entirely reliable: When your test script called stbt.press, sometimes the device-under-test wouldn’t react – it didn’t see the keypress.

When facing an intermittent defect, the first thing we need is statistically significant data on the reproducibility of the defect. Otherwise we won’t know if our changes are helping or hurting, or making any difference at all. So we picked a Roku box as our target –the Roku is very reliable, for a consumer electronics device– and we wrote a test script that sends 20 keypresses, checking that every keypress had the expected effect:

def test_ir_transmitter():
    to_roku_home()
    for _ in range(10):
        press("KEY_DOWN")
        assert wait_until(lambda: find_selection().text == "My Feed")
        press("KEY_UP")
        assert wait_until(lambda: find_selection().text == "Home")

Starting from the Roku home screen where the menu selection is on “Home”, this testcase presses DOWN and checks that the menu selection has moved to “My Feed”. Then it presses UP and checks that the selection has moved back to “Home”. It repeats this 10 times for a total of 20 keypresses. press and wait_until are functions provided by stb-tester; to_roku_home and find_selection are Roku-specific helper functions that we have defined elsewhere in our test-pack. The whole testcase only takes a few seconds; you can watch a single run of the testcase (where every keypress succeeded) in the video below:

After running the testcase repeatedly for a few hours we have 1,368 test-runs, and 79 of those failed. Each test-run sends 20 keypresses, so we saw 79 missed keypresses out of approximately 25,800 keypresses: 0.3% of all keypresses were missed.

In the video below you can watch me spot-check some of the test failures, just to make sure that the failures are really caused by missed keypresses and not some other issue.

Running the same testcase against a different Roku model gave much worse results: About 4% missed keypresses. So it seems that some infrared receivers are more susceptible than others. At least we’ve found a good device to test our future fixes against!

Now: Is it really the fault of our infrared transmitter? It could be caused by our infrared config file (which describes the protocol used by the Roku’s remote control), or by the Roku itself. To check, we ran the same test using the RedRat3 infrared transmitter. RedRat is a company based near Cambridge, UK, who specialise in infrared and other remote control technologies; they are the gold standard in infrared control automation. The RedRat3 is a USB infrared transmitter, but unlike ours, it outputs a high-power signal so it isn’t usually suitable in a test lab with dozens of set-top boxes that are running independent testcases.

Using the same test script, we didn’t see a single failure in over 30,000 keypresses using the RedRat3. This rules out the Roku and our infrared config file, leaving only our USB infrared transmitter hardware or software to blame.

We also tested the RedRat irNetBox and we didn’t see any missed keypresses either. So our customers with larger test rigs aren’t affected by this issue, as they use the RedRat irNetBox instead of our USB infrared transmitters.

Analysing the output from our infrared transmitter

Infrared signals are a sequence of pulses and spaces. This is the signal for the Roku’s DOWN button. It’s a long header pulse & space, followed by a sequence of shorter pulses & spaces:

During each pulse (in blue) the infrared transmitter is switching on and off at a certain carrier frequency, usually 38kHz. You can’t actually see the carrier signal in these diagrams. During the spaces, the infrared transmitter isn’t sending anything at all.

We set up a RedRat3 as an infrared receiver, and used the mode2 tool from lirc to log the pulses & spaces for each keypress as we ran our testcase.

Here are a good keypress and a bad keypress from one of the test-runs:

In the bad signal (the bottom one), about halfway through, it looks like the pulses start slightly earlier than the good signal. It also seems that the effect is cumulative, so towards the end of the signal the pulses get earlier and earlier.

After looking at several bad signals there were no other obvious differences, so we turned to some statistical measures. Here’s a histogram showing the mean difference between the ideal & actual pulse or space, within a given keypress:

The x axis is in microseconds. The green and red lines represent the good & bad keypresses from the stb-tester ONE’s infrared transmitter. There’s no obvious difference between the two groups of signals – maybe this is because in a bad keypress it only takes one bad pulse to cause the Roku to miss the keypress, but for a given keypress we’re measuring the average error across all the pulses & spaces for that keypress.

What is clear, however, is that there is a lot of variability to the duration of these pulses & spaces. Compare to the blue data, which is using a RedRat3 as the transmitter: Its pulses & spaces have much less variation from the ideal duration.

A histogram of the total signal length shows a similar curve, which confirms that variations in the length of one pulse have a knock-on effect on the rest of the signal:

At this point in the investigation, our best theory is: There is too much variability in our infrared transmitter’s output, and this is causing the missed keypresses. This could explain why some set-top boxes are affected more than others – perhaps their decoding algorithm is more or less tolerant of variations in the signal.

A minimal testcase: Generating a plain carrier frequency

Can we see the same variability in the carrier signal itself? The data we captured with mode2, above, doesn’t tell us the carrier frequency so we’ll need an Oscilloscope.

After purchasing a Hantek DSO-2090 and making a few patches to OpenHantek, we could see the raw signal coming out of our infrared transmitter. Here is what happens when we send a single pulse of a fixed length, repeatedly:

You can see that the total length of the pulse varies quite a bit, and the carrier signal within the pulse isn’t clean at all.

Our infrared transmitter used the FTDI FT232R chip, which is a USB to serial converter. The chip has two modes of operation: RS-232, and bit-bang. In bit-bang mode you specify a clock rate, and send it a sequence of ones and zeros; the chip writes these ones and zeros to its output pins at the specified clock rate. Unlike RS-232, there are no control bits (start bit, stop bit, parity bit) so we have complete control over the signal.

In theory we would set the clock rate to twice the infrared protocol’s carrier frequency, and send 1-0-1-0-1-0-etc to generate the carrier signal. In practice, the FT232R’s bit-bang mode only works at certain clock rates so we use a higher clock rate and send, for example, 1-1-1-1-1-0-0-0-0-0-1-1-1-1-1-etc (or however many ones we need to generate the right carrier frequency).

To analyse the chip’s behaviour in the simplest way possible, we wrote a simple C program using libftdi that sets the clock rate and sends 1-0-1-0-1-0-1-0-1-0-1-0-1-0-1-0-1-0-1-0-1-0-1-0-1-0-1-0-1-0-1-0-1-0-1-0-1-0-1-0 (20 ones and zeros). Then we re-wrote our test program so that it uses FTDI’s own D2XX SDK instead of libftdi. This is what we used to generate the signal you saw in the oscilloscope video, above.

We bought some new chips directly from FTDI, just in case our manufacturer was using counterfeit chips. Nope, the output is still bad. (Sorry manufacturer for doubting your supply chain skills.)

FTDI’s FT232R Errata Technical Note acknowledges that this is a known problem with Revision A of the chip, but claims that it has been fixed in Revision B. Our chips are Revision C, which is supposed to be identical to Revision B. But we have proved that big-bang mode is still unreliable in Revision C of the FT232R chip. We sent our code & data to FTDI support, who acknowledged the bug.

The solution: FTDI FT230X

The FTDI FT230X is a similar USB-to-serial chip that also has a bit-bang mode, and most importantly, has a stable clock. Even better, we can set any clock rate we want (to match the desired carrier frequency of our infrared signal) so that we can minimise the necessary USB bandwidth (we only need to send 1-0-1-0… instead of 1-1-1-1-1-0-0-0-0-0-1-1-1-1-1…).

With the FT230X, the signal as seen on an oscilloscope is perfectly stable, and our Roku testcase passes 100% of the time – we have tested more than 500,000 keypresses without seeing a single missed keypress.

We have written a lirc driver for the FT230X, which is now on the lirc master branch, in case anyone else wants to build their own USB infrared transmitter based on the FT230X. We are proud to have created the first reliable, low-powered, infrared transmitter on the market in a USB dongle form-factor.

If you’re a customer of ours and you have an stb-tester ONE, we will be posting you a replacement infrared transmitter within the next few days. All you have to do is install the v24.7 software update on your stb-tester ONE, unplug the old infrared transmitter, and plug in the new one.

• Introduction to the stb-tester ONE: Physical setup of the HDMI & infrared connections; using the web interface for manual control and running automated testcases; demonstration of an example testcase against a Roku set-top box; using the interactive results interface.

• Creating new testcases: The stbt Python API; capturing screenshots from the device-under-test and adding them to your test-pack; using the git version-control tool to manage testcases and deploy them to the stb-tester ONE.

• Measuring channel change time: Measuring channel change time on a set-top box, then using Jupyter notebook to explore the data interactively and generate a histogram of the measurements.

All three videos are available on our Videos & tutorials page. We have many more tutorials planned, so stay tuned – subscribe to this blog or follow us on twitter for updates.

Read how stb-tester.com supported YouView’s launch on Sony Bravia TVs.

DTVKit provides proven royalty-free DVB, MHEG, CI+, and HbbTV stacks offered through a shared source code model that is membership based and not-for-profit.

Stb-tester provides the perfect solution to allow automated system level testing of DTVKit, not only for our own testing, but also enabling us to make our testing scripts available for use to DTVKit members. Phil Evans, CEO, DTVKit.

A Frame Object is a class that extracts information from a frame of video, typically by calling stbt.ocr() or stbt.match(). All the rest of your testing code then uses these objects. A Frame Object translates from the vocabulary of low-level image processing functions and regions (like stbt.ocr(region=stbt.Region(213, 23, 200, 36))) to the vocabulary of high-level features and user-facing concepts (like programme_title).

Here’s an example of a Frame Object (GuideFrame) and a test-case that uses it:

class GuideFrame(object):
    def __init__(self, frame=None):
        if frame is None:
            frame = stbt.get_frame()
        self.frame = frame

    def __nonzero__(self):
        return self.is_visible

    def __repr__(self):
        if self.is_visible:
            return ("GuideFrame(is_visible=True, programme_title=%r, "
                    "current_time=%r)") % (
                    self.programme_title, self.current_time)
        else:
            return "GuideFrame(is_visible=False)"

    @property
    def is_visible(self):
        return bool(stbt.match('guide-header.png', frame=self.frame))

    @property
    def programme_title(self):
        return stbt.ocr(
            region=stbt.Region(213, 23, 200, 36), frame=self.frame)

    @property
    def current_time(self):
        m = stbt.match('clock.png', frame=self.frame)
        return stbt.ocr(
            region=m.region.extend(x=20, right=100), frame=self.frame)


def open_guide():
    stbt.press('KEY_GUIDE')
    guide = stbt.wait_until(GuideFrame)
    assert guide
    return guide


def test_that_guide_displays_the_correct_time():
    guide = open_guide()
    assert guide.current_time == datetime.datetime.now().strptime('%H:%M')

This will seem very much like a Page Object – it acts as a layer of abstraction on top of the low-level image processing functions. Unlike a Page Object it contains no behaviours or any means to stimulate the device under test:

• There are no implicit dependencies on time or external stimulus once constructed.
• There no calls to stbt.get_frame() (except in the constructor). We pass self.frame into every call of stbt.match(), stbt.ocr() or any other image-processing function.
• We perform no image-processing or other expensive work in the constructor - this object is very cheap to construct.
• The object is immutable – there are no methods to change the externally visible state of the object. If you want an updated view of the system-under-test’s state you can construct a new object for each frame.
• There are no calls to stbt.wait_until() or time.sleep() - this object can’t change so there is no point waiting.
• There are no calls to stbt.press() or anything else that affects the device under test.

These may seem like arbitrary or unreasonable restrictions but they provide some great benefits:

• Testability: Unlike full Page Objects, Frame Objects can be tested very cheaply. All you need is a corpus of example screenshots. This is because Frame Objects behave in an entirely deterministic way based on the parameters provided to the constructor (typically the only parameter is a frame of video). Most significantly, Frame Objects don’t depend on the dynamic behaviour of the device-under-test which is far more difficult to capture than a few screenshots.

  By providing a __repr__ method you can test these objects using doctests. This opens up other possibilities, which we will cover in a future blog post.

• Agility: Because of their improved testability, Frame Objects are cheap to maintain and easy to evolve. If the cosmetics of the application under test change, or you find a bug in your Frame Object, just capture a screenshot of that situation and add it to your test corpus. You can then fix it with confidence:

  • Confidence that the updated Frame Object fixes the issue – you can verify this by applying your new Frame Object to the new screenshots added to your corpus.
  • Confidence that you haven’t introduced regressions with your fixes – you can see that your Frame Object behaves the same when run against your existing corpus of screenshots.
  • Confidence that you haven’t changed your expectations of the behaviour of the device-under-test – the Frame Object contains no behavioural expectations.

• Consistency: Calling stbt.match or stbt.ocr multiple times to extract information from a frame can lead to confusing inconsistencies if each function sees a different frame. The Frame Object Pattern avoids this by capturing a frame exactly once.

• Performance: Frame Objects extract required information from the frame when needed, rather than in advance. This can avoid doing a lot of expensive image matching or OCR when it’s not necessary. Because the object is immutable and each property or method on the object is deterministic you can apply memoisation, allowing you to write tests in a natural manner without a performance penalty.

So that’s all well and good, but what about the behaviours that you removed? After all they are usually the thing you want to test in the underlying device. It turns out that the Page Object and the Frame Object patterns are actually complementary rather than conflicting. Where necessary you can still have page objects, but they use Frame Objects to extract the information from the underlying frames.

Two classes rather than one sounds complicated? In our experience the page object is no longer necessary in most cases and you can get away with just using helper functions or using the Frame Objects directly from test cases. But when you have more complex interactions and your helper functions need to store state you can still create page objects. These page objects will use Frame Objects: the additional overall complexity is more than offset by the reduction in complexity in the more expensive to maintain page objects – an overall win for maintainability.

For example here’s a helper function that works in conjunction with the above Frame Object:

def select_next_programme(self, direction=1):
    key = {-1: 'KEY_LEFT', 1: 'KEY_RIGHT'}[direction]
    original_title = GuideFrame().programme_title
    for _ in range(10):
        stbt.press(key)
        if wait_until(lambda: GuideFrame().programme_title != original_title,
                      timeout_secs=3):
            return GuideFrame()


def test_that_we_can_scroll_back_and_forth_through_programmes_in_the_guide():
    guide = open_guide()
    original_title = guide.programme_title
    assert original_title

    new_title = select_next_programme(direction=1).programme_title
    assert new_title

    assert select_next_programme(direction=-1).programme_title == original_title

So: the look of your device is captured by frame objects and the behaviour of the device is captured by helper functions and page objects. This works out very well; in our experience the look of a UI will tend to change a lot during development, whereas the underlying behaviour changes at a much slower rate.

Frame Objects checklist:

1. Take a frame in the constructor.
2. Provide properties describing the contents of that frame.
3. Contain no actions or behaviours.
4. Include a __repr__ method that prints as much data from the frame as possible to enable use with doctests and logging.
5. Include a __nonzero__ method dictating whether the current frame is suitable – for use with stbt.wait_until.

We’re considering adding specific support for Frame Objects in stb-tester to help reduce the amount of boilerplate required.

Update 2016-03-10:

A FrameObject base class was merged to stb-tester git master and will be in the stb-tester v25 release. It reduces the boilerplate involved in Frame Objects by taking a frame in the constructor, and defining __repr__ and __nonzero__ for you. So now all you have to do is derive from stbt.FrameObject and define some properties. See PR #332 for more information.

Discuss this on our mailing list

Footnotes:

Here’s what a test-case can look like when applying the page-object pattern:

def test_that_bbc_one_shows_the_one_show_at_7pm():
    guide = open_guide()
    guide.enter_channel(101)
    guide.scroll_to(time='19:00')
    assert guide.find_selected_programme() == "The One Show"

and this is what it can look like without:

def test_that_bbc_one_shows_the_one_show_at_7pm():
    stbt.press('KEY_GUIDE')
    assert stbt.wait_until(lambda: stbt.match('guide-header.png'))

    stbt.press('KEY_1')
    stbt.press('KEY_0')
    stbt.press('KEY_1')
    time.sleep(3)

    while stbt.ocr(region=stbt.Region(213, 10, 200, 36)) < "19:00":
        stbt.press('KEY_RIGHT')
        time.sleep(1)

    assert stbt.ocr(region=stbt.Region(213, 23, 200, 36)) == "The One Show"

The improvement in understandability is clear. Without the page-object, the meaning of your test-case is buried in the forest of calls to stbt.match, ocr, press and wait_until. Using the page-object pattern means that your test-case consists of high level concepts that expose the intent of the test-case. Notice that with the page-object pattern calls to stbt functions almost never appear in test-cases directly.

In this example the page-object is guide. This is an instance of a class that knows how to understand and navigate your set-top-box’s programme guide.

The class might look like this:

class Guide(object):
    def find_selected_programme(self):
        return stbt.ocr(region=stbt.Region(213, 23, 200, 36))

    def read_lcn(self):
        return stbt.ocr(region=stbt.Region(100, 20, 40, 20))

    def enter_channel(self, lcn):
        for x in "%03d" % lcn:
            stbt.press('KEY_%s' % x)
        assert stbt.wait_until(lambda: int(self.read_lcn()) == lcn)

    def read_programme_time(self):
        return stbt.ocr(region=stbt.Region(213, 10, 200, 36))

    def scroll_to(self, time):
        for _ in range(100):
            orig_time = self.read_programme_time()
            if orig_time == time:
                return
            stbt.press('KEY_RIGHT')
            wait_until(lambda: self.read_programme_time() != orig_time)
        else:
            assert False


def open_guide():
    stbt.press('KEY_GUIDE')
    assert stbt.wait_until(lambda: stbt.match('guide-header.png'))
    return Guide()

The class is used by many test-cases, which increases code reuse. It serves as a central place to handle any known quirks of your UI, and as a single place to update if your UI changes.

If you need to test multiple similar (but slightly different) variants of your UI with a single test-pack, this class is the place to abstract over those differences.

Note that I made open_guide a standalone function rather than a method or static method of class Guide. I don’t consider navigating to the guide to belong to the Guide page-object. Rather it belongs to the app as a whole. The rule of thumb is that the methods on the Guide class are only valid to call if the guide is currently visible.

So, using the page-object pattern:

• Improves the readability of your test-cases.
• Reduces the maintenance cost of your test-pack by sharing (and thus reducing) test code and providing a central place to apply fixes and updates.
• Assists with reusing test-cases for testing multiple variants of a single UI.

In next week’s blog post we will discuss an extension to this approach called the “Frame Object Pattern” which can improve the maintainability of your test scripts even further.

Discuss this on our mailing list

This is part 1 of a 2 part blog post. It focuses on how stb-tester uses LIRC to produce IR signals and how recording works in theory. Part 2 will apply this understanding to successfully record remote control codes where irrecord fails by default.

Overview of stb-tester and IR

stb-tester uses LIRC (Linux Infrared Remote Control) to send IR signals to control the set-top-boxes and smart-tvs under test. The test scripts refer to key names like “KEY_0” and “MENU”, but the IR blaster has to send out the pulses of IR light that the set-top-box/TV will understand.

LIRC does this conversion from key names to pulses based on a configuration file you provide it that describes the IR protocol to be used and what the valid key values are. You can use the lirc command irrecord to create a configuration file by pressing buttons on the remote.

Unfortunately this doesn’t always work perfectly. To see why let’s look a little more in detail at how lirc calculates what pulses to send:

The codes section of the config file describes how to map a key name (such as “KEY_1” or “KEY_MENU” to a number. This number will be encoded into a series of pulses and spaces. In the case in the diagram above 1s are encoded as long pulses and 0s as short pulses.

Recording IR codes

irrecord works in reverse: it will record the signals from your remote control and create a config file for lircd:

1. It first tries to derive the encoding parameters and the codes by getting the user to press lots of different buttons and analysing the pulse/space timings.
2. It then works out the code mapping by getting the user to enter the name of a button and then pressing the button on the remote.

Creating the code mapping is very easy to do once you know the encoding, but automatically working out the encoding parameters is more difficult and so can be unreliable. When this happens irrecord may fail to produce a config file or the config file it does produce may be ineffective.

irrecord will often succeed where it had failed before if you give it a helping-hand by deriving the encoder parameters for it. In the next blog post I describe how I determined the encoding parameters for a SONY Bravia smart TV remote control, and thus was able to create a full lirc config for this remote control.

References:

• WinLIRC Configuration File Format

stb-tester uses the open-source tesseract engine for OCR. This works really well but is not perfect. Tesseract was was primarily designed to operate on text which had been printed and then scanned. This is broadly the same but slightly different to OCR on the screen:

Over the years tesseract has evolved techniques for dealing with each of the problems listed on the left. According to my experiments it is not yet perfectly adapted for dealing with the issues on the right.

Measurement

But first: If we want to improve something, first we must be able to measure it. Fortunately the YouView UI which I’m using for this contains a 6163 word Terms and Conditions and Privacy Policy which is also available online. This gives me a basis for comparison. I can then OCR the screens and compare against the ideal with the word diff tool dwdiff. This ignores formatting differences and will print statistics to measure how good a job we’ve done:

old: 6278 words  6046 96% common  0 0% deleted  232 3% changed
new: 6199 words  6046 97% common  0 0% inserted 153 2% changed

For this case the number included in the table below would be 232/6278. e.g. 232 of the words of the original text were not correctly recognised.

The test script that I used to generate these results can be found on a branch on github.

Approaches to improve accuracy

Training

• Training Tesseract 3 - Tesseract wiki

The theory is that by training tesseract on the fonts you’re using in your specific UI stb-tester will be able to do a better job at recognising text. It turns out that this isn’t true. Training on the specific font used in the YouView UI had no beneficial effect, in fact the opposite turned out to be the case. We hypothesise that a lot of effort has gone into the eng.trainneddata file that tesseract ships with to make it work well with the majority of fonts and text so training with a specific font has little effect.

Training turns out to be much trickier than you might expect - despite the tesseract wiki proclaiming “And that’s all there is to it” after 12 pages of instructions. Eventually I was able to write a script to automate these instructions. As a point of interest you can find this on my train-tesseract branch on github. Instructions are included there. As training was ineffective I’ve no intention to merge this to stb-tester master at this time.

Result: No useful effect once other improvements are made, harmful in some cases.

Remove ligatures

Ligatures are when two letters are combined into a single glyph. e.g. fi might be rendered as the single glyph (and unicode codepoint) fi. This was used in traditional movable type typesetting to make typesetting easier and look better.

This is not really relevant for our case as we are interested in the content, not the rendering. I’ve tried two approaches to this:

• Tell tesseract to not recognise ligatures with -c tessedit_char_blacklist=fffiflffifflſtst. This is “+noligatures” below.
• Expand ligatures after tesseract has recognised them. This is “+replaceligatures” below.

Result: Replacing ligatures is superior to blacklisting them. Results in an improvement of 0.4%.

Normalise Punctuation

Unicode contains a whole bunch of punctuation which all look very similar. We consider for the purposes of set-top box testing our users are unlikely to be interested in the differences between a hyphen (‐), a minus sign (−) and any of the four types of unicode dash (‒, –, —, ―). Nor will there be much interest in the differences between the various types of quotation marks.

This approach cheats slightly as, strictly speaking, to match against some known text you need to additionally normalise the expected text. When this is merged to master stbt will provide an API for doing this.

Result: Positive effect reducing error rate by ~0.3%

Scale text up

Tesseract expects high resolution (300dpi) scans of text with no anti-aliasing. Screen text on the other hand tends to be relatively low resolution (72dpi) and anti-aliased. tesseract’s first step “thresholds” the image - turning the image into 1-bit monochrome black and white binary image. This throws away any information about the shapes of the letters provided by anti-aliasing before performing outline recognition.

Ideally we would patch tesseract to directly take anti-aliasing into account during outline recognition, but instead it seem that just scaling the image up before thresholding allows this information to be preserved until the outline extraction step.

Result: This provides the bulk of the improvement in error rate, bringing it down from 8.3% to 1.2%.

There are many image scaling algorithms. I tested against all the algorithms available in imagemagick at 2×, 3× and 4× scaling factor:

• Bartlett
• Blackman
• Bohman
• Box
• Catrom
• Cosine
• Cubic
• Gaussian
• Hamming
• Hanning
• Hermite
• Jinc
• Kaiser
• Lagrange
• Lanczos
• LanczosSharp
• Lanczos2
• Lanczos2Sharp
• Mitchell
• Parzen
• Point
• Quadratic
• Robidoux
• RobidouxSharp
• Sinc
• SincFast
• Spline
• Triangle
• Welsh

To cut a long story short the best algorithm is “Triangle” at 3× scaling. This turns out to be bilinear scaling which conveniently is the default interpolation mode used by OpenCV’s resize function (cv2.INTER_LINEAR).

3× worked better than 2× or 4× scaling. My theory is that this is because each pixel becomes 9 pixels (3×3) the sampling point in the middle can stay the same. This theory is unsubstantiated speculation however. Perhaps someone with a greater understanding of re-sampling can explain this?

Sharpening

The theory is that h264 encoding (such as performed by the TeradeK VidiU or the Hauppauge HD-PVR) causes some blurring. The sharpening is intended to make the edges of the words sharper in the hope that tesseract’s thresholding algorithm will then be able to do a better job.

Result: This had a small negative effect.

Results

With scaling and ligature and punctuation normalisation the error rate drops from 8.3% to 0.65%. This is a improvement of 12×. An extract of the full results is included below:

Full results with example output can be found here.

Admittedly this is all a little unscientific. Further testing with different text would be required to estimate error bars on the above measurements but I believe this is conclusive enough to justify the improvements to stb-tester.

Thanks

Thanks to YouView for sponsoring this effort.

Further reading:

• An Overview of the Tesseract OCR Engine - Ray Smith
• Training Tesseract 3 - Tesseract wiki
• Improving the quality of the output - Tesseract wiki
• Tesseract OCR Engine - What it is, where it came from, where it is going - Ray Smith - presented at OSCON 2007.

For a write up of GTAC 2013 see Dave’s article on LWN.