Python API Release Notes#

Changes to the stbt core Python API are version-controlled. You can specify the version you want to use in your .stbt.conf file. See test_pack.stbt_version in the Configuration Reference.

We generally expect that upgrading to new versions will be safe; we strive to maintain backwards compatibility. But there may be some edge cases that affect some users, so this mechanism allows you to upgrade in a controlled manner, and to test the upgrade on a branch first.

v34#

14 June 2023

Changes in behaviour since v33:

  • Tests are run using pytest. Pytest provides many useful features, including detailed information about failing assertions, and fixtures for re-usable setup and teardown steps. In v33, pytest was optional; in v34 pytest is the default (and only) test-runner. To learn more about pytest see New features in v33: Pytest support.

  • stbt.load_image, stbt.match: Normalize the alpha (transparency) channel when loading numpy arrays, in the same way we normalize the alpha channel of a PNG image. Calling load_image or match with a numpy array now behaves the same as calling it with a PNG filename.

  • stbt.match: Don’t ignore the reference image’s alpha (transparency) channel if frame is a single-channel (grayscale) image. Previously the reference image was converted to grayscale before matching; now the frame is converted to color before matching. This is unlikely to affect any existing tests as grayscale frames are only created as a result of custom image processing.

  • stbt.wait_for_motion, stbt.detect_motion, stbt.press_and_wait, stbt.wait_for_transition_to_end: Make BGRDiff the default diffing algorithm.

    The previous default (GrayscaleDiff) doesn’t detect changes that are different in color but of a similar overall intensity, such as the blue (focused) vs. white (unfocused) letters in this screenshot of the Xfinity on-screen keyboard.

    Note that the noise_threshold of BGRDiff is different in scale (it’s now 0-255 instead of 0.0-1.0) and in direction (a smaller number means a stricter threshold). BGRDiff’s default threshold is 25, so it’s slightly stricter (GrayscaleDiff’s threshold of 0.84 would correspond to roughly a value of 40 in the BGRDiff algorithm).

    To continue using the previous algorithm, run the following code early in your test script (for example at the top level of tests/__init__.py):

    stbt.detect_motion.differ = stbt.GrayscaleDiff()
stbt.press_and_wait.differ = stbt.GrayscaleDiff()

    Note that the above 2 lines also affect wait_for_motion and wait_for_transition_to_end.

Other new features and fixes:

  • New Smart Navigation functions stbt.navigate_1d and stbt.navigate_grid.

  • New integration with pyatv for sending commands to Apple TV devices over the network. See Apple TV for details.

  • stbt.Mask.from_alpha_channel: New constructor for creating a Mask from a PNG image’s alpha (transparency) channel. Useful when you have a reference image with some transparent parts, but you want to use that same mask for an operation other than match.

  • stbt.Mask.to_array: If adding or subtracting masks results in an empty region, log a warning. Useful for debugging issues with the coordinates in your test script.

  • stbt.press_and_wait, stbt.wait_for_transition_to_end: Add frames parameter (consistent with wait_for_motion). This allows performing custom transformations on the video (such as removing background behind a translucent overlay) before it’s processed by the motion detection of press_and_wait.

  • stbt.press_and_wait: Add retries parameter. If the keypress had no effect at all, retry this number of times. Defaults to 0, so you have to opt-in to this behaviour by specifying, for example, retries=1 or retries=2.

  • stbt.wait_until: Log how long the wait_until call took until it succeeded.

v33#

13 July 2022

Major new features:

  • v33 has a new test-runner environment based on Ubuntu 22.04 and Python 3.10. The most notable changes are:

    Software

    v32

    v33

    Ubuntu

    18.04

    22.04

    Python

    3.6

    3.10

    Tesseract (OCR engine)

    4.00~git2288-10f4998a-2

    4.1.1

    OpenCV

    3.2

    4.5

    Pylint

    1.8

    2.14

    Pytest

    3.3.2

    6.2.5

    ADB

    8.1

    29.0

    Any third-party packages that you install using apt in your setup script may be upgraded to a newer version too.

  • Python 2 is not supported. Customers who are still using Python 2 should continue using v32 or earlier. To upgrade see our guide: Porting a test-pack to Python 3.

  • Tests can optionally be run using pytest. Pytest provides many useful features, including detailed information about failing assertions, and fixtures for re-usable setup and teardown steps. For more details see New features in v33: Pytest support.

  • BDD-style test scripts are supported, using the pytest-bdd API. For more details see New features in v33: BDD support.

  • New Mask API to construct masks from regions. You can add, subtract or invert regions to construct a mask. This is much more convenient than creating a mask PNG in an image editor. See Regions and Masks for examples.

    Any API with a mask parameter can take a single Region, or a complex mask constructed from Regions as described above, or a filename of a black-and-white PNG image. Previously, the mask parameter could only take a filename.

  • Any API that takes a color can take a web-style “#rrggbb” string (for example "#f76600"). Previously colors had to be specified as a (blue, green, red) tuple of ints. Now both formats are accepted.

  • stbt.find_regions_by_color: New API to find any GUI element of a solid color (for example a button, or a “focus” or “selection” border).

  • stbt.segment: New API to find distinct foreground elements (such as lines of text). It’s pronounced like the verb (segMENT) not the noun.

  • New stbt.android API for interacting with AndroidTV devices using ADB.

  • New stbt.Roku API for interacting with Roku devices over the network using Roku’s External Control Protocol.

  • stbt.find_file: New API to resolve a filename relative to the python file that’s calling it. It uses the same search algorithm as stbt.load_image, but it works with any type of file.

  • stbt.MultiPress: New API to enter text using a numeric keypad using the ITU-T E.161 mapping.

Changes in behaviour since v32:

  • Dropped support for Python 2.

  • Debug logging from stbt APIs uses Python’s logging framework. Each debug line now starts with the logger name and logging level (namely “DEBUG:stbt:”).

  • Reworked stbt.android API. Previously this was a private, undocumented API in _stbt.android but a small number of customers were already using it. The changes are:

    • Changed order & defaults of AdbDevice constructor arguments to make it more suitable for Android TV devices (as opposed to mobile devices).

    • Made AdbDevice.adb more like subprocess.run.

    • Support ADB host key committed to test-pack, to support stateless test-runners. From the root of your test-pack (git checkout) run adb keygen config/android/adbkey and commit the config/android directory to git. Warning: Anyone with read access to your test-pack git repository and with network access to your AndroidTV devices will be able to control them using ADB.

  • stbt.find_selection_from_background: The region attribute of the return value is set even if it failed the max_size / min_size checks. Previously region would have been None.

    If you have code like this:

    @property
def selection(self):
    return stbt.find_selection_from_background(...).region  # Wrong in v33!

    …rewrite it to this:

    @property
def selection(self):
    sel = stbt.find_selection_from_background(...)
    if sel:
        return sel.region

  • stbt.Keyboard: Removed the first parameter of the constructor. Since v32 it would raise an exception if you passed this parameter, so nobody should be using it by now. All the remaining parameters must be specified by keyword.

  • stbt.load_image: The returned image is read only; call its copy() method to make a writeable copy if you need to modify it.

Deprecated APIs:

Minor additions, bugfixes & improvements:

  • stbt.Frame and stbt.Image:

    • Add width, height, and region properties.

    • Fix IndexError in __repr__ when the Frame or Image has undergone numpy operations that change its shape (such as numpy.max, which will preserve the stbt.Image or stbt.Frame type of its argument).

  • stbt.FrameObject:

    • The object’s __repr__ only prints the values of properties that have already been calculated. That is, it doesn’t trigger evaluation of all public properties.

    • The __repr__ shows the object’s frame (self._frame) so you can see the timestamp of the frame associated with each FrameObject instance.

    • Fix comparison operators (== and !=). Previously they would raise TypeError if either operand had a property that returned None; or they could return the wrong result if comparing an instance of a class F against an instance of F’s subclass.

    • Remove ordering operators (<, etc). They were buggy and there’s no use-case for ordering Page Object instances.

  • stbt.RmsVolumeResult: Added dBov method to convert the RMS amplitude to decibels.

  • stbt.Keyboard:

    • Added type stbt.Keyboard.Key: This is the type returned from Keyboard.find_key. Previously it was an opaque, private type; now it is a public, documented API. This is so that you can use it in type annotations for your FrameObject properties.

    • Better support for slow/laggy keyboards:

      • Recover from missed or double keypresses by re-calculating the path from the current state of the device-under-test. To disable this behaviour specify retries=0 when calling Keyboard.enter_text or Keyboard.navigate_to. retries defaults to 2.

      • Increased default navigate_timeout from 20 to 60.

      • Wait longer for the selection to reach the final target when we’re not verifying every keypress.

    • Better error message when user’s Page Object’s selection property returns None (it’s a bug in your Page Object if it says is_visible==True but selection==None).

  • stbt.load_image:

    • Cache the last 5 loaded images. This will avoid repeating the same PNG decoding for every frame when you do something like stbt.wait_until(lambda: stbt.match("reference.png")).

    • New color_channels parameter, replacing flags which is now deprecated.

    • Raise FileNotFoundError with the correct errno, instead of IOError without an errno. Note that FileNotFoundError is a subclass of IOError.

    • Normalize the alpha channel (if any) so that each pixel is either fully transparent (0) or fully opaque (255). Previously this normalization was done in stbt.match.

  • stbt.match: Fixed the position of the region drawn in the Object Repository “Debug” tool.

  • stbt.ocr: corrections parameter: Fix matching non-word characters at word boundaries.

  • stbt.press: The key argument can be an Enum (press will use the Enum’s value, which must be a string).

  • stbt.press_and_wait: The return value has new started, complete, and stable attributes. This is often clearer than checking the value of the status attribute:

    transition = stbt.press_and_wait("KEY_OK")
if not transition.started:
    ...
# versus:
# if transition.status == stbt.TransitionStatus.START_TIMEOUT:

  • stbt.Size: New helper type. It’s a tuple with width and height.

  • pylint plugin: Increase Astroid’s inference limit to fix various false positives.

v32#

1 October 2020.

Warning

When upgrading the stb-tester package to v32 locally with pip (for your IDE) please follow these steps to uninstall the previous version first:

pip uninstall stb-tester stbt-premium-stubs stbt-extra-stubs
pip install stb-tester

Major new features:

  • stbt.Keyboard: Support keyboards with multiple modes (for example lowercase, uppercase, and symbols).

  • stbt.find_selection_from_background: New function to detect if a page is visible, and simultaneously find the position of the current “selection” or “highlight” on the page.

  • stbt.ocr:

    • Calls to Tesseract are cached if all the parameters are identical (including all the pixels in the frame & region). This cache is persisted on disk between test-jobs. This can greatly speed up calls to ocr when reading common text, for example when navigating menus.

    • New corrections parameter: A dict of {bad: good} mappings to correct known OCR mistakes.

    • New function stbt.apply_ocr_corrections to apply the same corrections to any string — useful for post-processing old test artifacts using new corrections.

    • New function stbt.set_global_ocr_corrections to specify the default value for ocr’s corrections parameter. Call this early in your tests, for example in the top-level of tests/__init__.py.

  • stbt.press_and_wait: New parameter min_size to ignore motion in small regions (useful when you can’t predict the exact position of those regions by specifying a mask).

  • stbt.Region:

  • stbt.detect_pages: New function to find the Page Objects that are relevant for the current video frame.

  • stbt.last_keypress: New function that returns information about the last key-press sent to the device under test.

  • stbt.stop_job: New function to stop a job of multiple testcases or a soak-test.

  • Pylint plugin new checker: Check that the return value from FrameObject.refresh is used (FrameObjects are immutable, so refresh() returns a new object instead of modifying the object it’s called on).

Changes in behaviour since v31:

  • stbt.crop: Implicitly clamp at the edges of the frame, if the region extends beyond the frame. Previously, this would have raised an exception. It still raises ValueError if the region is entirely outside of the frame.

  • stbt.draw_text: Also write text to stderr.

  • stbt.get_config: Allow None as a default value.

  • stbt.is_screen_black: Increase default threshold from 10 to 20.

  • stbt.Keyboard:

    • Changed the internal representation of the Directed Graph. Manipulating the networkx graph directly is no longer supported.

    • Removed Keyboard.parse_edgelist and grid_to_navigation_graph. Instead, first create the Keyboard object, and then use its add_key, add_transition, add_edgelist, and add_grid methods to build the model of the keyboard.

    • Removed the Keyboard.Selection type. Instead, your Page Object’s selection property should return a Key value obtained from Keyboard.find_key.

    • The edgelist format now allows key names with “#” in them. Previously anything starting with “#” was treated as a comment. Now comments are lines starting with “###” (three hashes), optionally preceded by whitespace.

    • Keyboard.enter_text adds a short inter-press delay when entering the same letter twice, because some keyboard implementations ignore the second keypress if pressed too quickly.

  • stbt.load_image:

    • Fix UnicodeDecodeError when filename is utf8-encoded bytes.

    • Allow passing a numpy array.

    • Return type changed from numpy.ndarray to stbt.Image, which is a sub-class of numpy.ndarray with the additional attributes filename, relative_filename, and absolute_filename.

  • stbt.match: Disable the “pyramid” performance optimisation if the reference image has too few non-transparent pixels. This fixes false negatives when the reference image is mostly transparent (for example a thin border of opaque pixels around a large transparent centre).

  • stbt.MatchResult (the return value from stbt.match): The image attribute is now an instance of stbt.Image. Previously it was a string or a numpy array, depending on what you had passed to stbt.match.

  • stbt.ocr and stbt.match_text: If region is entirely outside the frame, raise ValueError instead of returning an empty string. (This is likely to be an error in your test-script’s logic, not desired behaviour.) This is now consistent with all the other image-processing APIs such as stbt.match.

  • stbt.press_and_wait:

    • Now uses the same difference-detection algorithm as stbt.wait_for_motion. This algorithm is more tolerant of small noise-like differences (less than 3 pixels wide). To use the previous algorithm, run the following code early in your test script (for example at the top level of tests/__init__.py):

      stbt.press_and_wait.differ = stbt.StrictDiff

    • If you were passing a numpy array for the mask parameter, now it needs to be a single-channel image (grayscale) not a 3-channel image (BGR). (But if you were passing the mask as the filename of an image on disk, you don’t need to change anything.)

v31#

19 September 2019.

Major new features:

  • Supports test-scripts written in Python 3. Python 2 is still supported, too. When upgrading your test-pack to v31 you will need to specify a Python version in your test-pack’s .stbt.conf file like this:

    [test_pack]
stbt_version = 31
python_version = 3

    Valid values are 2 or 3. The test-run environment is Ubuntu 18.04, so you get Python 2.7 or 3.6.

    We recommend Python 3 for all new test-packs. For large existing test-packs we will continue to support Python 2 until all our large customers have migrated.

  • stbt.Keyboard: New API for testing & navigating on-screen keyboards.

  • stbt.Grid: New API for describing grid-like layouts.

Minor additions, bugfixes & improvements:

  • stbt.match: Fix false negative when using MatchMethod.SQDIFF and a reference image that is mostly transparent except around the edges (for example to find a “highlight” or “selection” around some dynamic content).

  • stbt.match: Improve error message when you give it an explicit region that is smaller than the reference image.

  • stbt.ocr: New parameter char_whitelist. Useful when you’re reading text of a specific format, like the time from a clock, a serial number, or a passcode.

  • stbt.press_and_wait: Ignore small moiré-like differences between frames (temporal dithering?) seen with Apple TV.

  • stbt.press_and_wait: Draw motion bounding-box on output video (similar to stbt.wait_for_motion).

  • stbt.press_and_wait: Add key attribute (the name of the key that was pressed) to the return value.

  • stbt.Region: The static methods intersect and bounding_box will fail if called on an instance. That is, instead of calling self.intersect(other) you must call stbt.Region.intersect(self, other). Previously, if called on an instance it would silently return a wrong value.

  • stbt.wait_for_motion: More sensitive to slow motion (such as a slow fade to black) by comparing against the last frame since significant differences were seen, instead of always comparing against the previous frame.

  • stbt lint improvements:

    • New checker stbt-frame-object-get-frame: FrameObject properties must use self._frame, not stbt.get_frame().

    • New checker stbt-frame-object-property-press: FrameObject properties must not have side-effects that change the state of the device-under-test by calling stbt.press() or stbt.press_and_wait().

    • New checker stbt-assert-true: “assert True” has no effect.

    • Teach pylint that assert False is the same as raise AssertionError. This fixes incorrect behaviour of pylint’s “unreachable code” and “inconsistent return statements” checkers.

v30#

25 February 2019.

Minimum version supported by the Stb-tester Platform.