Release notes

v28.1

22 February 2018

The following new features will only be enabled if you set test_pack.stbt_version=28 in .stbt.conf:

Breaking changes:

  • Passing region=None to stbt.ocr raises a TypeError instead of printing a deprecation warning. Use region=stbt.Region.ALL instead.

  • Passing type_=bool to stbt.get_config now returns False for values of “0”, “false”, “off”, and “no” (ignoring case). Previously it would always return True for any non-empty value.

  • stbt.get_frame and stbt.frames now return read-only frames for better performance. Use frame.copy() to get a writeable copy of the frame.

  • stbt.get_frame is no longer guaranteed to return a new frame; it may return the same frame that the previous call to stbt.get_frame() returned. This may have subtle effects on the timing of existing test-scripts. Functions that depend on this behaviour should be refactored to use the stbt.frames iterator function instead.

    The benefit is that you can now call stbt.get_frame() from multiple threads. Also, code like wait_until(lambda: match('a.png') or match('b.png')) can run faster as the second match will no longer block waiting for a new frame.

    If this change causes you problems, you can add the following to your .stbt.conf file to restore the old behaviour:

    [global]
    use_old_threading_behaviour = true
    

New features:

  • Multithreading support: stbt functions like wait_for_match can now be used from multiple threads simultaneously. Each call to stbt.frames returns an independent iterator that can be used concurrently.
  • New function stbt.crop to crop a region from a video-frame.
  • New function stbt.load_image to load an image from disk, using the same path lookup algorithm used by wait_for_match and friends.
  • ocr and match_text have a new parameter text_color. Specifying this can improve OCR results when tesseract’s default thresholding algorithm doesn’t detect the text, for example for light-colored text or text on a translucent overlay.
  • The pre-processing performed by ocr and match_text can now be disabled by passing upscale=False. This is useful if you want to do your own pre-processing.
  • The default lang (language) parameter to ocr and match_text is now configurable. Set lang in the [ocr] section of .stbt.conf.
  • press_until_match, detect_motion, wait_for_motion, and is_screen_black can take a new region parameter.
  • The mask parameter to detect_motion, wait_for_motion, and is_screen_black can be an OpenCV image (previously it could only be a filename). This makes it easier to construct masks programmatically using numpy.
  • The MotionResult object returned by detect_motion and wait_for_motion includes the video-frame that was analysed (in the new frame attribute). This allows you to perform additional analysis – for example if there was no motion is the frame black?
  • wait_until has two new parameters: predicate and stable_secs. Together they allow waiting for something to stabilise (for example to wait for the position of a moving selection to stop moving).
  • wait_until will try one last time after the timeout is reached. This allows you to use a short timeout with operations that can take a long time.
  • The is_visible property of stbt.FrameObject subclasses can now call other public properties. Furthermore, the value of is_visible is always a bool, so you don’t have to remember to cast it to bool in your subclass’s implementation.
  • The video recording in the test-results now runs at the full frame-rate rather than slowing down during wait_for_match. As a side-effect, if the image processing is particularly slow the annotations won’t appear on the output video. Apart from that caveat, annotations now appear if you got the frame using stbt.get_frame (previously the annotations only appeared if the frames came from a stbt.frames iterator). “Annotations” means the yellow & red rectangles showing the current match or motion region, or text added with draw_text.

v27.4

24 February 2017

  • Managing large test farms: You can now override configuration for individual stb-tester devices in your test farm. See Configuration files for details.
  • Managing large test farms: New test_pack.default_remote_control configuration setting allows you to specify the default remote control for each stb-tester device in your test farm.
  • REST API: New endpoint /api/v1/results which allows retrieving test-results by any search expression supported in the web interface (previously you could only retrieve results for a specific job).
  • Android: Support android control over TCP/IP.
  • Bugfix: The Job category can now contain unicode characters.

v27.3

23 January 2017

  • Add support for controlling Android mobile phones, for selected partners. Contact support@stb-tester.com for details.

  • Add support for controlling devices that don’t have an infrared receiver, such as the Sony PlayStation 4, via HDMI CEC. See here for instructions.

  • Add support for the latest stb-tester Python API v27.

  • Web UI: Show X and Y coordinates when you move the mouse over the live video. This is useful when you need to define a Region in your test script.

  • Web UI: Fix rendering issue that caused some remote control SVG images to be cropped rather than scaled.

  • images/v27.3-job-logs.png

    Web UI: Add link to job logs. This is useful to debug issues when a test-job fails to start (for example because you forgot to plug in the infrared transmitter, or because your setup script failed).

The following new features will only be enabled if you set test_pack.stbt_version=27 in .stbt.conf:

  • The pip Python package manager is installed by default in the test-run environment. If you use a setup script to install third-party packages with pip install, you no longer need to install pip with apt-get install python-pip.
  • The version of pylint (a Python static analysis tool) installed in the test-run environment is upgraded to version 1.6.4 (from 1.1.0). If you run pylint or stbt lint with stbt-docker, you may need to update your pylint configuration file.
  • Python API: stbt.Region has the following new methods: above, below, right_of and left_of. They return a new Region relative to the current region.
  • Python API: stbt.match_text can take single-channel images (black-and-white or greyscale).
  • Python API: stbt.match_text normalises punctuation such as em-dash and en-dash, just like stbt.ocr already does.
  • Python API: stbt.match_text has a new parameter case_sensitive. It defaults to False (that is, ignore case), which was the previous behaviour.

v26.6

4 October 2016

  • The stb-tester ONE will delete the oldest test results when its disk runs out of inodes. This can happen if your test-cases write many small test-artifact files. Note that the stb-tester ONE already monitored disk space, just not the inode usage.

  • Web UI: Add option to view JPEG screenshots instead of the Flash-based video player. The screenshot is updated once per second. It is a low quality thumbnail to keep the bandwidth low.

    This is useful when you don’t have Flash installed, or when you are accessing the stb-tester device through a firewall that blocks Flash video, or over a network that doesn’t have sufficient bandwidth.

v26.2

22 July 2016

  • New features for managing large test farms:

    • Stb-tester devices on your network auto-discover each other using Zeroconf.
    • Web UI: New drop-down menu that shows other stb-tester devices on your network, allowing you to quickly switch between your devices-under-test.
    • Web UI: New monitoring view (accessed from the above drop-down menu) that shows the live video from all the devices in your test farm.

    For more details see Large test farms in the User Interface reference.

  • Web UI: Scale the remote control image if it is too large.

v26.1

13 July 2016

  • Add support for frame-accurate performance measurements. See the new stbt APIs below for retrieving the timestamp attached to each frame.

    See How we test the stb-tester ONE: Video-capture latency & accuracy for details on the precision provided by the stb-tester ONE hardware.

  • Add support for the latest stb-tester Python API v26.

The following new features will only be enabled if you set test_pack.stbt_version=26 in .stbt.conf:

  • stbt.get_frame and stbt.frames now return a stbt.Frame instead of a numpy.ndarray. stbt.Frame is a subclass of numpy.ndarray with an additional attribute: time. This is the time at which the video-frame was captured, as a floating point number expressed in seconds since the epoch, so you can compare it against Python’s time.time().

    Note that stbt.frames yields pairs of frame, timestamp. The timestamp element of that pair is now deprecated (it’s a number in nanoseconds not seconds). Use the frame’s time attribute instead.

  • stbt.MatchResult, stbt.TextMatchResult, and stbt.MotionResult have a new time attribute. This is the time at which the matching video-frame was captured. The timestamp attribute of these classes is now deprecated. See the previous bullet point for details.

  • stbt.wait_for_motion reports the time when the motion started, not the time after consecutive_frames of motion have elapsed.

  • stbt.MotionResult now defines __nonzero__. This means you can write if result: rather than having to write if result.motion. This is a minor ergonomic improvement for consistency with MatchResult and TextMatchResult.

  • stbt.MotionResult has a new region attribute which indicates where in the frame the motion was detected.

  • The way motion detection is visualised on the test-result video recording has changed. Instead of colouring the in-motion areas red we draw a red rectangle around the area with motion. This is consistent with the match visualisation.

v25.1

3 June 2016

Add support for the latest stb-tester Python API v25.

The following new features will only be enabled if you set test_pack.stbt_version in .stbt.conf to 25:

  • The frame parameter of the MatchResult returned from stbt.match, stbt.detect_match and stbt.wait_for_match is now read-only. Use frame.copy() if you need a writable copy.
  • New base class stbt.FrameObject for user-defined Frame Objects. The Frame Object pattern simplifies testcase development and maintenance; Frame Objects are a layer of abstraction between your testcases and the stbt image processing APIs. See this tutorial for details.
  • New function stbt.match_all that searches for all instances of a reference image within a single video frame. It returns an iterator of zero or more stbt.MatchResult objects (one for each position in the frame where the reference image matches).
  • Bugfix: When match and wait_for_match can’t find the specified reference image, the error message now says the actual path & filename you specified (that is, a path relative to the test script), not an absolute path under the current working directory.
  • match_text now logs a line saying whether it found a match or not, much like match and wait_for_match already do. Thanks to Rinaldo Merlo for the patch.

v24.8

27 May 2016

  • Fix regression introduced in v24.7 that caused the stb-tester ONE’s old USB infrared transmitter to fail for signals longer than 125 milliseconds.

v24.7

26 May 2016

  • Add support for new infrared transmitter hardware.

    A timing fault in the old infrared transmitter hardware would sometimes cause missed keypresses. The failure rate depends on the receiver hardware. Some Roku models were particularly sensitive, missing up to 4% of keypresses (but most devices we tested were under 0.3%).

    The new hardware is completely reliable. We haven’t seen a single missed keypress in over 500,000 presses. We are shipping the new hardware to all our existing customers; you should hear from us within the next few days.

    The old infrared transmitter will continue to work as before.

    How to tell them apart: The new infrared transmitter is a bit larger than the old one (see photo).

    If you’re interested in the technical details of the fault, or how we tested the new hardware, see this blog post.

v24.5

22 April 2016

New features:

  • Web UI: Show image thumbnails (instead of filenames) in the live testcase view when you’re running a test.

Bugfixes:

  • This only affects two devices: stb-tester-one-b8aeed72e364 and stb-tester-one-b8aeed72c28e.

    A bug in our factory provisioning process caused video-capture to be done in the YUY2 colourspace instead of the full BGR24 colourspace on these stb-tester ONE devices. This causes a slight difference in colours captured in screenshots and during testing, compared to other stb-tester ONEs. A user-visible difference is that the affected devices show black as RGB #101010 rather than #000000. These differences are below the default threshold of stbt.match so applying this fix shouldn’t affect the behaviour of any test-cases that use screenshots captured before the fix.

    Stb-tester ONE devices are all intended to be identical so this release offers the affected users the option to fix this behaviour, for consistency with other stb-tester ONEs. As it is a slight change in behaviour we’ve made this opt-in rather than applying the fix automatically. To apply this fix, after installing v24.5 make an HTTP POST to http://<hostname>:8001/software-update/bgr24-video-capture – for example with curl: curl -d "" http://stb-tester-one-b8aeed72e364.local:8001/software-update/bgr24-video-capture. The change will take about 5 minutes and the device will reboot when complete.

v24.4

11 February 2016

New features:

  • Web UI: Added branch selector. If you push your changes to a git branch other than master, now you can view and run your tests from any git branch:

    images/v24.4-branch-selector.png

    This enables git workflows where you try your changes out on branches before merging to master. This should make sharing an stb-tester ONE between test engineers more convenient.

    Note: This feature was already available through the REST API since v14.

Bugfixes:

  • The infrared transmitter can now handle signals longer than 125 milliseconds.
  • The “Stop” button would occasionally fail to stop a running test job, especially if the testcases were doing a lot of OCR at the time. This was caused by an interaction between docker and a bug in the AUFS driver in the ubuntu kernel.
  • Web UI: Fix links to results with a “&” or “=” in the “job category” name.

v24.3

20 January 2016

New features:

  • Manual control: Show feedback & logs when pressing a button on the remote control doesn’t work (for example, if the lircd.conf file is invalid).
  • You can plug in the IR transmitter after booting the stb-tester ONE (previously the IR transmitter couldn’t be hotplugged). It still needs to be plugged in before you start a test job (that is, before pressing the “Run tests” button in the UI or using the run_tests REST API).
  • REST API: The test_pack_revision parameter to /api/v1/run_tests can be the name of a git branch or tag instead of a git SHA.

Bugfixes:

v24.1

14 January 2016

Add support for the latest stb-tester Python API v24.

The following new features will only be enabled if you set test_pack.stbt_version in .stbt.conf to 24:

  • Python API: New method stbt.Region.replace to set any of the edges of a region to the given coordinates. It is similar to stbt.Region.extend, but it takes absolute coordinates within the image instead of adjusting the edge by a relative number of pixels.
  • Python API: stbt.wait_for_match takes an optional region parameter, just like stbt.match already did.
  • Python API: stbt.match_text now adds the expected text to tesseract’s dictionary, which fixes some false negatives.
  • Python API: Bug fix: stbt.wait_until no longer raises an exception if you passed a functools.partial object (or an instance of a class with a __call__ method) and wait_until timed out. Thanks to Martyn Jarvis for the patch.
  • Python API: You can now raise an exception with unicode in the exception’s message. Previously the testrun’s failure reason would show a “UnicodeEncodeError” instead of your actual exception message.
  • The video-recording of a test run will show the name of the image that stbt.wait_for_match is searching for. Thanks to Lewis Haley for the patch.
  • The screenshot in the results view is more likely to be relevant to the issue seen. Now we save the last frame that the test-script saw; previously we took a new screenshot at some point soon after the test-run had completed.
  • Test runs start faster because the old HTML results view is no longer regenerated before and after each test-run. This is possible since the new test-results interface was introduced in v23.7. The improvement is particularly marked during long soak-tests with thousands of test runs as the old system would scan all the results in the category every time.

v23.7

11 January 2016

  • New test-results interface with much better search capabilities.

    • Interactive search that updates the results as you type. The search language is documented here.
    • Ability to search & display results from multiple job categories at once.
    • This lays the groundwork for future features such as interactive and automated classification of results.
    • The search language is part of our stable public API. This means that the results for a given search string won’t change across stb-tester ONE releases.
    • You can still reach the old static html results view by manually entering the URL http://<stb-tester ONE address>/results/ (for example if your workflow depends on retrieving and processing that HTML – please let us know if this is the case). Note also that the stb-tester ONE provides a results REST API since v22.5.
  • Prettier default image for the remote control in the “manual control” view.

  • Bugfix: Automatically recover from “No video-capture device detected” on some stb-tester ONEs after powering off & on via the front-panel button.

v23.6

16 December 2015

This is a bugfix release.

  • The stb-tester ONE now works if you plug it into a network in the 172.17.x.x range (previously the stb-tester ONE reserved that address range for the containers it uses to isolate your test-runs).
  • If the stb-tester ONE is plugged into a subnet that isn’t in the 172.17.0.0/16 range, but you also have a 172.17.x.x subnet that is routable from the stb-tester ONE’s subnet: Test scripts running on the stb-tester ONE can now contact hosts on the 172.17.x.x subnet.
  • Fix very occasional race condition on boot, where the stb-tester ONE would fail to detect the video-capture hardware so you would see a test pattern in the video instead.

v23.5

9 November 2015

  • Reliability: Repair possible corruption of results database if the stb-tester ONE is shut down uncleanly (for example if there is a sudden power loss).
  • Technology preview: New results view with better search capabilities. For now this isn’t linked directly from the user interface; to see it go to http://<stb-tester ONE address>/app/#/results. For more details see the User Interface Reference.

v23.4

9 September 2015

  • The stb-tester ONE will delete the oldest test results when available disk space is low. Each stb-tester ONE has enough space to store results (including logs, screenshots and videos) for approximately 6 months of continuous testing – less if you collect & store your own logs from the system-under-test.
  • REST API: Bugfix: Fix 404 error from the artifacts endpoint if the result’s job category had spaces in the name.
  • REST API: Bugfix: “job_url” and “result_url” in the responses from various API endpoints now include the correct port number if your stb-tester ONE is behind a port-forwarding firewall where the exposed port is not the default port 80.

v23.3

10 July 2015

  • Add support for the latest stb-tester Python API v23.
  • Fixed bug where no new test jobs could be started after running tests for a week without a reboot.

The following new features will only be enabled if you set test_pack.stbt_version in .stbt.conf to 23:

v22.5

27 June 2015

v22.4

10 June 2015

  • New REST API endpoints /api/v1/jobs/(job_uid) and /api/v1/jobs/(job_uid)/stop. These can be used to integrate an stb-tester ONE with continuous integration systems.

    In the near future we will release further endpoints to retrieve more detailed results from a job (broken down by testcase).

  • Log files are served with the correct encoding, so unicode characters are displayed correctly in the browser.

  • The test-pack git URL is displayed in the stb-tester ONE’s user interface in the “Automated tests” tab. This is for convenience, so that you don’t have to look it up in the user manual.

v22.2

14 May 2015

  • Improved feedback of test results as a job is running:

    images/v22.2-status-display.png
  • Added HOME and TV buttons to the default manual remote control.

  • Fixed issue where IR key-presses were transmitted incorrectly when the stb-tester ONE is under heavy load. This was causing key-presses to be lost.

v22.1

27 March 2015

  • Implement mechanism to apply software updates to the stb-tester core Python API independently of updates to the stb-tester ONE’s user interface. See test_pack.stbt_version in .stbt.conf for details.
  • Fix intermittent incorrect error message during software update.

The following new features will only be enabled if you set test_pack.stbt_version in .stbt.conf to 22:

  • Fix printing unicode strings (such as the result of stbt.ocr) to the test-run log.

  • stbt.MatchResult.image now returns the image name as passed to stbt.match, rather than the absolute path to the image. This is the previously documented behaviour and allows constructs like:

    m = stbt.wait_until(
        lambda: stbt.match("success.png") or stbt.match("error.png"))
    assert m
    if m.image == "error.png":
        recover()
    
  • stbt.is_screen_black no longer requires a frame to be passed in. If one is not specified it will be grabbed from live video, much like stbt.match.

  • Improvements to the test-run results viewer:

    • Failing assert statements in your testcases are now treated as test failures (red), not as test errors (yellow). See Exceptions for more details. Also, the failing assertion is displayed in the test-run’s “failure reason” field.
    • The video recording of each test-run now displays the wall-clock time at the top of each frame. This makes it easier to relate timestamps from the logfiles with the video.
    • The text drawn on the video recording (that shows keypresses, etc.) now uses a more legible font and background.
    • The graph of test-runs now uses a logarithmic y-axis for test duration, so the graph is more useful when there are very long test runs alongside shorter ones.
    • Improve the layout of the results table.
    • Fix a bug in the search function, where it wouldn’t work if there were some particular values in the first row of results.

v18-1

5 March 2015

  • Allow selecting which remote control configuration to use when running tests.
  • Add ability to run soak-tests (run the selected tests forever until you stop them by clicking the “Stop” button).
  • Improve video-capture reliability when the system-under-test changes output framerate.

v17-1

25 Feb 2015

  • Add ability to customise the test run environment.

  • Fix bug where git push with large files would fail with:

    RPC failed; result=22, HTTP code = 413
    fatal: The remote end hung up unexpectedly
    
  • Expose test-job ID to test-pack in JOB_ID environment variable.

  • Fix bug where last job id would not be reported correctly after a reboot.

  • Fix incorrect error message during software update.

v14-1

16 Jan 2015

  • Improve software update usability and feedback.
  • Add button to stop the currently running test job.
  • Add job status banner above live video.
  • Add public REST API.
  • Remove job status debugging text below “Run Tests”.
  • Highlight the currently running line in the testcase Python script.

v13-1

2 Jan 2015

  • Add Manual Remote Control in the Web UI.
  • Improvements to the video capture and streaming.
  • Show current version number in the Web UI.
  • Show disconnection icon (flash) in the Web UI.
  • “Help” link to the User Guide in the Web UI.
  • Don’t stop a test run (of multiple tests) if one test fails.
  • Zeroconf now advertises as an HTTP service so the stb-tester ONE appears in Bonjour browsers on Mac OS X.

v8-1

19 Dec 2014

Initial pilot release to select partners.