22 February 2018
stbt.ocrraises a TypeError instead of printing a deprecation warning. Use
stbt.get_confignow returns False for values of “0”, “false”, “off”, and “no” (ignoring case). Previously it would always return True for any non-empty value.
stbt.get_frameis no longer guaranteed to return a new frame; it may return the same frame that the previous call to
stbt.get_frame()returned. This may have subtle effects on the timing of existing test-scripts. Functions that depend on this behaviour should be refactored to use the
stbt.framesiterator function instead.
The benefit is that you can now call
stbt.get_frame()from multiple threads. Also, code like
wait_until(lambda: match('a.png') or match('b.png'))can run faster as the second
matchwill no longer block waiting for a new frame.
If this change causes you problems, you can add the following to your .stbt.conf file to restore the old behaviour:
[global] use_old_threading_behaviour = true
- Multithreading support: stbt functions like
wait_for_matchcan now be used from multiple threads simultaneously. Each call to
stbt.framesreturns an independent iterator that can be used concurrently.
- New function
stbt.cropto crop a region from a video-frame.
- New function
stbt.load_imageto load an image from disk, using the same path lookup algorithm used by
match_texthave a new parameter
text_color. Specifying this can improve OCR results when tesseract’s default thresholding algorithm doesn’t detect the text, for example for light-colored text or text on a translucent overlay.
- The pre-processing performed by
match_textcan now be disabled by passing
upscale=False. This is useful if you want to do your own pre-processing.
- The default
lang(language) parameter to
match_textis now configurable. Set
[ocr]section of .stbt.conf.
is_screen_blackcan take a new
is_screen_blackcan be an OpenCV image (previously it could only be a filename). This makes it easier to construct masks programmatically using numpy.
MotionResultobject returned by
wait_for_motionincludes the video-frame that was analysed (in the new
frameattribute). This allows you to perform additional analysis – for example if there was no motion is the frame black?
wait_untilhas two new parameters:
stable_secs. Together they allow waiting for something to stabilise (for example to wait for the position of a moving selection to stop moving).
wait_untilwill try one last time after the timeout is reached. This allows you to use a short timeout with operations that can take a long time.
stbt.FrameObjectsubclasses can now call other public properties. Furthermore, the value of
is_visibleis always a bool, so you don’t have to remember to cast it to bool in your subclass’s implementation.
- The video recording in the test-results now runs at the full frame-rate
rather than slowing down during
wait_for_match. As a side-effect, if the image processing is particularly slow the annotations won’t appear on the output video. Apart from that caveat, annotations now appear if you got the frame using
stbt.get_frame(previously the annotations only appeared if the frames came from a
stbt.framesiterator). “Annotations” means the yellow & red rectangles showing the current match or motion region, or text added with
24 February 2017
- Managing large test farms: You can now override configuration for individual stb-tester devices in your test farm. See Configuration files for details.
- Managing large test farms: New test_pack.default_remote_control configuration setting allows you to specify the default remote control for each stb-tester device in your test farm.
- REST API: New endpoint /api/v1/results which allows retrieving test-results by any search expression supported in the web interface (previously you could only retrieve results for a specific job).
- Android: Support android control over TCP/IP.
- Bugfix: The Job category can now contain unicode characters.
23 January 2017
Add support for controlling Android mobile phones, for selected partners. Contact email@example.com for details.
Add support for controlling devices that don’t have an infrared receiver, such as the Sony PlayStation 4, via HDMI CEC. See here for instructions.
Add support for the latest stb-tester Python API v27.
Web UI: Show X and Y coordinates when you move the mouse over the live video. This is useful when you need to define a Region in your test script.
Web UI: Fix rendering issue that caused some remote control SVG images to be cropped rather than scaled.
Web UI: Add link to job logs. This is useful to debug issues when a test-job fails to start (for example because you forgot to plug in the infrared transmitter, or because your setup script failed).
- The pip Python package manager is installed by default in the test-run
environment. If you use a setup script to
install third-party packages with
pip install, you no longer need to install pip with
apt-get install python-pip.
- The version of pylint (a Python static analysis tool) installed in the
test-run environment is upgraded to version 1.6.4 (from 1.1.0). If you run
stbt lintwith stbt-docker, you may need to update your pylint configuration file.
- Python API:
stbt.Regionhas the following new methods:
left_of. They return a new Region relative to the current region.
- Python API:
stbt.match_textcan take single-channel images (black-and-white or greyscale).
- Python API:
stbt.match_textnormalises punctuation such as em-dash and en-dash, just like
- Python API:
stbt.match_texthas a new parameter
case_sensitive. It defaults to False (that is, ignore case), which was the previous behaviour.
4 October 2016
The stb-tester ONE will delete the oldest test results when its disk runs out of inodes. This can happen if your test-cases write many small test-artifact files. Note that the stb-tester ONE already monitored disk space, just not the inode usage.
Web UI: Add option to view JPEG screenshots instead of the Flash-based video player. The screenshot is updated once per second. It is a low quality thumbnail to keep the bandwidth low.
This is useful when you don’t have Flash installed, or when you are accessing the stb-tester device through a firewall that blocks Flash video, or over a network that doesn’t have sufficient bandwidth.
22 July 2016
New features for managing large test farms:
- Stb-tester devices on your network auto-discover each other using Zeroconf.
- Web UI: New drop-down menu that shows other stb-tester devices on your network, allowing you to quickly switch between your devices-under-test.
- Web UI: New monitoring view (accessed from the above drop-down menu) that shows the live video from all the devices in your test farm.
For more details see Large test farms in the User Interface reference.
Web UI: Scale the remote control image if it is too large.
13 July 2016
Add support for frame-accurate performance measurements. See the new stbt APIs below for retrieving the timestamp attached to each frame.
See How we test the stb-tester ONE: Video-capture latency & accuracy for details on the precision provided by the stb-tester ONE hardware.
Add support for the latest stb-tester Python API v26.
stbt.framesnow return a
stbt.Frameinstead of a
stbt.Frameis a subclass of
numpy.ndarraywith an additional attribute:
time. This is the time at which the video-frame was captured, as a floating point number expressed in seconds since the epoch, so you can compare it against Python’s
stbt.framesyields pairs of
frame, timestamp. The timestamp element of that pair is now deprecated (it’s a number in nanoseconds not seconds). Use the frame’s
stbt.MotionResulthave a new
timeattribute. This is the time at which the matching video-frame was captured. The
timestampattribute of these classes is now deprecated. See the previous bullet point for details.
stbt.wait_for_motionreports the time when the motion started, not the time after
consecutive_framesof motion have elapsed.
__nonzero__. This means you can write
if result:rather than having to write
if result.motion. This is a minor ergonomic improvement for consistency with
stbt.MotionResulthas a new
regionattribute which indicates where in the frame the motion was detected.
The way motion detection is visualised on the test-result video recording has changed. Instead of colouring the in-motion areas red we draw a red rectangle around the area with motion. This is consistent with the
3 June 2016
Add support for the latest stb-tester Python API v25.
frameparameter of the
stbt.wait_for_matchis now read-only. Use
frame.copy()if you need a writable copy.
- New base class
stbt.FrameObjectfor user-defined Frame Objects. The Frame Object pattern simplifies testcase development and maintenance; Frame Objects are a layer of abstraction between your testcases and the stbt image processing APIs. See this tutorial for details.
- New function
stbt.match_allthat searches for all instances of a reference image within a single video frame. It returns an iterator of zero or more
stbt.MatchResultobjects (one for each position in the frame where the reference image matches).
- Bugfix: When
wait_for_matchcan’t find the specified reference image, the error message now says the actual path & filename you specified (that is, a path relative to the test script), not an absolute path under the current working directory.
match_textnow logs a line saying whether it found a match or not, much like
wait_for_matchalready do. Thanks to Rinaldo Merlo for the patch.
27 May 2016
- Fix regression introduced in v24.7 that caused the stb-tester ONE’s old USB infrared transmitter to fail for signals longer than 125 milliseconds.
26 May 2016
Add support for new infrared transmitter hardware.
A timing fault in the old infrared transmitter hardware would sometimes cause missed keypresses. The failure rate depends on the receiver hardware. Some Roku models were particularly sensitive, missing up to 4% of keypresses (but most devices we tested were under 0.3%).
The new hardware is completely reliable. We haven’t seen a single missed keypress in over 500,000 presses. We are shipping the new hardware to all our existing customers; you should hear from us within the next few days.
The old infrared transmitter will continue to work as before.
How to tell them apart: The new infrared transmitter is a bit larger than the old one (see photo).
If you’re interested in the technical details of the fault, or how we tested the new hardware, see this blog post.
22 April 2016
- Web UI: Show image thumbnails (instead of filenames) in the live testcase view when you’re running a test.
This only affects two devices: stb-tester-one-b8aeed72e364 and stb-tester-one-b8aeed72c28e.
A bug in our factory provisioning process caused video-capture to be done in the YUY2 colourspace instead of the full BGR24 colourspace on these stb-tester ONE devices. This causes a slight difference in colours captured in screenshots and during testing, compared to other stb-tester ONEs. A user-visible difference is that the affected devices show black as RGB #101010 rather than #000000. These differences are below the default threshold of
stbt.matchso applying this fix shouldn’t affect the behaviour of any test-cases that use screenshots captured before the fix.
Stb-tester ONE devices are all intended to be identical so this release offers the affected users the option to fix this behaviour, for consistency with other stb-tester ONEs. As it is a slight change in behaviour we’ve made this opt-in rather than applying the fix automatically. To apply this fix, after installing v24.5 make an HTTP POST to
http://<hostname>:8001/software-update/bgr24-video-capture– for example with curl:
curl -d "" http://stb-tester-one-b8aeed72e364.local:8001/software-update/bgr24-video-capture. The change will take about 5 minutes and the device will reboot when complete.
11 February 2016
Web UI: Added branch selector. If you push your changes to a git branch other than master, now you can view and run your tests from any git branch:
This enables git workflows where you try your changes out on branches before merging to master. This should make sharing an stb-tester ONE between test engineers more convenient.
Note: This feature was already available through the REST API since v14.
- The infrared transmitter can now handle signals longer than 125 milliseconds.
- The “Stop” button would occasionally fail to stop a running test job, especially if the testcases were doing a lot of OCR at the time. This was caused by an interaction between docker and a bug in the AUFS driver in the ubuntu kernel.
- Web UI: Fix links to results with a “&” or “=” in the “job category” name.
20 January 2016
- Manual control: Show feedback & logs when pressing a button on the remote control doesn’t work (for example, if the lircd.conf file is invalid).
- You can plug in the IR transmitter after booting the stb-tester ONE (previously the IR transmitter couldn’t be hotplugged). It still needs to be plugged in before you start a test job (that is, before pressing the “Run tests” button in the UI or using the run_tests REST API).
- REST API: The
test_pack_revisionparameter to /api/v1/run_tests can be the name of a git branch or tag instead of a git SHA.
- REST API: The
test_pack_revisionparameter to /api/v1/run_tests can be an abbreviated SHA again (v24.1 introduced a regression where
test_pack_revisionhad to be the full 40-character git SHA).
- Customising the test-run environment didn’t work if you set stbt_version=24 in .stbt.conf.
- Web UI: The “filter” text field in the results view doesn’t lose focus when you’re typing.
14 January 2016
Add support for the latest stb-tester Python API v24.
- Python API: New method
stbt.Region.replaceto set any of the edges of a region to the given coordinates. It is similar to
stbt.Region.extend, but it takes absolute coordinates within the image instead of adjusting the edge by a relative number of pixels.
- Python API:
stbt.wait_for_matchtakes an optional
regionparameter, just like
- Python API:
stbt.match_textnow adds the expected text to tesseract’s dictionary, which fixes some false negatives.
- Python API: Bug fix:
stbt.wait_untilno longer raises an exception if you passed a
functools.partialobject (or an instance of a class with a
wait_untiltimed out. Thanks to Martyn Jarvis for the patch.
- Python API: You can now raise an exception with unicode in the exception’s message. Previously the testrun’s failure reason would show a “UnicodeEncodeError” instead of your actual exception message.
- The video-recording of a test run will show the name of the image that
stbt.wait_for_matchis searching for. Thanks to Lewis Haley for the patch.
- The screenshot in the results view is more likely to be relevant to the issue seen. Now we save the last frame that the test-script saw; previously we took a new screenshot at some point soon after the test-run had completed.
- Test runs start faster because the old HTML results view is no longer regenerated before and after each test-run. This is possible since the new test-results interface was introduced in v23.7. The improvement is particularly marked during long soak-tests with thousands of test runs as the old system would scan all the results in the category every time.
11 January 2016
New test-results interface with much better search capabilities.
- Interactive search that updates the results as you type. The search language is documented here.
- Ability to search & display results from multiple job categories at once.
- This lays the groundwork for future features such as interactive and automated classification of results.
- The search language is part of our stable public API. This means that the results for a given search string won’t change across stb-tester ONE releases.
- You can still reach the old static html results view by manually entering
http://<stb-tester ONE address>/results/(for example if your workflow depends on retrieving and processing that HTML – please let us know if this is the case). Note also that the stb-tester ONE provides a results REST API since v22.5.
Prettier default image for the remote control in the “manual control” view.
Bugfix: Automatically recover from “No video-capture device detected” on some stb-tester ONEs after powering off & on via the front-panel button.
16 December 2015
This is a bugfix release.
- The stb-tester ONE now works if you plug it into a network in the 172.17.x.x range (previously the stb-tester ONE reserved that address range for the containers it uses to isolate your test-runs).
- If the stb-tester ONE is plugged into a subnet that isn’t in the 172.17.0.0/16 range, but you also have a 172.17.x.x subnet that is routable from the stb-tester ONE’s subnet: Test scripts running on the stb-tester ONE can now contact hosts on the 172.17.x.x subnet.
- Fix very occasional race condition on boot, where the stb-tester ONE would fail to detect the video-capture hardware so you would see a test pattern in the video instead.
9 November 2015
- Reliability: Repair possible corruption of results database if the stb-tester ONE is shut down uncleanly (for example if there is a sudden power loss).
- Technology preview: New results view with better search capabilities. For now
this isn’t linked directly from the user interface; to see it go to
http://<stb-tester ONE address>/app/#/results. For more details see the User Interface Reference.
9 September 2015
- The stb-tester ONE will delete the oldest test results when available disk space is low. Each stb-tester ONE has enough space to store results (including logs, screenshots and videos) for approximately 6 months of continuous testing – less if you collect & store your own logs from the system-under-test.
- REST API: Bugfix: Fix 404 error from the artifacts endpoint if the result’s job category had spaces in the name.
- REST API: Bugfix: “job_url” and “result_url” in the responses from various API endpoints now include the correct port number if your stb-tester ONE is behind a port-forwarding firewall where the exposed port is not the default port 80.
10 July 2015
- Add support for the latest stb-tester Python API v23.
- Fixed bug where no new test jobs could be started after running tests for a week without a reboot.
- New Run forever in random order soak
option and an associated
shuffleargument to /api/v1/run_tests.
- The text drawn on the recorded video for each keypress sent with
stbt.press, and text drawn explicitly with
stbt.draw_text, now fades out over a few seconds. This makes it easier to distinguish the new messages from older messages.
27 June 2015
New REST API endpoints:
These can be used to retrieve detailed information on each of the test runs which ran during a test job. This is useful for integrating an stb-tester ONE with external test reporting systems.
10 June 2015
In the near future we will release further endpoints to retrieve more detailed results from a job (broken down by testcase).
Log files are served with the correct encoding, so unicode characters are displayed correctly in the browser.
The test-pack git URL is displayed in the stb-tester ONE’s user interface in the “Automated tests” tab. This is for convenience, so that you don’t have to look it up in the user manual.
14 May 2015
Improved feedback of test results as a job is running:
Added HOME and TV buttons to the default manual remote control.
Fixed issue where IR key-presses were transmitted incorrectly when the stb-tester ONE is under heavy load. This was causing key-presses to be lost.
27 March 2015
- Implement mechanism to apply software updates to the stb-tester core Python API independently of updates to the stb-tester ONE’s user interface. See test_pack.stbt_version in .stbt.conf for details.
- Fix intermittent incorrect error message during software update.
Fix printing unicode strings (such as the result of
stbt.ocr) to the test-run log.
m = stbt.wait_until( lambda: stbt.match("success.png") or stbt.match("error.png")) assert m if m.image == "error.png": recover()
Improvements to the test-run results viewer:
assertstatements in your testcases are now treated as test failures (red), not as test errors (yellow). See Exceptions for more details. Also, the failing assertion is displayed in the test-run’s “failure reason” field.
- The video recording of each test-run now displays the wall-clock time at the top of each frame. This makes it easier to relate timestamps from the logfiles with the video.
- The text drawn on the video recording (that shows keypresses, etc.) now uses a more legible font and background.
- The graph of test-runs now uses a logarithmic y-axis for test duration, so the graph is more useful when there are very long test runs alongside shorter ones.
- Improve the layout of the results table.
- Fix a bug in the search function, where it wouldn’t work if there were some particular values in the first row of results.
5 March 2015
- Allow selecting which remote control configuration to use when running tests.
- Add ability to run soak-tests (run the selected tests forever until you stop them by clicking the “Stop” button).
- Improve video-capture reliability when the system-under-test changes output framerate.
25 Feb 2015
Add ability to customise the test run environment.
Fix bug where
git pushwith large files would fail with:
RPC failed; result=22, HTTP code = 413 fatal: The remote end hung up unexpectedly
Expose test-job ID to test-pack in
Fix bug where last job id would not be reported correctly after a reboot.
Fix incorrect error message during software update.
16 Jan 2015
- Improve software update usability and feedback.
- Add button to stop the currently running test job.
- Add job status banner above live video.
- Add public REST API.
- Remove job status debugging text below “Run Tests”.
- Highlight the currently running line in the testcase Python script.
2 Jan 2015
- Add Manual Remote Control in the Web UI.
- Improvements to the video capture and streaming.
- Show current version number in the Web UI.
- Show disconnection icon () in the Web UI.
- “Help” link to the User Guide in the Web UI.
- Don’t stop a test run (of multiple tests) if one test fails.
- Zeroconf now advertises as an HTTP service so the stb-tester ONE appears in Bonjour browsers on Mac OS X.
19 Dec 2014
Initial pilot release to select partners.