Stb-tester Blog

Stb-tester news, product announcements & testing topics.

Tags: Announcements Device Setup Test Development Test Strategy

14 Feb 2022.

Lint checks in pull requests

When you push a git commit, Stb-tester uses any idle Node in your test-farm to run a “lint” check using Pylint to catch common mistakes in your code before you even run a test. Read more...

13 Dec 2021.

New user role: Manual control only

The "manual control" role allows users to control devices-under-test remotely, and see live video from the devices, without access to the test scripts in the test-pack git repository. This allows you to grant access to a third party without letting them see the test scripts you have developed which are your own intellectual property. Read more...

18 Oct 2021.

Testing the Nvidia Shield with Stb-tester

Stb-tester supports automated testing on the Nvidia Shield TV and Shield TV Pro. Read more...

01 Oct 2021.

SSL errors accessing Stb-tester REST API from RHEL/CentOS 7

If you started getting errors today when accessing the REST API on your Stb-tester Portal (either directly using curl or python requests, or using our stbt_rig.py command-line tool) then read on for the solution. Read more...

07 Sep 2021.

New features: Prometheus/Grafana integration; Bluetooth & RF4CE support; interactive log viewer

2021 has been a busy year for us, with major new features implemented to support specific customers’ requirements: Prometheus and Grafana dashboards for logging and reporting performance measurements and other real-time metrics; support for some Bluetooth and RF4CE remote controls via RedRat-X hardware; an interactive log viewer with frame-accurate synchronisation between each log line and the video recording of the test-run; and a new test-farm administration page. Read more...

24 Aug 2021.

New instructions for VSCode setup on Windows

Our VSCode IDE integration makes test-script development convenient. With it you get: Read more...

11 Feb 2021.

How to structure your test scripts for maximum re-use across multiple platforms

If you want to test the same app on several different platforms, you want to re-use as much test-script code as possible. For example you might have your app on Android TV, Apple TV, Fire TV, PlayStation, Roku, XBox, and possibly even some operator-specific set-top box platforms. This article shows how to use the same test scripts and the same Page Objects for all these platforms, and how to manage the cases where your app has differences in appearance or behaviour across platforms. Read more...

24 Dec 2020.

Better latency in Stb-tester's live video (goodbye Flash, hello WebRTC)

The Stb-tester Portal shows live video from the device-under-test, right in your web browser. Previously, this video was streamed from your Stb-tester Nodes using RTMP (Flash’s video-streaming protocol). With our latest firmware release, Stb-tester Nodes now stream this live audio & video using WebRTC – the W3C standard protocol for low-latency, real-time communication. WebRTC allowed us to achieve much better latency when you are interacting manually with your device-under-test via the Stb-tester Portal. Read more...

02 Oct 2020.

Testing on-screen keyboards with Stb-tester

Stb-tester v32 added new APIs that make it much easier to navigate on-screen keyboards from your test scripts. In this tutorial we’ll model the behaviour of YouTube’s search keyboard on the Apple TV, and we’ll write a Page Object (a Python class) to navigate the keyboard from our test scripts. Read more...

02 Oct 2020.

Stb-tester v32 released: Better on-screen keyboard support, new API to find selections, and more.

Version 32 of the stb-tester core Python API is now available. We have re-architected stbt.Keyboard to easily test on-screen keyboards with multiple modes (such as lowercase, uppercase, and symbols modes). We have added a powerful new API called stbt.find_selection_from_background to easily detect the position of a “selection” or “highlight” on the screen. We have sped up stbt.ocr hugely by caching previous results for identical inputs — so now you can use OCR more freely in navigation code without worrying about slowing down your test scripts. Read more...

30 Apr 2020.

Stb-tester allows you to test set-top boxes remotely

Stb-tester’s cloud-based portal gives you remote access to all the set-top boxes and OTT devices in your test-farm, from anywhere in the world. Many of our customers are using this capability to allow their staff, who are in quarantine during the current coronavirus pandemic, to continue QA activities from home — both manual & automated testing. Read more...

06 Feb 2020.

Announcing Stb-tester integration with GitHub Enterprise

Stb-tester now provides tight integration with your organisation’s own GitHub Enterprise instance. Read more...

28 Jan 2020.

Visual Studio Code integration with the Stb-tester Portal

Stb-tester provides deep integration with Visual Studio Code, the world’s most popular IDE.¹ You can now execute tests remotely (on your Stb-tester Node hardware) right from your IDE. This can greatly speed up your test-development cycle.

According to Stack Overflow’s 2019 [developer survey]. ↩

23 Jan 2020.

Static analysis for your test scripts: Catching common mistakes before you run the tests!

Pylint is a static analysis tool for Python code. This means that it analyses your code to catch common mistakes, without actually running your code. This saves a lot of development time by catching the mistakes immediately —your IDE shows red squiggles under each mistake— instead of running the test and waiting for it to reach the line of code where the mistake is. Read more...

05 Dec 2019.

Python's for...else: Surprisingly useful for implementing retry logic in test scripts

The optional else clause after a for loop is an obscure Python feature that is surprisingly useful when writing automated GUI tests. Read more...

02 Dec 2019.

IDE Configuration: Smart code completion & linting for Stb-tester APIs

The Stb-tester Python APIs are now available on PyPI (the Python Package Index) so that you can install them on your development PC — it doesn’t matter if you’re using Windows, MacOS, or Linux. Read more...

14 Oct 2019.

Using the Object Repository to debug your Page Objects

I have written a Page Object for the “Search” page in the YouTube app on Apple TV. My Page Object has a property that reads the search term using OCR, but the OCR is mis-reading some words. Let’s investigate! Read more...

20 Sep 2019.

Porting a test-pack to Python 3

Stb-tester v31 adds support for test-scripts written in Python 3 (Python 2 is still supported, too). To migrate an existing test-pack to Python 3, we recommend the following steps. Read more...

19 Sep 2019.

Stb-tester v31 released: Python 3 support! On-screen-keyboard navigation! And more!

Version 31 of the Stb-tester core Python API is available, with support for test-scripts written in Python 3 (Python 2 is still supported, too). Read more...

26 Feb 2019.

Using match transparency

Stb-tester v30 includes a new feature: Searching for a reference image that has some transparent pixels. This blog post describes three ways you can use this new feature that makes writing test-scripts much easier than before. Read more...

25 Feb 2019.

Stb-tester v30 released: Transparency support, Ubuntu 18.04 and more!

Version 30 of the stb-tester core Python API is available. stbt.match now supports transparency: Transparent pixels in the reference image will be ignored when looking for a match within the video-frame. This is a huge usability improvement when looking for buttons, selections/highlights, and similar outlines that have dynamic content or non-rectangular shapes. Read more...

22 Oct 2018.

Testing Alexa integration

An end-to-end test for your Alexa integration is easy to automate with Stb-tester. In this short video we issue commands to Alexa by playing pre-recorded audio files (“Alexa, play Mr. Robot”). Then we validate that the command was successful by looking at the video output from the set-top box (in this video, a Fire TV Stick). Read more...

11 Oct 2018.

Stb-tester support for PyCharm

Stb-tester now provides PyCharm integration out of the box, allowing you to run tests against the real set-top-box, right from your IDE: Read more...

21 Aug 2018.

Meet Stb-tester at IBC 2018

The Stb-tester team will be at the IBC tradeshow in Amsterdam 14-18 September 2018. Read more...

20 Aug 2018.

Testing the Roku Streaming Stick+ with Stb-tester

Stb-tester has always supported Roku boxes (Roku HD, 2, 3, 4, Express, Express+, Premiere, Premiere+, and Ultra). Now, Stb-tester also supports the Roku Streaming Stick and Streaming Stick+. Read more...

22 Jun 2018.

Stb-tester v29 released: Audio support and more!

Version 29 of the stb-tester core Python API is now available. This is a significant release with major new APIs for testing audio. Read more...

10 Apr 2018.

Test-script development workflow

When you are developing new test scripts (or modifying existing ones) we recommend this workflow: Read more...

26 Mar 2018.

Testing the Amazon Fire TV with Stb-tester

Stb-tester supports automated testing on the Amazon Fire TV and Fire TV Stick. Read more...

23 Mar 2018.

Announcing Stb-tester integration with Atlassian Bamboo CI Server

The Stb-tester Platform now supports Atlassian's Bamboo Server for Continuous Integration (CI). Many of our customers already use Bamboo for continuous integration, deployment, and delivery. Now, it will only take a few minutes to get your Stb-tester end-to-end tests running from your Bamboo jobs. Read more...

20 Mar 2018.

Cropping reference images in your browser

When you are capturing reference images for your test scripts, the Stb-tester Portal now allows you to crop the images directly in your browser. Read more...

06 Feb 2018.

Stb-tester v28 released: Multi-threading support, better OCR, better video recordings

Version 28 of the stb-tester core Python API is now available. Read more...

17 Jan 2018.

Announcing the Stb-tester Cloud Platform

Stb-tester’s cloud-based portal centralizes your test results and dashboards. It allows remote testing and live monitoring from anywhere in the world — your devices can be spread across multiple network locations, and your test team can be remote too. Read more...

27 Feb 2017.

Stb-tester ONE v27.4: Better test farm configuration, new test-results API, and remote Android ADB control

Software update v27.4 for the stb-tester ONE is now available. It includes a more convenient configuration system for large test farms, a new REST API for searching past test results, support for controlling Android devices without a USB connection, and more. Read more...

21 Feb 2017.

Testing video playback on mobile devices

There are several mature open source and proprietary tools for testing the user interface of mobile apps on Android and iOS devices, but there aren’t many solutions for internet service providers or video-on-demand streaming services who want to test video playback on real mobile devices to validate their end-to-end video delivery. In this article I’ll discuss various ways of getting video frames from a mobile device for automated testing, and the advantages and trade-offs of each approach. Read more...

23 Jan 2017.

Stb-tester ONE v27.3 released

Software update v27.3 for the stb-tester ONE is now available. It adds support for the Sony PlayStation 3 & 4, Android devices, and more! Read more...

05 Jan 2017.

Controlling the Sony PlayStation 4 for test automation

To test a PS4 app you need a mechanism to control the PS4 and a way to get feedback to check that your app is behaving as it should. Read more...

12 Dec 2016.

Announcing the stb-tester camera for TVs and mobile devices

The stb-tester camera allows you to test the user interface of devices that don’t have an HDMI output, such as Smart TVs and mobile devices. Read more...

28 Oct 2016.

Build your own serial-to-ethernet server with a Raspberry Pi

A Serial Terminal Server makes a serial connection available over the network, so that your application can access your devices without a direct serial connection. Typically this is used to establish a bi-directional connection between networked applications and older industrial equipment. In this article we will focus on a much simpler use case: Capturing serial logs, where we only need communication in one direction — a common use case when testing set-top boxes. Read more...

04 Oct 2016.

Stb-tester ONE v26.6 released

Software update v26.6 for the stb-tester ONE is now available. It contains a JPEG thumbnail alternative to the Flash video in the web interface, and a bugfix for inode exhaustion on the stb-tester ONE’s test-results partition. Read more...

20 Sep 2016.

Add visual verification to your Selenium tests with stb-tester

Selenium is a test automation tool for web applications. It tests your web app by looking at the DOM – you can check that your web page contains a particular element like a div or a button, identified by id or css selector or xpath expression; once the element is located, you can check that it contains a particular text or other elements. However you can’t really check what your web app looks like. If you have complex CSS it can be all too easy to introduce unintended defects in your app’s visual appearance. Read more...

08 Aug 2016.

Meet the stb-tester team at IBC

The stb-tester team will be at the IBC exhibition in Amsterdam 9-13 September 2016. Read more...

22 Jul 2016.

Stb-tester ONE v26.2 released

Software update v26.2 for the stb-tester ONE is now available, with new features for managing & monitoring large test farms. Read more...

13 Jul 2016.

Stb-tester v26 released

Stb-tester v26 is now available, with support for frame-accurate performance measurements. Read more...

05 Jul 2016.

How we test the stb-tester ONE: Video-capture latency & accuracy

Updated 17 May 2017 with figures for our new hardware’s accuracy. Read more...

29 Jun 2016.

Run your tests!

If you have already automated one or more testcases, run them! Ideally run them every time a developer commits a change to the system-under-test. If that isn’t possible, at least run the tests against nightly builds. Run! Your! Tests! Read more...

21 Jun 2016.

New tutorial: Test development workflow with stbt.FrameObject, stbt auto-selftest and stbt-docker

We have just published an extensive tutorial about our recommended workflow for implementing automated test scripts, using stbt.FrameObject to extract information from the screen, using stbt lint and stbt auto-selftest to test your scripts locally, and using stbt-docker to develop on Mac or Windows. Read more...

03 Jun 2016.

Stb-tester v25 released

Stb-tester v25 is now available, with features that significantly improve the maintenance cost of your test scripts: Read more...

27 May 2016.

Stb-tester ONE v24.8 released

Software update v24.8 for the stb-tester ONE is now available. It fixes a regression introduced in v24.7 that caused the stb-tester ONE’s old USB infrared transmitter to fail for signals longer than 125 milliseconds. Read more...

26 May 2016.

Stb-tester ONE v24.7 released, with support for new infrared hardware

Software update v24.7 for the stb-tester ONE is now available. Read more...

26 May 2016.

A post-mortem report on the stb-tester ONE's infrared transmitter reliability

This is a technical report on our investigation into an intermittent problem with the stb-tester ONE’s infrared transmitter. Read more...

06 May 2016.

New tutorial video: How to create a custom remote control image

How to create a custom remote control image for the stb-tester ONE’s “manual control” web interface: Read more...

22 Apr 2016.

Stb-tester ONE v24.5 released

Software update v24.5 for the stb-tester ONE is now available. This release delivers a minor UI improvement, and a bugfix that affects a small number of users. Read more...

29 Feb 2016.

New videos & tutorials

We have just published three videos showcasing the stb-tester ONE, including a detailed guide for measuring channel change times on a set-top box: Read more...

11 Feb 2016.

Stb-tester ONE v24.4 released

Software update v24.4 for the stb-tester ONE is now available, with the ability to run tests from any git branch, and several bugfixes. Read more...

20 Jan 2016.

Stb-tester ONE v24.3 released

Software update v24.3 for the stb-tester ONE is now available. This includes better feedback when you have errors in your infrared configuration file, and other small usability improvements & bugfixes. Read more...

14 Jan 2016.

Stb-tester v24 released

Stb-tester v24 is now available, with many small Python API improvements and fixes. Read more...

11 Jan 2016.

Stb-tester ONE v23.7 released

Software update v23.7 for the stb-tester ONE is now available. This includes a new test-results interface with much better search capabilities, plus some minor fixes & updates. Read more...

18 Dec 2015.

New case study: stb-tester.com supports YouView’s launch on Sony Bravia TVs

In November 2015 YouView launched on a range of Sony Bravia televisions across the UK. The biggest technical challenges of the project were in the integration, with frequent releases from YouView, Sony, MediaTek, and Google. Read more...

16 Dec 2015.

Stb-tester ONE v23.6 released

Software update v23.6 for the stb-tester ONE is now available. This is a bugfix release. Read more...

02 Dec 2015.

Stb-tester.com to help DTVKit with test automation solution

Stb-tester.com is pleased to announce that DTVKit will be using our stb-tester tool as part of their test automation strategy, and we will be providing our expertise and support as a Partner Member of DTVKit. Read more...

09 Nov 2015.

Stb-tester ONE v23.5 released

Software update v23.5 for the stb-tester ONE is now available. It has a technology preview of upcoming improvements to the results viewer, as well as bugfixes. Read more...

24 Sep 2015.

Improve black-box testing agility: automatic self-regression tests

In the previous blog post we discussed the frame object pattern and how it could lead to greater agility and lower maintenance costs thanks to improved self-testability. This blog post discusses a technique to get the maximum value from this testability at near-zero cost. Read more...

08 Sep 2015.

Improve black-box testing agility: meet the Frame Object pattern

In the previous blog post we discussed the [Page Object pattern] and how it could be used to improve the readability of your test scripts. In this blog post we introduce the Frame Object pattern, a specialisation of the Page Object pattern that can be especially effective in stb-tester’s style of black-box UI testing¹. Frame Objects can reduce the maintenance cost of your test-packs, particularly when the UI-under-test changes.

By “black-box UI testing” we mean that you’re testing what the user sees, by capturing a video stream of the rendered UI and using image processing to check it. ↩

02 Sep 2015.

Applying the "page object" pattern with stb-tester

Often you will hear us recommend that you should structure your test-packs as a set of library functions, plus test-cases that use those library functions. In this blog post we attempt to make this recommendation more concrete by demonstrating how to apply the “page object” pattern when writing stb-tester test-cases. Read more...

25 Aug 2015.

Handling UI elements of non-fixed size

In this blog post we use the OpenCV [cv2.floodFill][floodfill] function to improve the robustness of test helper functions for dynamic UIs. Read more...

10 Jul 2015.

Stb-tester v23 released

Stb-tester v23 is now available, with a new option to run your testcases in random order, aiming to spend approximately the same amount of time in each testcase. Read more...

27 Jun 2015.

Stb-tester ONE v22.5 released

v22.5 for the stb-tester ONE is now available. It provides a REST API to retrieve detailed test results and artifacts, intended for integrating the stb-tester ONE with external reporting systems. Read more...

10 Jun 2015.

Stb-tester ONE v22.4 released

v22.4 for the stb-tester ONE is now available. Read more...

14 May 2015.

Stb-tester ONE v22.2 released

v22.2 for the stb-tester ONE is now available. Read more...

31 Mar 2015.

Stb-tester v22 released

Stb-tester v22 is now available. Read more...

31 Mar 2015.

A better way to organise your testcases

Up until now, stb-tester testcases were Python scripts with one testcase per file, and you referred to your testcases by the filename. Read more...

26 Mar 2015.

New documentation for stb-tester's Python API

We have overhauled the stb-tester Python API reference documentation, with more examples, introductory material, a categorised index, and a search function. Read more...

25 Sep 2014.

Upcoming stb-tester features: Re-imagining the test runner

stbt batch run works really well for stress-testing and soak-testing, but it’s lacking a few features that would be useful for functional acceptance testing: Run each test once, then re-run each failing test; run the tests in a random order or specify ordering dependencies between tests; update an external bug database with the results. Or for a continuous integration service, only run the fast tests and don’t fail the build if a test fails due to a known defect. The list goes on. Read more...

22 Sep 2014.

Upcoming stb-tester features: Composable API

Stb-tester’s wait_for_match function works well enough, but what if you want to wait for a UI element to disappear? Or when you want to check that 2 different UI elements are present in the same frame? Or when you just want to know if the element is present or not, without raising an exception? An addition to stb-tester’s Python API, due in stb-tester 0.21, will make this much easier. Read more...

23 Aug 2014.

Upcoming stb-tester features: stb-tester as a stateless service

Do you wish that your stb-tester test farm was as easy to set up as Travis CI? Want a neat way of specifying your test script’s dependencies? Or maybe you want to run stb-tester from your Windows PC? The upcoming stbt service command solves these problems. Read more...

30 Jul 2014.

Upcoming stb-tester features: Testing HTML apps and Smart TVs

Stb-tester.com Ltd is currently working with a global TV manufacturer to provide testing services for their latest development project. To support this work we have developed a lot of exciting new features in stb-tester. Read more...

10 Jun 2014.

Creating LIRC config files - part 1: theory

TL;DR version:: stb-tester reqires a LIRC config file to know how to translate key names into IR pulses. irrecord can be used to create them by recording IR signals but can have difficulty sometimes. In those cases you can provide additional information to make recording succeed which can be determined from the output of mode2 and xmode2 and by consulting the lirc config file documentation.

10 Jun 2014.

stb-tester 0.20 released with improved OCR

stb-tester 0.20 has been released. The key new developments are: Read more...

16 May 2014.

Stb-tester Continuous Integration with your GitHub repo

A Continuous Integration (CI) system, which runs your automated tests against every change made to your source code, is an important part of every Agile development process. With stb-tester your CI system can run integration tests on the real target device. As a demonstration we have set up stb-tester CI for a simple “Hello World” Android app at this git repo. Read more...

14 Apr 2014.

Improving OCR accuracy

TL;DR version:: stbt.ocr’s misreadings aren’t due to it not being familiar with your fonts. The root cause is that the OCR system we use (tesseract) is intended for reading the printed word. With this new understanding we’ve made improvements to OCR resulting in a 12× reduction of errors. This will be in the next stb-tester release.

09 Apr 2014.