30 Jul 2014.

Stb-tester.com Ltd is currently working with a global TV manufacturer to provide testing services for their latest development project. To support this work we have developed a lot of exciting new features in stb-tester.

All posts in this series:

We’ve been far too busy to release these features officially, but over the next months you should see them making their way to an stb-tester release.

Testing HTML apps

Modern Connected TV standards such as HbbTV include an HTML run-time environment so you can develop the main UI of your device, and applications targeted at the device, in HTML + CSS + JavaScript.

One huge advantage of HTML as a development platform is that programmers can test their code in a desktop browser such as Chromium. If you don’t have to deploy your code to an external set-top box or TV in order to test it, your code-deploy-test cycle is much shorter, and your productivity is much higher.

When you’re testing a local application you want an easy way to launch and tear down the application under test, or to launch different versions of the application at the same time. So we’re adding a new command: stbt virtual-stb.

stbt virtual-stb <command> <args> will run the specified application in a headless X environment, and will modify your stbt configuration file so that commands like stbt run, stbt batch run, stbt tv, stbt screenshot and stbt control all operate against that application. For example, the following command will allow you to test YouTube’s living-room interface with stb-tester:

stbt virtual-stb chromium-browser --app=https://www.youtube.com/tv#/

Stb-tester uses GStreamer’s “ximagesrc” to capture video from the application under test’s X11 window, and uses stb-tester’s new X11 control to send keypresses to the application.

Strictly speaking this isn’t limited to HTML applications. For example if you wanted to test the venerable xclock application with stb-tester:

stbt virtual-stb xclock -update 1

So even if your UI isn’t implemented in HTML, if you can run it in a VM on your desktop then you can test it with stb-tester.

The code currently lives on the wmanley/virtual-stb branch on github, but it is still undergoing heavy modification. If you can think of a better name for the new command than “virtual-stb”, let us know!

Where does stbt virtual-stb fit in the development lifecycle?

With stbt virtual-stb you can run automated UI tests without real set-top box or TV hardware. This makes automated testing much cheaper and it allows you to move automated testing earlier into the development lifecycle.

For example in our current TV project every change made to UI code is automatically run through stb-tester sanity tests and the result is displayed next to the corresponding branch or pull request. If you haven’t experienced a decent Continuous Integration system first-hand, you might not appreciate just what a productivity booster this is!

You don’t need video-capture hardware either. This makes it much cheaper and easier to build and maintain automated test infrastructure for your UI development team, and it makes your Continuous Integration service much more reliable.

At the other end of the development lifecycle, you can still use stb-tester for integration testing on real production hardware. In fact you can use the same test scripts that you wrote for UI development testing! The only difference is that instead of the “virtual-stb” video-capture and control, you configure stb-tester to use a video-capture card and an infrared emitter.

Testing internet-connected TVs

Building on earlier work we did for a large European video-on-demand provider, we will be polishing stb-tester’s Smart TV support to bring it to a production-ready standard.

TVs tend not to have a video output (HDMI or anything else), so to support testing a TV with stb-tester we point a digital camera at the TV screen. After an initial calibration process we apply image processing to the live video captured by the camera so that it looks just like the video coming from an equivalent set-top box. We apply the following corrections:

  • Geometric correction to account for the position of the camera and the TV.
  • Colour correction.
  • Vignetting correction for distortion caused by the camera lens.

This will allow you to test your Smart TV / IPTV / HbbTV product, or applications targeted to such TVs, with stb-tester. The end result is that you can use the same test scripts to test the app running on the desktop, on a set-top box, and on a TV.

Going back to the software development lifecycle, stb-tester covers the spectrum from development testing and Continuous Integration (using the “virtual-stb” software-based video capture), to integration testing against a dev board (using a video-capture card), to integration and acceptance testing against the final TV product (using a camera).

Many consumer electronics projects don’t have much hardware available for testing in the early stages of the project. Stb-tester allows you to develop your automated test scripts in parallel with the development of the system under test, helping you to keep automated test development off of the critical path.

There’s more!

In the next blog posts we will describe API improvements that make it easier to compose stb-tester’s image-processing operations to meet the demands of complex real-world scripts; a more powerful and more pythonic test runner; stateless docker images to simplify the management of servers in your test farm; and more. Stay tuned.

Comments? Join the discussion on the project mailing list.