23 Aug 2014.

Do you wish that your stb-tester test farm was as easy to set up as Travis CI? Want a neat way of specifying your test script’s dependencies? Or maybe you want to run stb-tester from your Windows PC? The upcoming stbt service command solves these problems.

All posts in this series:

Stb-tester.com Ltd is currently working with a global TV manufacturer to provide testing services for their latest development project. To support this work we have developed a lot of exciting new features in stb-tester. We’ve been far too busy to release these features officially, but over the next months you should see them making their way to an stb-tester release.

stbt service

The stb-tester service is intended to:

  • Simplify management of stb-tester test farms,
  • provide greater guarantees about reproducibility of test runs,
  • allow more power when creating a test-pack,
  • make it easier to integrate stb-tester into external systems such as Jenkins CI,
  • and make it easier to use stb-tester remotely.

We do this using Docker containers to manage any system state explicitly to provide traceability and reproducibility without constraining user flexibility:

  • Your test scripts are run in their own container (sandbox), which you can customise fully: Install any packages or libraries you need to use in your test scripts, or choose a different or customised version of stb-tester. You have a git repo containing your test scripts and a Dockerfile that specifies the dependencies needed to run those scripts; we call this a “test pack”.

  • Your test pack is reset between test runs so you know exactly what state it’s in. No more worries that the test machine was left in some funny state or that your dependencies are missing, broken or at the wrong version.

  • The command to run in the test pack is unconstrained, much like sshing into a machine. It doesn’t even have to be stbt run or stbt batch run.

  • The git checkout of the test pack and the creation of the test pack Docker image are cached, so running a command remotely is fast and can be treated as if stbt were running locally. Stb-tester doesn’t run on your Windows desktop PC? No problem, you can be just as productive remotely.

  • We apply the same approach to the stbt service infrastructure itself. It is entirely containerised allowing easy installation and upgrades. By separating the code from the state we can upgrade/rollback the code with confidence.

  • The “virtual-stb” running your system-under-test is also sandboxed using Docker.

Instructions and further technical details are available on the stb-tester next branch. (This branch is still subject to change, and the stbt service APIs are not yet stable.)

stbt service and the stb-tester ONE appliance

All of the above is a crucial part of the upcoming stb-tester ONE appliance, which is still under development.

It allows our appliance to be a stateless system: Rebooting the appliance is essentially the same as performing a factory reset. The only state that is preserved on the appliance are your test results and your authentication information (keys). All other configuration is stored in your test pack (your git repository).

It also allows us to upgrade the software infrastructure running on the appliance without changing the run-time environment of your test scripts, and it allows you to install any libraries and tools you like without harming the appliance’s core operating system.

There’s more!

In the next blog posts we will describe API improvements that make it easier to compose stb-tester’s image-processing operations to meet the demands of complex real-world scripts; a more powerful and more pythonic test runner; and more. Stay tuned.

Comments? Join the discussion on the project mailing list.