If you have already automated one or more testcases, run them! Ideally run them every time a developer commits a change to the system-under-test. If that isn’t possible, at least run the tests against nightly builds. Run! Your! Tests!

It sounds obvious, but so many organisations fail at test automation because they don’t run the tests often enough to get real value. We have seen teams spend a year (no exaggeration) building elaborate test infrastructure without producing any test results. Run your tests! Don’t spend any more time automating new testcases until you are consistently running the tests you already have!

Costs of automated vs. manual testing

The costs of automated testing are very different to the costs of manual testing. A manual testcase is relatively cheap to write and relatively expensive to run, whereas automated testcases are relatively expensive to write and very cheap to run — almost free.

So what’s going to give you more value for the least effort: Writing more automated tests, or running them more often?

I’m not suggesting that you shouldn’t aim for high test coverage. But pick the low-hanging fruit first!

Run your tests against more builds

If your QA team gets infrequent releases from your development team, and you only run your automated tests against each release, some of your tests will fail because they haven’t kept up with changes to your set-top box’s UI.

If you only run your tests once a month, and you also have to fix the tests each time, then it might be cheaper to stick with manual testing!

To take advantage of automation’s strengths, you want to be running your tests against every change made by the development teams, in a Continuous Integration system (CI). The benefits of CI are well known: Developers can work faster and more productively, with confidence that they aren’t introducing regressions. Bugs are caught earlier, and thus they are easier and cheaper to fix.

Run your tests on more devices

If you need to support several different set-top box models, you will often get more value by extending your existing tests so that they work on all the different models, than by increasing the test coverage for a single device.

If the user journeys are similar across your devices, you can use the Frame Object technique to handle UI variations while using the same test scripts for all your devices.

Run your tests more times: Soak testing for stability

Even the most trivial testcases can sometimes find memory leaks and crashes if you run them long enough. If your set-top box is going to sit idle, why not leave your tests running all night? You’ll be surprised at the bugs you’ll find.

Soak-testing is especially valuable if you’re a set-top box manufacturer or integrator. If you’re developing an app for someone else’s device you’ll be less interested in device instabilities, but a soak test can still find resource leaks in your own app.

Even if you don’t have time to look into every test failure, you can get good value by implementing some simple health-checks to detect crashes, spontaneous reboots, and memory leaks.

And even if you don’t look at the results of every overnight soak, run it anyway! That way you’ll have data if you need to answer “when did this stop working?” or “when did this start getting slower?”.

Run your tests more times: Soak testing for functional coverage

If you run your tests in a random order you will end up exercising user journeys that aren’t explicitly coded into your test scripts, and this can often uncover bugs that you wouldn’t find otherwise. (These bugs tend to be the kind that are really annoying to diagnose if they’re only noticed in the wild!)

What do I do next?

Don’t spend any more time automating new testcases until you are consistently running the testcases you already have.

Don’t let this be a manual process. Set up a Jenkins job (or similar) that runs every night: It deploys the latest software build to your device-under-test, and runs all your automated tests in a random order, for 16 hours (or however long it takes before you get into the office the next morning).

Depending on your development team’s process and tools, deploying the latest software build to your device-under-test could be the hardest part to automate. Don’t give up! On previous projects we have automated the update process of set-top box firmware that requires plugging in a physical USB stick, by using a controllable USB switch.

Once this is in place, the next step (automatically testing every change in Continuous Integration) will be easy!