Our approach to testing

The benefits of test automation come at scale. By “scale” we mean that you are running tests constantly: Ideally against every software change made by your programmers, and possibly against multiple hardware variants or the different platforms that you target.

Automated test scripts need to be extremely reliable – at least one or two orders of magnitude more reliable than the system they are testing. Otherwise you will get false results that will require manual analysis, limiting the scalability of the automation.

Our experience has taught us that achieving this reliability requires treating test development with the same rigour as software development. This means that tests need to be written by experienced software developers. A “record - playback” approach, where test scripts are recorded by unskilled operators, is initially cheaper but cannot produce the reliability required to scale enough to provide real value.

Stb-tester’s user interface is designed for scale. Automatic classification of known defects, statistical measures, and a very powerful triage user interface let you cope with thousands of test results.

Run all your automated tests as early as possible in the development lifecycle, ideally as part of a Continuous Integration system. The earlier the test is run –and catches a defect– the cheaper the defect is to fix.

Your manual QA team now have a known baseline of quality for each release, because it has already been tested before it has reached them. By automating the tedious repetitive testing, the QA team can focus on what humans do best.