Improve black-box testing agility: automatic self-regression tests
24 Sep 2015. By William Manley.
In the previous blog post we discussed the frame object pattern and how it could lead to greater agility and lower maintenance costs thanks to improved self-testability. This blog post discusses a technique to get the maximum value from this testability at near-zero cost.
All software has costs associated with it. Costs to write in the first place, costs to fix bugs, and costs associated with ongoing maintenance. Having automated test-cases reduces these costs. Of course test-cases are software themselves, with their own associated costs. There’s a strong argument that we should test the tests, but at some point the marginal cost of an additional layer of self-testing will exceed its value.
Here’s how we reduce the cost of some self-tests (tests that test the test-pack itself) to near zero: We automatically generate a set of Python doctests which capture the behaviour of our frame objects based on a corpus of images.
Doctests and the frame object pattern
In the previous blog post we said that Frame Objects should define a
__repr__ method to make them usable with doctests. Here’s what a
doctest based on last week’s
GuideFrame frame-object might look like:
This is a clean and human-readable form of recording our expectations of the
GuideFrame class when provided with a variety of different images. We defined
GuideFrame.__repr__() to return as much information from the frame as it could
extract and we see this here.
We can verify that the behaviour of the
GuideFrame class is as expected with the
We can run this command every time we make a change to the test-pack and preferably as a part of selftest CI (a subject for a future blog post). With this we gain confidence that we haven’t caused any regressions with our changes. With this confidence comes agility. Fantastic so far…
It’s not so arduous to write these tests out by hand when your test image corpus is small and you have just a few frame objects, but the task quickly becomes tedious, particularly when we are extracting many pieces of information and we start to support more and more UI variants. Adding a new property to our frame object now requires updating as many doctests as we have example frames.
Doctests are validated automatically, and they can be generated automatically too. We’ve created a tool to generate doctest files based on some annotations added to our test-case modules.
For example, at the beginning of
guide.py we might have:
We would then run our tool:
and it would generate this:
Then we commit this file to git. It may seem unusual to commit a generated file, but is has a fantastic effect. See this diff from a real fix that we applied to the helper function introduced in a previous blog post:
By looking at the diff from one commit from the next you can see the change to the test code, and also the effect of that change! In the regression test output you can see that we are now correctly reading “BBC London News”, and more significantly you can see that we haven’t caused a regression to any of the other results.
In this instance we added the example screenshot
iplayer-playing-bbc-london-news.png in a previous commit. This allowed us
to capture the incorrect behaviour in the diff too, so we can better see that
it was fixed.
This makes creating test-cases easier and cheaper and improves agility:
- Creating Frame Objects is easier: you just need some screenshots. Checking that they work is automatic.
- Code review is easier: the changes are captured alongside the results.
- It’s easier to test multiple hardware or UI variants: you can be sure that any changes you make to support these variants work correctly and don’t cause any regressions.
- The auto-regression-tests serve as a form of documentation – much like manually written doctests do.
Given this, the process we follow when fixing a defect with our test-cases is:
- Find a frame of video that reproduces the issue.
- Commit it to our repo with the incorrect behaviour recorded in the auto-self-tests.
- Fix the issue.
- Verify the fix by checking that the incorrect recorded behaviour is now corrected and that there have been no regressions in the other results.
- Commit both the fix and the changes to the auto-self-tests to git.
We’re currently looking at adding this tool to stbt proper.
- Update 2016-03-31:
- A new command
stbt auto-selftestwas merged to stb-tester git master and will be in the stb-tester v25 release. It’s easier to use than described above:
FrameObjecthas tests generated for it automatically.
- You don’t need to list the screenshots to be included: they all are by default.
- You don’t need to specify the input and output files: tests are generated for each of the Python files in your test-pack
See PR #343 for more information.