Improve black-box testing agility: meet the Frame Object pattern
08 Sep 2015. By William Manley.
In the previous blog post we discussed the Page Object pattern and how it could be used to improve the readability of your test scripts. In this blog post we introduce the Frame Object pattern, a specialisation of the Page Object pattern that can be especially effective in stb-tester’s style of black-box UI testing1. Frame Objects can reduce the maintenance cost of your test-packs, particularly when the UI-under-test changes.
A Frame Object is a class that extracts information from a frame of video,
typically by calling
stbt.match(). All the rest of your
testing code then uses these objects. A Frame Object translates from the
vocabulary of low-level image processing functions and regions (like
stbt.ocr(region=stbt.Region(213, 23, 200, 36))) to the vocabulary of
high-level features and user-facing concepts (like
Here’s an example of a Frame Object (
GuideFrame) and a test-case that uses it:
This will seem very much like a Page Object – it acts as a layer of abstraction on top of the low-level image processing functions. Unlike a Page Object it contains no behaviours or any means to stimulate the device under test:
- There are no implicit dependencies on time or external stimulus once constructed.
- There no calls to
stbt.get_frame()(except in the constructor). We pass
self.frameinto every call of
stbt.ocr()or any other image-processing function.
- We perform no image-processing or other expensive work in the constructor - this object is very cheap to construct.
- The object is immutable – there are no methods to change the externally visible state of the object. If you want an updated view of the system-under-test’s state you can construct a new object for each frame.
- There are no calls to
time.sleep()- this object can’t change so there is no point waiting.
- There are no calls to
stbt.press()or anything else that affects the device under test.
These may seem like arbitrary or unreasonable restrictions but they provide some great benefits:
Testability: Unlike full Page Objects, Frame Objects can be tested very cheaply. All you need is a corpus of example screenshots. This is because Frame Objects behave in an entirely deterministic way based on the parameters provided to the constructor (typically the only parameter is a frame of video). Most significantly, Frame Objects don’t depend on the dynamic behaviour of the device-under-test which is far more difficult to capture than a few screenshots.
Agility: Because of their improved testability, Frame Objects are cheap to maintain and easy to evolve. If the cosmetics of the application under test change, or you find a bug in your Frame Object, just capture a screenshot of that situation and add it to your test corpus. You can then fix it with confidence:
- Confidence that the updated Frame Object fixes the issue – you can verify this by applying your new Frame Object to the new screenshots added to your corpus.
- Confidence that you haven’t introduced regressions with your fixes – you can see that your Frame Object behaves the same when run against your existing corpus of screenshots.
- Confidence that you haven’t changed your expectations of the behaviour of the device-under-test – the Frame Object contains no behavioural expectations.
stbt.ocrmultiple times to extract information from a frame can lead to confusing inconsistencies if each function sees a different frame. The Frame Object Pattern avoids this by capturing a frame exactly once.
Performance: Frame Objects extract required information from the frame when needed, rather than in advance. This can avoid doing a lot of expensive image matching or OCR when it’s not necessary. Because the object is immutable and each property or method on the object is deterministic you can apply memoisation, allowing you to write tests in a natural manner without a performance penalty.
So that’s all well and good, but what about the behaviours that you removed? After all they are usually the thing you want to test in the underlying device. It turns out that the Page Object and the Frame Object patterns are actually complementary rather than conflicting. Where necessary you can still have page objects, but they use Frame Objects to extract the information from the underlying frames.
Two classes rather than one sounds complicated? In our experience the page object is no longer necessary in most cases and you can get away with just using helper functions or using the Frame Objects directly from test cases. But when you have more complex interactions and your helper functions need to store state you can still create page objects. These page objects will use Frame Objects: the additional overall complexity is more than offset by the reduction in complexity in the more expensive to maintain page objects – an overall win for maintainability.
For example here’s a helper function that works in conjunction with the above Frame Object:
So: the look of your device is captured by frame objects and the behaviour of the device is captured by helper functions and page objects. This works out very well; in our experience the look of a UI will tend to change a lot during development, whereas the underlying behaviour changes at a much slower rate.
Frame Objects checklist:
- Take a frame in the constructor.
- Provide properties describing the contents of that frame.
- Contain no actions or behaviours.
- Include a
__repr__method that prints as much data from the frame as possible to enable use with doctests and logging.
- Include a
__nonzero__method dictating whether the current frame is suitable – for use with
We’re considering adding specific support for Frame Objects in stb-tester to help reduce the amount of boilerplate required.
- Update 2016-03-10:
FrameObjectbase class was merged to stb-tester git master and will be in the stb-tester v25 release. It reduces the boilerplate involved in Frame Objects by taking a frame in the constructor, and defining
__nonzero__for you. So now all you have to do is derive from
stbt.FrameObjectand define some properties. See PR #332 for more information.
By “black-box UI testing” we mean that you’re testing what the user sees, by capturing a video stream of the rendered UI and using image processing to check it. ↩