Using Frame Objects to extract information from the screen
In this tutorial we’ll learn how to use Frame Objects to extract information
from the screen. A Frame Object is a class that extracts information from a
frame of video, typically by calling
stbt.match. All your
testcases then use these objects, instead of calling
directly. A Frame Object translates from the vocabulary of low-level image
processing functions to the vocabulary of high-level features and user-facing
I’ll step you through the workflow I use to develop a Frame Object:
- Save screenshots under
- Implement a stbt.FrameObject subclass that extracts the required information from a frame of video.
- Test your Python code with
./stbt-docker stbt lint.
- Test your Frame Objects against your screenshots with
./stbt-docker stbt auto-selftest generate.
- Commit your code, screenshots, and the generated selftests to git.
./stbt-docker stbt auto-selftest generatewhenever you change test code or when you add new screenshots.
Start with a screenshot
Similarly to the previous tutorial, we’ll be looking at the Roku’s main menu. I’m using a Roku set-top box as my device-under-test, but the concepts in this tutorial apply to any device.
The first step is to save a screenshot from the device-under-test into
selftest/screenshots/ in your test-pack. Give the file a descriptive name,
not just “screenshot.png”. You can create subdirectories to organise your
screenshots. Here I’ve saved a screenshot of the Roku’s main menu as
Extracting information from the screen with a Frame Object
I want to know which menu item is currently selected. We can extract that information by running stbt.ocr on the region where the current selection (highlight) is.
Here is the implementation of my
main_menu.Selection Frame Object. It
represents the current selection of the Roku’s main menu. A
instance takes a single frame of video; it has properties (such as
extract information about the menu selection in that frame.
from stbt import FrameObject, match, MatchParameters, ocr, Region class Selection(FrameObject): @property def is_visible(self): mp = MatchParameters(confirm_method="none") return bool(match("menu-selection.png", frame=self._frame, match_parametes=mp)) @property def text(self): return ocr(frame=self._frame, region=Region(x=120, y=164, right=470, bottom=200))
Let’s walk through this class line by line:
Our class inherits from
stbt.FrameObject. This means that our class takes a
frame of video when it is constructed, and it automatically behaves in certain
ways (we’ll see some of that behaviour below).
@property def is_visible(self): mp = MatchParameters(confirm_method="none") return bool(match("menu-selection.png", frame=self._frame, match_parametes=mp))
Every Frame Object must define a property called
is_visible. It should return
True or False. This
Selection Frame Object should be considered visible if
the Roku’s main menu is visible on screen. Here we are checking that the menu
selection (highlight) is visible (see the
previous tutorial for details on the reference
image and the match parameters we’ve used here).
Note that when we call
match, we pass in
frame=self._frame. This is very
important! It means that the match is performed on the frame that we attached
to our Frame Object when we created it.
A Frame Object is considered truthy if
is_visible is True. For example, if we
have an object
s that is an instance of our
s = Selection()
…then the following two
if statements are equivalent:
if s.is_visible: # do something
if s: # do something
Now let’s look at the
text property. It tells us which menu item is currently
selected, by using stbt.ocr:
@property def text(self): return ocr(frame=self._frame, region=Region(x=120, y=164, right=470, bottom=200))
The Roku’s main menu uses “fixed focus” – as you scroll up or down, the menu
text moves but the selection stays in the same place. So I’ve just hard-coded
the Region where the selection always is. If your menu’s selection moves
around, you could use stbt.match to find the selection’s position (the
MatchResult has a
region parameter that tells you the position where it
Again, note that when we call
ocr we pass in
Public properties (like
text) will automatically return
None if the Frame
Object isn’t visible. This avoids having to write awkward code like
s.is_visible and s.text == "Home". All properties of a Frame Object will be
cached the first time they’re used, so if you use a property more than once it
won’t need to re-compute the result of
For more details see the Python API reference for stbt.FrameObject.
Testing our code: Static analysis with stbt lint
Before we try running the code, it is useful to run a static analyser. This can find typos and other bugs, without actually running the code (that’s why it’s called “static” analysis). For Python code, the best such tool is pylint.
stbt lint is a tool that uses pylint, and also adds a few checks that are
unique to stb-tester scripts (for example, it checks that the reference images
you give to
stbt.match actually exist on disk).
Here’s how to run it:
$ stbt lint --errors-only tests/main_menu.py No config file found, using default configuration ************* Module main_menu E: 8,27: Passing unexpected keyword argument 'match_parametes' in function call (unexpected-keyword-arg)
This is telling us that there’s an error at line 8, column 27, of
main_menu.py. Indeed, I made a typo. This is the fix:
@@ class Selection(FrameObject): @property def is_visible(self): mp = MatchParameters(confirm_method="none") return bool(match("menu-selection.png", frame=self._frame, - match_parametes=mp)) + match_parameters=mp))
stbt lint is happy now:
$ stbt lint --errors-only tests/main_menu.py No config file found, using default configuration
stbt lint and the other
stbt tools, see the Appendix in the
Testing our code: Automatic regression tests with stbt auto-selftest
This is where the benefits of stb-tester’s Frame Objects will become clearer.
All you have to do is run
stbt auto-selftest generate; this will test all
your Frame Objects against all your screenshots. It will generate a selftest
selftest/auto_selftest/ for every Python file that’s under
tests/. Here is
def auto_selftest_Selection(): r""" >>> Selection(frame=f("main-menu/home-selected.png")) Selection(is_visible=True, text=u'Home') """ pass
It tested our
Selection Frame Object against the
screenshot that you saw earlier in the tutorial, and it recorded the result.
Commit the generated files to git, so that we can see the effects of future
changes by re-running
stbt auto-selftest generate followed by
stbt auto-selftest are a powerful combination. They allow
us to quickly develop functionality offline (without testing every change
against a real set-top box, which is slow), and they allow us to make
significant changes and refactorings with confidence that we haven’t broken
Using the Frame Object
In your testcases, you might use the
Selection Frame Object like this:
from stbt import press, wait_until import main_menu ... assert main_menu.Selection().text == "Home" press("KEY_DOWN") assert wait_until(lambda: main_menu.Selection().text == "My Feed")
Continuous Integration for your test scripts
It is a good idea to run
stbt lint and
stbt auto-selftest every time a
developer makes a change to the test-pack. It’s even better to do this
automatically from a Continuous Integration (CI) system, like Jenkins. We are
test automation developers, after all.
stbt-docker it is very easy to set this up. On your CI server you only
need to install Python and Docker;
stbt-docker takes care of installing all
your test-pack’s dependencies. (See the Appendix in the previous tutorial
for more details on
In CI you should use
stbt auto-selftest validate instead of
validate doesn’t change any files in your test-pack,
and it returns a useful exit status to determine whether there were any
unexpected changes in behaviour.
In the next tutorial we will use our
Object to implement a function that knows how to navigate around the menu.