Looking for inexact matches
In this tutorial we’ll learn how to find the location of a menu selection on screen. Once you know this information you can use OCR on that region to read the text of the currently-selected item, and you can use this to build a function that knows how to navigate the menu (as we’ll see in a later tutorial).
To match the selection regardless of the text inside the selection, we will
relax stb-tester’s default image-matching algorithm by supplying custom
MatchParameters. I’ll walk you through the workflow I use to test various
stbt match to test my code off-line (that is, without
pushing it to the test-pack git repository or needing a live set-top box), and
stbt-docker to develop on Mac or Windows.
Start with a screenshot
I’ll be using a Roku set-top box as my device-under-test throughout this tutorial, but the concepts apply to any device.
The first step is to save a screenshot from the device-under-test:
Save the screenshot in your test-pack somewhere under
(make those directories if they don’t already exist) and commit it to git. Give
the file a descriptive name, not just “screenshot”. This screenshot has the
selection on the “Home” menu entry, so I’ve called it
main-menu/home-selected.png (note that I created a subdirectory called
main-menu — you can create any directory structure you like). So the full
path of my screenshot is:
This directory is called
selftest because we will use the screenshots to test
our test scripts. This will allow us to develop our test scripts more quickly,
without having to test every change against a real device.
I’ll try to find the selection by matching the following reference image, which
I have saved under
I created this image by copying the previous screenshot to a new file. Then I cropped it so that it only contained the selection, and I copied & pasted part of the white background over the “Home” text. I used GIMP, a free cross-platform image editor.
I removed the “Home” text because I need this image to match any menu entry. (I’ll also have to relax stb-tester’s match parameters; we’ll get to that in a minute.)
When I cropped the image I kept a few pixels of the dark border around the selection. It’s important to include some structure in your reference images, because a plain white blob might match at a location you don’t expect. In this screenshot it probably doesn’t matter, but if your device-under-test shows live TV in the background, the structure is important.
We can test our reference image against our screenshot by using the
match command-line tool. (See the Appendix of this tutorial for instructions
on how to run the
stbt tools on your PC.)
$ stbt match -v \ > selftest/screenshots/main-menu/home-selected.png \ > tests/images/menu-selection.png stbt-match: Original image (720, 1280, 3), template (55, 383, 3) stbt-match: Level 2: image (180, 320, 3), template (14, 96, 3) stbt-match: Level 2: Searching in _Rect(x=0, y=0, w=225, h=167) stbt-match: Level 2: Matched at Position(x=27, y=39) with certainty 0.99172242824 stbt-match: Level 1: image (360, 640, 3), template (28, 192, 3) stbt-match: Level 1: Searching in _Rect(x=385, y=245, w=19, h=11) stbt-match: Level 1: Searching in _Rect(x=171, y=151, w=137, h=67) stbt-match: Level 1: Searching in _Rect(x=0, y=63, w=308, h=77) stbt-match: Level 1: Matched at Position(x=54, y=77) with certainty 0.985478747636 stbt-match: Level 0: image (720, 1280, 3), template (55, 383, 3) stbt-match: Level 0: Searching in _Rect(x=381, y=309, w=209, h=123) stbt-match: Level 0: Searching in _Rect(x=0, y=131, w=604, h=141) stbt-match: Level 0: Matched at Position(x=108, y=155) with certainty 0.989060676657 stbt-match: No match found. Closest match: MatchResult(timestamp=None, match=False, region=Region(x=108, y=155, right=491, bottom=210), first_pass_result=0.9890606766566634, frame=<1280x720x3>, image='tests/images/menu-selection.png') No match found. Closest match: MatchResult(timestamp=None, match=False, region=Region(x=108, y=155, right=491, bottom=210), first_pass_result=0.9890606766566634, frame=<1280x720x3>, image='tests/images/menu-selection.png')
The reference image didn’t match — I expected this, because of the “Home” text
in the screenshot that isn’t present in the reference image. To help us
stbt match has generated some debug images in the
directory under the current directory. Open
in your browser. It looks like this:
This output is showing debug images for each step in stb-tester’s image-matching algorithm. The algorithm consists of two passes:
The first pass searches the entire frame for the position that minimises the sum of squared errors (that is, the differences between the pixels in the reference image and the corresponding pixels in the frame).
As a performance optimisation, this is done on a scaled-down version of the frame & the reference image; if a match is found, it is repeated on the full resolution, but only within the regions identified as potential matches in the scaled-down version. That is why you see 3 “pyramid levels” under the first pass.
In the debug output we can see that the first pass did find a match, at the correct location:
The second pass only operates on the match position identified by the first pass. It checks that no pixel differs from the reference image by more than the default threshold.
In the debug we can see that the second pass does find differences: the “Home” text, which is present in the frame but not in the reference image.
The parameters we can use to tweak stb-tester’s image matching are documented
in the Python API reference for stbt.MatchParameters. You can also specify
them on the command line to
stbt match. Here we disable the second pass,
because we don’t want a pixel-perfect match:
$ stbt match -v \ > selftest/screenshots/main-menu/home-selected.png \ > tests/images/menu-selection.png \ > confirm_method=none [...] Match found: MatchResult(timestamp=None, match=True, region=Region(x=108, y=155, right=491, bottom=210), first_pass_result=0.9890606766566634, frame=<1280x720x3>, image='tests/images/menu-selection.png')
In most cases you should not be specifying custom match parameters — we only needed it here because we want to ignore part of the image (the text within the menu selection).
Appendix: Running the stbt tools on your PC
stbt-docker is a script that runs any Linux command in a Docker container
that is set up like an Stb-tester HDMI Node but without video-capture or
If you’re not familiar with Docker, think of it as a light-weight Virtual
Machine. The Docker container will have
stbt and all its dependencies
stbt-docker by copying it to the root of your test-pack. You’ll
need to install Docker, and on Windows you’ll need to install Python 2.7
(Macs already come with Python installed). For more details see stbt-docker
In the tutorial above, instead of running
stbt match you would run
./stbt-docker stbt match. Alternatively, you could start an interactive bash
./stbt-docker bash and then run
stbt match, like this:
$ ./stbt-docker bash stb-tester@d517c121109e:~/test-pack$ stbt match -v selftest/screenshots/main-menu/home-selected.png tests/images/menu-selection.png confirm_method=none [...] Match found: MatchResult(timestamp=None, match=True, region=Region(x=108, y=155, width=383, height=55), first_pass_result=0.989060676657, frame=1280x720x3, image='/var/lib/stbt/test-pack/tests/images/menu-selection.png')