Looking for inexact matches

In this tutorial we’ll learn how to find the location of a menu selection on screen. Once you know this information you can use OCR on that region to read the text of the currently-selected item, and you can use this to build a function that knows how to navigate the menu (as we’ll see in a later tutorial).

To match the selection regardless of the text inside the selection, we will relax stb-tester’s default image-matching algorithm by supplying custom MatchParameters. I’ll walk you through the workflow I use to test various parameters, using stbt match to test my code off-line (that is, without pushing it to the test-pack git repository or needing a live set-top box), and using stbt-docker to develop on Mac or Windows.

Start with a screenshot

I’ll be using a Roku set-top box as my device-under-test throughout this tutorial, but the concepts apply to any device.

The first step is to save a screenshot from the device-under-test:

Save the screenshot in your test-pack somewhere under selftest/screenshots/ (make those directories if they don’t already exist) and commit it to git. Give the file a descriptive name, not just “screenshot”. This screenshot has the selection on the “Home” menu entry, so I’ve called it main-menu/home-selected.png (note that I created a subdirectory called main-menu — you can create any directory structure you like). So the full path of my screenshot is: selftest/screenshots/main-menu/home-selected.png.

This directory is called selftest because we will use the screenshots to test our test scripts. This will allow us to develop our test scripts more quickly, without having to test every change against a real device.

Reference image

I’ll try to find the selection by matching the following reference image, which I have saved under tests/images/menu-selection.png:

I created this image by copying the previous screenshot to a new file. Then I cropped it so that it only contained the selection, and I copied & pasted part of the white background over the “Home” text. I used GIMP, a free cross-platform image editor.

I removed the “Home” text because I need this image to match any menu entry. (I’ll also have to relax stb-tester’s match parameters; we’ll get to that in a minute.)

When I cropped the image I kept a few pixels of the dark border around the selection. It’s important to include some structure in your reference images, because a plain white blob might match at a location you don’t expect. In this screenshot it probably doesn’t matter, but if your device-under-test shows live TV in the background, the structure is important.

stbt match

We can test our reference image against our screenshot by using the stbt match command-line tool. (See the Appendix of this tutorial for instructions on how to run the stbt tools on your PC.)

$ stbt match -v \
>    selftest/screenshots/main-menu/home-selected.png \
>    tests/images/menu-selection.png
stbt-match: Original image (720, 1280, 3), template (55, 383, 3)
stbt-match: Level 2: image (180, 320, 3), template (14, 96, 3)
stbt-match: Level 2: Searching in _Rect(x=0, y=0, w=225, h=167)
stbt-match: Level 2: Matched at Position(x=27, y=39) with certainty 0.99172242824
stbt-match: Level 1: image (360, 640, 3), template (28, 192, 3)
stbt-match: Level 1: Searching in _Rect(x=385, y=245, w=19, h=11)
stbt-match: Level 1: Searching in _Rect(x=171, y=151, w=137, h=67)
stbt-match: Level 1: Searching in _Rect(x=0, y=63, w=308, h=77)
stbt-match: Level 1: Matched at Position(x=54, y=77) with certainty 0.985478747636
stbt-match: Level 0: image (720, 1280, 3), template (55, 383, 3)
stbt-match: Level 0: Searching in _Rect(x=381, y=309, w=209, h=123)
stbt-match: Level 0: Searching in _Rect(x=0, y=131, w=604, h=141)
stbt-match: Level 0: Matched at Position(x=108, y=155) with certainty 0.989060676657
stbt-match: No match found. Closest match: MatchResult(timestamp=None, match=False, region=Region(x=108, y=155, right=491, bottom=210), first_pass_result=0.9890606766566634, frame=<1280x720x3>, image='tests/images/menu-selection.png')
No match found. Closest match: MatchResult(timestamp=None, match=False, region=Region(x=108, y=155, right=491, bottom=210), first_pass_result=0.9890606766566634, frame=<1280x720x3>, image='tests/images/menu-selection.png')

The reference image didn’t match — I expected this, because of the “Home” text in the screenshot that isn’t present in the reference image. To help us investigate, stbt match has generated some debug images in the stbt-debug directory under the current directory. Open stbt-debug/match/00001/index.html in your browser. It looks like this:

This output is showing debug images for each step in stb-tester’s image-matching algorithm. The algorithm consists of two passes:

  1. The first pass searches the entire frame for the position that minimises the sum of squared errors (that is, the differences between the pixels in the reference image and the corresponding pixels in the frame).

    As a performance optimisation, this is done on a scaled-down version of the frame & the reference image; if a match is found, it is repeated on the full resolution, but only within the regions identified as potential matches in the scaled-down version. That is why you see 3 “pyramid levels” under the first pass.

    In the debug output we can see that the first pass did find a match, at the correct location:

  2. The second pass only operates on the match position identified by the first pass. It checks that no pixel differs from the reference image by more than the default threshold.

    In the debug we can see that the second pass does find differences: the “Home” text, which is present in the frame but not in the reference image.

Match parameters

The parameters we can use to tweak stb-tester’s image matching are documented in the Python API reference for stbt.MatchParameters. You can also specify them on the command line to stbt match. Here we disable the second pass, because we don’t want a pixel-perfect match:

$ stbt match -v \
>    selftest/screenshots/main-menu/home-selected.png \
>    tests/images/menu-selection.png \
>    confirm_method=none
[...]
Match found: MatchResult(timestamp=None, match=True, region=Region(x=108, y=155, right=491, bottom=210), first_pass_result=0.9890606766566634, frame=<1280x720x3>, image='tests/images/menu-selection.png')

Success!

In most cases you should not be specifying custom match parameters — we only needed it here because we want to ignore part of the image (the text within the menu selection).

Appendix: Running the stbt tools on your PC

stbt-docker is a script that runs any Linux command in a Docker container that is set up like an Stb-tester HDMI Node but without video-capture or infrared hardware.

If you’re not familiar with Docker, think of it as a light-weight Virtual Machine. The Docker container will have stbt and all its dependencies installed.

You install stbt-docker by copying it to the root of your test-pack. You’ll need to install Docker, and on Windows you’ll need to install Python 2.7 (Macs already come with Python installed). For more details see stbt-docker on github.

In the tutorial above, instead of running stbt match you would run ./stbt-docker stbt match. Alternatively, you could start an interactive bash shell with ./stbt-docker bash and then run stbt match, like this:

$ ./stbt-docker bash
stb-tester@d517c121109e:~/test-pack$ stbt match -v selftest/screenshots/main-menu/home-selected.png tests/images/menu-selection.png confirm_method=none
[...]
Match found: MatchResult(timestamp=None, match=True, region=Region(x=108, y=155, width=383, height=55), first_pass_result=0.989060676657, frame=1280x720x3, image='/var/lib/stbt/test-pack/tests/images/menu-selection.png')