Using Frame Objects to extract information from the screen

In this tutorial we’ll learn how to use Frame Objects to extract information from the screen. A Frame Object is a class that extracts information from a frame of video, typically by calling stbt.ocr or stbt.match. All your testcases then use these objects, instead of calling stbt.ocr or stbt.match directly. A Frame Object translates from the vocabulary of low-level image processing functions to the vocabulary of high-level features and user-facing concepts.

I’ll step you through the workflow I use to develop a Frame Object:

  1. Save screenshots under selftest/screenshots/.
  2. Implement a stbt.FrameObject subclass that extracts the required information from a frame of video.
  3. Test your Python code with ./stbt-docker stbt lint.
  4. Test your Frame Objects against your screenshots with ./stbt-docker stbt auto-selftest generate.
  5. Commit your code, screenshots, and the generated selftests to git.
  6. Re-run ./stbt-docker stbt auto-selftest generate whenever you change test code or when you add new screenshots.

Start with a screenshot

Similarly to the previous tutorial, we’ll be looking at the Roku’s main menu. I’m using a Roku set-top box as my device-under-test, but the concepts in this tutorial apply to any device.

The first step is to save a screenshot from the device-under-test into selftest/screenshots/ in your test-pack. Give the file a descriptive name, not just “screenshot.png”. You can create subdirectories to organise your screenshots. Here I’ve saved a screenshot of the Roku’s main menu as selftest/screenshots/main-menu/home-selected.png:

Extracting information from the screen with a Frame Object

I want to know which menu item is currently selected. We can extract that information by running stbt.ocr on the region where the current selection (highlight) is.

Here is the implementation of my main_menu.Selection Frame Object. It represents the current selection of the Roku’s main menu. A Selection instance takes a single frame of video; it has properties (such as text) that extract information about the menu selection in that frame.

from stbt import FrameObject, match, MatchParameters, ocr, Region

class Selection(FrameObject):
    @property
    def is_visible(self):
        mp = MatchParameters(confirm_method="none")
        return bool(match("menu-selection.png", frame=self._frame,
                          match_parametes=mp))

    @property
    def text(self):
        return ocr(frame=self._frame,
                   region=Region(x=120, y=164, right=470, bottom=200))

tests/main_menu.py

Let’s walk through this class line by line:

class Selection(FrameObject):

Our class inherits from stbt.FrameObject. This means that our class takes a frame of video when it is constructed, and it automatically behaves in certain ways (we’ll see some of that behaviour below).

@property
def is_visible(self):
    mp = MatchParameters(confirm_method="none")
    return bool(match("menu-selection.png", frame=self._frame,
                      match_parametes=mp))

Every Frame Object must define a property called is_visible. It should return True or False. This Selection Frame Object should be considered visible if the Roku’s main menu is visible on screen. Here we are checking that the menu selection (highlight) is visible (see the previous tutorial for details on the reference image and the match parameters we’ve used here).

Note that when we call match, we pass in frame=self._frame. This is very important! It means that the match is performed on the frame that we attached to our Frame Object when we created it.

A Frame Object is considered truthy if is_visible is True. For example, if we have an object s that is an instance of our Selection class:

s = Selection()

…then the following two if statements are equivalent:

if s.is_visible:
    # do something
if s:
    # do something

Now let’s look at the text property. It tells us which menu item is currently selected, by using stbt.ocr:

@property
def text(self):
    return ocr(frame=self._frame,
               region=Region(x=120, y=164, right=470, bottom=200))

The Roku’s main menu uses “fixed focus” – as you scroll up or down, the menu text moves but the selection stays in the same place. So I’ve just hard-coded the Region where the selection always is. If your menu’s selection moves around, you could use stbt.match to find the selection’s position (the MatchResult has a region parameter that tells you the position where it matched).

Again, note that when we call ocr we pass in frame=self._frame.

Public properties (like text) will automatically return None if the Frame Object isn’t visible. This avoids having to write awkward code like if s.is_visible and s.text == "Home". All properties of a Frame Object will be cached the first time they’re used, so if you use a property more than once it won’t need to re-compute the result of ocr.

For more details see the Python API reference for stbt.FrameObject.

Testing our code: Static analysis with stbt lint

Before we try running the code, it is useful to run a static analyser. This can find typos and other bugs, without actually running the code (that’s why it’s called “static” analysis). For Python code, the best such tool is pylint.

stbt lint is a tool that uses pylint, and also adds a few checks that are unique to stb-tester scripts (for example, it checks that the reference images you give to stbt.match actually exist on disk).

Here’s how to run it:

$ stbt lint --errors-only tests/main_menu.py
No config file found, using default configuration
************* Module main_menu
E: 8,27: Passing unexpected keyword argument 'match_parametes' in function call (unexpected-keyword-arg)

This is telling us that there’s an error at line 8, column 27, of main_menu.py. Indeed, I made a typo. This is the fix:

@@ class Selection(FrameObject):
     @property
     def is_visible(self):
         mp = MatchParameters(confirm_method="none")
         return bool(match("menu-selection.png", frame=self._frame,
-                           match_parametes=mp))
+                           match_parameters=mp))

And stbt lint is happy now:

$ stbt lint --errors-only tests/main_menu.py
No config file found, using default configuration

To run stbt lint and the other stbt tools, see the Appendix in the previous tutorial.

Testing our code: Automatic regression tests with stbt auto-selftest

This is where the benefits of stb-tester’s Frame Objects will become clearer. All you have to do is run stbt auto-selftest generate; this will test all your Frame Objects against all your screenshots. It will generate a selftest file under selftest/auto_selftest/ for every Python file that’s under tests/. Here is selftest/auto_selftest/tests/main_menu.py:

def auto_selftest_Selection():
    r"""
    >>> Selection(frame=f("main-menu/home-selected.png"))
    Selection(is_visible=True, text=u'Home')
    """
    pass

It tested our Selection Frame Object against the home-selected.png screenshot that you saw earlier in the tutorial, and it recorded the result.

Commit the generated files to git, so that we can see the effects of future changes by re-running stbt auto-selftest generate followed by git diff.

FrameObject and stbt auto-selftest are a powerful combination. They allow us to quickly develop functionality offline (without testing every change against a real set-top box, which is slow), and they allow us to make significant changes and refactorings with confidence that we haven’t broken existing functionality.

Using the Frame Object

In your testcases, you might use the Selection Frame Object like this:

from stbt import press, wait_until
import main_menu
...
assert main_menu.Selection().text == "Home"
press("KEY_DOWN")
assert wait_until(lambda: main_menu.Selection().text == "My Feed")

Continuous Integration for your test scripts

It is a good idea to run stbt lint and stbt auto-selftest every time a developer makes a change to the test-pack. It’s even better to do this automatically from a Continuous Integration (CI) system, like Jenkins. We are test automation developers, after all.

With stbt-docker it is very easy to set this up. On your CI server you only need to install Python and Docker; stbt-docker takes care of installing all your test-pack’s dependencies. (See the Appendix in the previous tutorial for more details on stbt-docker.)

In CI you should use stbt auto-selftest validate instead of stbt auto-selftest generate. validate doesn’t change any files in your test-pack, and it returns a useful exit status to determine whether there were any unexpected changes in behaviour.

Next steps

In the next tutorial we will use our Selection Frame Object to implement a function that knows how to navigate around the menu.