20 Sep 2016.

Selenium is a test automation tool for web applications. It tests your web app by looking at the DOM – you can check that your web page contains a particular element like a div or a button, identified by id or css selector or xpath expression; once the element is located, you can check that it contains a particular text or other elements. However you can’t really check what your web app looks like. If you have complex CSS it can be all too easy to introduce unintended defects in your app’s visual appearance.

assert match("guide-selected.png",
             get_screenshot(webdriver))

Stb-tester is a test automation tool originally developed for set-top boxes and TVs, where you don’t always have the luxury of an easily-accessible programmer-friendly representation of your GUI, such as a DOM. Stb-tester works by capturing video from the device-under-test (traditionally via an HDMI capture card) and looking for specific images in the video.

In this article we’ll be looking at stb-tester’s match function, and how to use it to test screenshots captured via Selenium. match takes a reference image and a screenshot from your app; it tells you whether the reference image is present in the screenshot, and at what location. match is available as a Python function (stbt.match) and as a command-line tool (stbt match).

Installing stb-tester

  • On Ubuntu: We provide deb packages. Instructions here.
  • Anywhere else: Stb-tester has a lengthy set of dependencies, so the easiest way is to run it in an Ubuntu docker container.

Using stbt.match from the Selenium Python bindings

If you write your Selenium tests in Python, you can call stb-tester’s match function directly:

import stbt
assert stbt.match("expected.png", get_screenshot(webdriver))

expected.png is the image you’re looking for – cropped from a screenshot taken previously.

get_screenshot is a function that uses Selenium to take a screenshot of your browser window, and converts it to the format that stb-tester expects (OpenCV’s format, which is a (height x width x 3) numpy array in BGR order). Here is the implementation:

def get_screenshot(webdriver):
    # https://stackoverflow.com/questions/11552926
    return cv2.imdecode(
        numpy.asarray(
            bytearray(base64.decodestring(webdriver.get_screenshot_as_base64())),
            dtype=numpy.uint8),
        cv2.CV_LOAD_IMAGE_UNCHANGED)

(The webdriver parameter is a WebDriver object, such as selenium.webdriver.Chrome or selenium.webdriver.Firefox.)

See also: stbt.match API reference.

Using stbt match from other languages

Use your language’s version of WebDriver.get_screenshot_as_file to save the screenshot to disk. See Taking a screenshot in the Selenium documentation.

Then run stbt match <screenshot.png> <reference.png> in a subprocess. The exit status will be 0 (success) if a match was found, 1 (failure) otherwise.

For documentation on the command-line tool, run stbt match --help:

$ stbt match --help
usage: stbt match [-h] [-v]
                  source_file template_file
                  [match_parameters [match_parameters ...]]

Run stbt's image-matching algorithm against a single frame (which you can
capture using `stbt screenshot`).

positional arguments:
  source_file       The screenshot to compare against (you can capture it
                    using 'stbt screenshot')
  template_file     The image to search for
  match_parameters  Parameters for the image processing algorithm. See
                    'MatchParameters' in the stbt API documentation. For
                    example: 'confirm_threshold=0.20')

optional arguments:
  -h, --help        show this help message and exit
  -v, --verbose     Dump image processing debug images to ./stbt-debug
                    directory

Window size

stbt.match does a pixel-by-pixel comparison, so the reference image has to be the same size as the corresponding part of the screenshot. To ensure consistent behaviour across test runs on different PCs, you might need to specify certain browser parameters explicitly:

  • Set the browser window size with webdriver.set_window_size or the equivalent in your language’s Selenium bindings.

  • I like to run my Selenium tests headlessly under Xvfb so that I can control the (virtual) screen size:

    xvfb-run --server-args="-screen 0 1600x1200x32" my-test-runner ...

  • To get the same results on High-DPI screens, I had to use Chrome’s --force-device-scale-factor=1. In Python you do it like this:

import selenium
options = selenium.webdriver.ChromeOptions()
options.add_argument("--force-device-scale-factor=1")
webdriver = selenium.webdriver.Chrome(chrome_options=options)

Save a screenshot for failing tests

Whichever test-runner you are using to run your Selenium tests will surely provide a way to run custom code after each test, or when a test fails. Here you can save a screenshot to disk with your language’s version of WebDriver.get_screenshot_as_file (see Taking a screenshot in the Selenium documentation).

If you already have a screenshot in stb-tester/OpenCV format (for example, the stbt.MatchResult returned by stbt.match has a frame attribute) you can save it with the OpenCV function cv2.imwrite.

The following aren’t related to stb-tester in any way, and listing here doesn’t constitute our endorsement. However they’re worth checking out:

  • Applitools eyes is a proprietary web service that integrates with Selenium and runs image-matching on Applitools’s servers in the cloud. Instead of specifying a reference image in your test scripts, Applitools eyes compares the screenshot against a screenshot from the previous test-run, and flags any differences.

  • Testdroid is a service for running automated tests on real Android & iOS devices managed by Testdroid in their cloud. They have written about using the AKAZE algorithm for image matching. Instead of matching pixel-by-pixel, AKAZE extracts some abstract “features” from the reference image, and looks for those features in the screenshot. These features are resolution & rotation independent, so it will match a smaller or larger version of the image (rotated or skewed too?). I don’t know how well it performs in practice.