Add visual verification to your Selenium tests with stb-tester
20 Sep 2016.
Selenium is a test automation tool for web applications. It tests your web app by looking at the DOM – you can check that your web page contains a particular element like a div or a button, identified by id or css selector or xpath expression; once the element is located, you can check that it contains a particular text or other elements. However you can’t really check what your web app looks like. If you have complex CSS it can be all too easy to introduce unintended defects in your app’s visual appearance.
Stb-tester is a test automation tool originally developed for set-top boxes and TVs, where you don’t always have the luxury of an easily-accessible programmer-friendly representation of your GUI, such as a DOM. Stb-tester works by capturing video from the device-under-test (traditionally via an HDMI capture card) and looking for specific images in the video.
In this article we’ll be looking at stb-tester’s match function, and how to use it to test screenshots captured via Selenium. match takes a reference image and a screenshot from your app; it tells you whether the reference image is present in the screenshot, and at what location. match is available as a Python function (stbt.match) and as a command-line tool (stbt match).
- On Ubuntu: We provide deb packages. Instructions here.
- Anywhere else: Stb-tester has a lengthy set of dependencies, so the easiest way is to run it in an Ubuntu docker container.
Using stbt.match from the Selenium Python bindings
If you write your Selenium tests in Python, you can call stb-tester’s match function directly:
expected.png is the image you’re looking for – cropped from a screenshot taken previously.
get_screenshot is a function that uses Selenium to take a screenshot of your browser window, and converts it to the format that stb-tester expects (OpenCV’s format, which is a (height x width x 3) numpy array in BGR order). Here is the implementation:
(The webdriver parameter is a WebDriver object, such as selenium.webdriver.Chrome or selenium.webdriver.Firefox.)
See also: stbt.match API reference.
Using stbt match from other languages
Use your language’s version of WebDriver.get_screenshot_as_file to save the screenshot to disk. See Taking a screenshot in the Selenium documentation.
stbt match <screenshot.png> <reference.png> in a subprocess. The
exit status will be 0 (success) if a match was found, 1 (failure) otherwise.
For documentation on the command-line tool, run
stbt match --help:
stbt.match does a pixel-by-pixel comparison, so the reference image has to be the same size as the corresponding part of the screenshot. To ensure consistent behaviour across test runs on different PCs, you might need to specify certain browser parameters explicitly:
Set the browser window size with webdriver.set_window_size or the equivalent in your language’s Selenium bindings.
I like to run my Selenium tests headlessly under Xvfb so that I can control the (virtual) screen size:
xvfb-run --server-args="-screen 0 1600x1200x32" my-test-runner ...
To get the same results on High-DPI screens, I had to use Chrome’s
--force-device-scale-factor=1. In Python you do it like this:
Save a screenshot for failing tests
Whichever test-runner you are using to run your Selenium tests will surely
provide a way to run custom code after each test, or when a test fails. Here
you can save a screenshot to disk with your language’s version of
WebDriver.get_screenshot_as_file (see Taking a screenshot in the Selenium
The following aren’t related to stb-tester in any way, and listing here doesn’t constitute our endorsement. However they’re worth checking out:
Applitools eyes is a proprietary web service that integrates with Selenium and runs image-matching on Applitools’s servers in the cloud. Instead of specifying a reference image in your test scripts, Applitools eyes compares the screenshot against a screenshot from the previous test-run, and flags any differences.
Testdroid is a service for running automated tests on real Android & iOS devices managed by Testdroid in their cloud. They have written about using the AKAZE algorithm for image matching. Instead of matching pixel-by-pixel, AKAZE extracts some abstract “features” from the reference image, and looks for those features in the screenshot. These features are resolution & rotation independent, so it will match a smaller or larger version of the image (rotated or skewed too?). I don’t know how well it performs in practice.