How to structure your test scripts for re-use across multiple platforms#
If you want to test the same app on several different platforms, you want to re-use as much test-script code as possible. For example you might have your app on Android TV, Apple TV, Fire TV, PlayStation, Roku, XBox, and possibly even some operator-specific set-top box platforms. This article shows how to use the same test scripts and the same Page Objects for all these platforms, and how to manage the cases where your app has differences in appearance or behaviour across platforms.
In this article we’ll use YouTube as the app-under-test, for demonstration purposes. Our recommended structure will also work if you have several different apps to test.
Directory structure#
We’ll use the following directory structure:
Configuration files#
In the Node-specific configuration files in your test-pack, specify the type of device that is connected to each Stb-tester Node as device_under_test.device_type. For example:
[device_under_test]
device_type = appletv
The name (appletv) should match the directory name from the directory structure we saw earlier.
Your test scripts can read this configuration with stbt.get_config. See the next section for an example.
Launching your app#
Typically your tests will start like this:
from .pages.home import launch_youtube
def test_play_featured_content():
launch_youtube()
...
launch_youtube is a function that knows how to launch the app on any of your supported platforms. It looks like this:
import stbt
def launch_youtube():
device_type = stbt.get_config("device_under_test", "device_type")
if device_type == "androidtv":
from tests.androidtv.home import launch_app
elif device_type == "appletv":
from tests.appletv.home import launch_app
elif ... # and so on for each of your supported devices
...
else:
raise stbt.ConfigurationError(
"Unknown device_under_test.device_type: %s" % (device_type,))
launch_app("YouTube")
You need to implement each launch_app function for each of your supported platforms. Then, once your test is inside the app, it’ll work the same on all devices and you won’t need to check device_type again (assuming your app has the same GUI on all the platforms; read on for ways to handle minor & major differences).
Page Objects#
class Search(stbt.FrameObject):
@property
def is_visible(self):
return stbt.match("Search.png",
frame=self._frame)
@property
def results(self):
... # implementation not shown
def enter_text(self, text):
... # implementation not shown
Page Objects are classes (defined by you) that have properties to read information from the screen and methods to interact with the application-under-test. For example a Page Object for YouTube’s search page might have a property called “results” that returns a list of the visible titles, and a method called “enter_text” that navigates the on-screen keyboard and types in the specified text. All Stb-tester Page Objects must also have a property called “is_visible” that returns True if the relevant page is present on the screen.
As a general rule, your test scripts shouldn’t read something from the screen by calling stbt.match or stbt.ocr directly. Instead, they should call a property of a Page Object (like page.results or page.is_visible from the example above).
Similarly, your test scripts shouldn’t interact with the device-under-test by calling stbt.press; instead, they should call a method of the Page Object, like page.enter_text(“peppa pig”).
By encapsulating the low-level image-processing operations inside your Page Objects, your test-scripts will be more maintainable and easier to read. Of Specific interest to this article, the Page Objects provide a place to encapsulate any device-specific differences — in a similar way to the launch_youtube function that we saw earlier.
To learn more about Page Objects see Object Repository in the Stb-tester manual.
Page Objects directory structure#
tests/
<app>/
pages/
__init__.py
<page_name>/
__init__.py
<page_name>.py
<reference_image>.png
<another_page>/
__init__.py
<another_page>.py
<reference_image>.png
For example, for the YouTube search page:
tests/
youtube/
pages/
__init__.py # empty
search/
__init__.py # from .search import Search
search.py
Search.png
This directory structure is created automatically when you use
in the Object Repository.
Reference images#
Reference images used with the stbt.match API should live in the same directory as the Python code that uses them (the Page Object). In the example above, Search.png used by the Page Object property Search.is_visible lives in youtube/pages/search/.
Do keep reference images next to the code that uses them.
Don’t put all reference images together in a top-level “images” directory.
Don’t write an “Images” class or Python constants for each image. It’s just extra boilerplate that adds maintenance burden for no extra value. If you’re worried about making a typo in the filename, don’t worry: Stb-tester’s lint tool automatically checks filenames in the strings given to stbt.match.
Object Repository screenshots#
The Object Repository in the Stb-tester Portal shows your Page Object’s behaviour using the screenshots committed to git in selftest/screenshots/. These screenshots serve as documentation (examples of the Page Object’s behaviour on each platform) but they’re also a development tool: It is faster to create or modify Page Object classes by testing against a screenshot in the Object Repository, than it is to run a test script on a live device.
These screenshots are organised in a similar directory structure but under selftest/screenshots/ instead of tests/:
Note that the app names are lowercase like Python module names, but page names are capitalised like Python class names, as defined by PEP-8 (Python’s standard style guide).
You can have any amount of subdirectories under each page’s directory. When testing a particular screenshot, the Object Repository will use the configuration from any stbt.conf files in the screenshot’s directory, and any parent directories. For example, youtube/Search/appletv/stbt.conf would contain the device_type configuration:
[device_under_test]
device_type = appletv
…and youtube/Search/germany/stbt.conf would contain the OCR configuration:
[ocr]
lang = de
The Object Repository will merge this configuration with the global configuration from .stbt.conf at the root of your test-pack repository; but it won’t use the configuration from Node-specific configuration files.
Minor differences in app appearance and behaviour#
Let’s say that your page looks slightly different on 2 different platforms —different enough that you’ll require 2 different reference images— but the interaction is the same. There are 2 ways you can handle this in your Page Object code: Accept both looks, or check the configured device_type and select the reference image accordingly.
The first approach looks like this:
class Search(stbt.FrameObject):
@property
def is_visible(self):
return (stbt.match("Search.png", frame=self._frame) or
stbt.match("Search-appletv.png", frame=self._frame))
The second approach looks like this:
device_type = stbt.get_config("device_under_test", "device_type")
class Search(stbt.FrameObject):
@property
def is_visible(self):
if device_type == "appletv":
return stbt.match("Search-appletv.png", frame=self._frame)
else:
# Common appearance on the other platforms:
return stbt.match("Search.png", frame=self._frame)
The first approach is also suitable for handling different versions or releases of the same app (do remember to delete the old code & old reference images once the old version is no longer relevant).
The second approach will be faster if you have a large number of different images to check.
Major differences in app behaviour#
If your app is significantly different on one platform —it looks different and the behaviour is different too— then you should treat it as a different app. Typically this happens on platforms like Chromecast where the nature of the platform means that the interaction model is completely different; or with major re-designs of an app (not mere cosmetic re-skins); or with “legacy” versions of an app on some older platforms.
In this case, instead of the tests/youtube directory you would have a separate directory such as tests/youtube_chromecast with its own test scripts and its own Page Objects.
What are those _init_.py files?#
The _init_.py files tell Python that the various directories and Python modules are all part of the same “package”. This allows you to use relative imports like from .pages.search import Search (from a test script), or from ..search import Search (from a sibling Page Object). See Packages in the official Python.org tutorial.
Most of the _init_.py files are empty to avoid potential problems with circular dependencies. The only exception is the last _init_.py in each Page Object directory. For example:
from .search import Search
This saves a small amount of boilerplate when importing the Page Object (from .pages.search import Search instead of from .pages.search.search import Search).
Technically, you could define the Page Object directly in search/_init_.py instead of search/search.py, but in your IDE you’ll appreciate having the file name match the page name (imagine 5 different tabs all named _init_.py).
Rationale#
We want to re-use the same test-scripts as much as possible for testing the same app on different platforms. The benefits of code re-use should be obvious.
Small differences in appearance & behaviour across platforms should be encapsulated in Page Objects. This allows the actual test scripts to focus on the high-level user-facing concepts of the application under test. It’s more maintainable because many different tests will use the same Page Objects, but you’ll only have to handle platform differences once (in the Page Object).
Static configuration (committed to git) specifies which device_type is connected to each Stb-tester Node. Don’t be tempted to write scripts that figure out dynamically (at run time) what device they’re looking at: errors in this type of logic are hard to debug because the script will proceed with the wrong assumptions, and only fail later in a seemingly unrelated part of your code.
You should never (or rarely) check the configured device_type explicitly in your tests; this should be encapsulated in helper functions (like the launch_youtube example earlier) and Page Objects.
Store reference images in the same directory as the Python file that uses them. Typically this will be the directory of a Page Object. This reduces coupling; the users of each image (Python classes and functions) are clear, and obsolete (unused) images are easy to identify.
Following these standards will allow your Python code to work with the current & future tooling provided in the Stb-tester Portal (such as the Object Repository).
Questions?#
Please contact support@stb-tester.com if you have any questions or feedback.