Before today, to detect which page of the application is currently visible, your test script would use Stb-tester APIs such as image-matching to detect a particular reference image; but when the page is very dynamic, this can be difficult to do. Stb-tester has a new AI that can be trained to recognize each page of your application-under-test. Watch this video to see how it works:

Local training & inference

Stb-tester’s AI model trains & runs directly on your Stb-tester Node hardware — it isn’t sending each frame of video to the cloud for processing. This allows privacy and low latency (the model runs at about 20 frames per second).

There is a new training interface in the Stb-tester Portal, that allows very quickly collecting the necessary training screenshots, and verifying the behaviour of the model interactively.

How to use it in your Python test scripts

The stbt.FrameObject base class for your Page Objects has a default implementation of the “is_visible” property. Before, each of your Page Objects had to implement this property, perhaps using stbt.match like this:

 class Settings(stbt.FrameObject):
-    @property
-    def is_visible(self):
-        return bool(stbt.match("Settings.png", frame=self._frame))

Now, if you remove the is_visible property from your Page Object, it will use the default implementation, which asks the AI model if the current page is visible:

class Settings(stbt.FrameObject):
    pass

Class names are mapped to the AI labels as “device or application name” slash “Python class name”, for example “appletv/Settings”. The application name comes from the directory layout of your Python code, like this:

tests/{app_name}/pages/{module_name}/{module_name}.py

If you are using Stb-tester’s standard directory layout, this mapping works automatically. (We also support a few other directory layouts used by existing customers with large code-bases, to ease adoption of new Object Repository and AI features.)

This is Stb-tester’s first AI model, and it only applies to the “is_visible” property. Stay tuned for more AI features to come!

Your AI model is version-controlled

The training screenshots are stored in your test-pack git repository under the folder ai/is_visible/training/, so whenever you run a test it will use the model that was trained from that specific git commit.

This way, the behaviour of your tests continues to be version-controlled and reproducible — which, in our opinion, are fundamental requirements for any serious test automation.

Try it out!

You can adopt this new AI model incrementally. Use it for new development —any new Page Objects you write— or if you have an existing Page Object that is unreliable, or is taking a lot of maintenance effort, then try replacing it with AI!

We expect this new AI feature to speed up your test development and simplify your Page Object code significantly, while integrating seamlessly with your existing test scripts and emphasizing a “code first” approach of managing test automation as code. We plan to iterate quickly on ever more powerful and general AI models, so please send your feedback to support@stb-tester.com.