Configuration of the stb-tester nodes in your test farm is done by modifying the following files in the test-pack git repository.
- .stbt.conf: An “ini” format file that configures your test farm and the run-time environment in which testcases are executed. See .stbt.conf below.
- config/test-farm/<hostname>.conf (optional): Allows overriding configuration values for a specific stb-tester node in your test farm. See node-specific-config below.
- config/setup/setup: An optional script to perform custom configuration of the environment in which your testcases are run. See Customising the test-run environment.
.stbt.conf in the root of your test-pack is a configuration file
in the “ini” format that specifies
the run-time environment in which testcases are executed.
By default this file is a symlink to
[test_pack] stbt_version = 31 python_version = 3 default_remote_control = XBox [ocr] lang = eng
test_pack.stbt_version specifies the version of the stbt Python API that your testcases use. This pins the test execution environment to a specific version of the stbt core Python API.
This allows you to manage upgrades of the stbt core Python API in a controlled manner, and to test the upgrade (by changing this setting in a branch of the test-pack git repo before committing it to the master branch). See the Release notes for the differences between each version.
test_pack.python_version specifies which Python interpreter runs your test_scripts. Valid values are:
python_version is ignored if stbt_version <= 30.
test_pack.default_remote_control specifies which remote control will be selected by default in the web interface, and in the run_tests REST API if you don’t specify
remote_control. This is most useful in node-specific config files, when you have different stb-tester nodes connected to different models of set-top boxes.
For example if you have an infrared configuration file called
XBox.lircd.conf, here you would specify
default_remote_control = XBox.
The special value
roku_httpenables a proprietary HTTP protocol for controlling Roku models that don’t have an infrared receiver. See Testing the Roku Streaming Stick+ with Stb-tester.
test_pack.portal_url specifes the URL of the portal this test-pack is associated with. This is used by stbt_rig to know which portal to talk to. You shouldn’t need to modify this value.
portal_url = https://customer.stb-tester.com.
test_pack.watchdog_timeout limits the maximum time spent running any single testcase. If a test-run takes longer than this value it will be stopped and the result will be recorded as an error.
This can be useful to make soak tests robust against deadlocks or infinite loops in your test scripts.
This duration can be specified with units, for example “2hr 30min”. If no units are specified it is assumed to be in seconds. The following time units are understood:
- seconds, second, sec, s
- minutes, minute, min, m
- hours, hour, hr, h
A value of
0means the watchdog will be disabled and the only limit on test-run time will be disk-space for the test artifacts. This is the default.
Node-specific configuration files¶
You can configure individual stb-tester nodes by creating the file config/test-farm/<hostname>.conf (for example config/test-farm/stb-tester-c03fd562ae33.conf). The hostname is labelled on the physical stb-tester node, and it is also shown in the stb-tester portal.
[node] friendly_name = PS4 [device_under_test] ip_address = 192.168.1.7
You can specify any value that is allowed in .stbt.conf above; as well as the following values:
- node.friendly_name specifies a name for displaying in the portal. This makes it easier to identify each stb-tester node. You can use it to describe the device-under-test that it’s connected to (for example “Roku” or “PS4”) or a physical location (for example “Rack 3 Shelf 2”).
- device_under_test.ip_address: Only used by the Roku HTTP control (see Testing the Roku Streaming Stick+ with Stb-tester).
Automatic configuration keys¶
stb-tester sets some configuration keys at run-time to expose information about
the environment in which the tests are being run. You can then retrieve this
information during the test-run using the
stbt.get_config Python API. These
are set automatically and you should not set these values in your
- node.node_id is the id of the node that the test is currently being run
- node.portal_url is the url of the portal that the test is currently being
run on, e.g.
https://customer.stb-tester.com. This will usually have the same value as
test_pack.portal_url- but it is set automatically during a test-run rather than recorded in the configuration file itself.
The following config values mirror the return type from the /api/v2/results/(result_id) REST API. See the documentation there for reference.
- result.tags.<tagname> - Use
get_config("result.tags", "tagname")to retrieve the value of a tag passed to the /api/v2/run_tests REST API endpoint when starting the test job. This provides a mechanism for passing parameters to tests.
The following config values mirror the return type from the /api/v2/jobs/(job_uid) REST API. See the documentation there for reference:
Reading configuration values in your test scripts¶
You can store your own custom configuration values in the .stbt.conf and config/test-farm/<hostname>.conf files. Please use a different section name (not [test_pack]) to avoid clashes with configuration keys that we might add in the future. For example:
[my_company_name] stb_ip = 192.168.1.23
Then you can access this configuration value in your test scripts by using
stbt.get_config. For example, if your device-under-test allows retrieving
logs over HTTP you might use something like this:
stb_ip = stbt.get_config("my_company_name", "stb_ip") print requests.get("http://%s/logs" % stb_ip).text
Customising the test-run environment¶
You may wish to use different Python libraries or other packages in your test
scripts. For example you might want the
socat program installed to collect
more information from the system-under-test.
Stb-tester allows you to run arbitrary commands to set-up your test
environment, including installing Ubuntu packages with
apt-get install and
Python packages with
pip. Each test job will then run in a pristine copy of
the environment you design.
Save the following file into your test-pack as
#!/bin/bash -ex # Install Ubuntu packages export DEBIAN_FRONTEND=noninteractive sudo apt-get update sudo apt-get install -y \ python-requests \ socat # Install Python packages sudo pip install pylint==1.3.1
Set the executable bit:
chmod a+x config/setup/setup
Commit and push your change:
git add config/setup/setup git commit git push
Now the first time you run a test, the setup script will be run to create the
test environment and you will be able to
import requests in your Python
Each test-environment will be built just before it is needed and cached so you don’t need to wait for it to be built for every test run. Cached environments are forgotten when the Stb-tester node is powered off, so it will be re-created as needed after a reboot.
If you change the setup script, then a new environment will be built the next time you run a test.
When the setup script is run it will have no access to the rest of the test-pack sources. It will be installed into
/var/lib/stbt/test-pack/config/setup/setupand executed from the working directory
The setup script will be run in a vanilla Ubuntu environment with stb-tester installed. The stb-tester version is determined by test_pack.stbt_version in .stbt.conf. The Ubuntu version is 18.04 for stbt_version ≥ 30, or Ubuntu 14.04 for stbt_version ≤ 29.
The setup script must return a non-zero exit status if it fails to set up the environment correctly (for example if it is a shell script, use set -e).
If the setup script fails no tests will be run. To see the output from the setup script click the “Logs” link.
You can test your setup script by running stbt-docker from a checkout of the test-pack on your PC.
Customising the manual remote control image¶
The following instructions are also available as a video here.
To add an infrared remote-control configuration see Remote control configuration. The following instructions are to configure a custom graphic for your remote control in the Manual control tab of the Stb-tester portal:
The available remote controls (in the drop-down list) come from the
config/remote-control/*.lircd.conf files in your
test-pack git repository.
When you select a remote control from the drop-down list, the Stb-tester portal displays an image of the remote control with buttons you can click to send the corresponding infrared signal to the system-under-test.
You can customise this image by creating an SVG file with the same prefix as
*.lircd.conf file. For example, if there’s an infrared configuration
rcu123.lircd.conf, the corresponding SVG file is
You only need to do this if Stb-tester’s built-in SVG doesn’t have all the buttons you need.
Create a new SVG with a vector-graphics program (we recommend Inkscape).
- Instead of starting from scratch, you can copy Stb-tester’s built-in SVG.
Each SVG element that will be a clickable button must have a class attribute set to the value key, and a key attribute set to the name of the key as specified in the lircd.conf file.
To do this in Inkscape: select Edit > XML Editor, then click on the element you want to edit, and add the class and key attributes.
Commit the SVG image to the test-pack git repository, and use
git pushto push your commit to the upstream repository on GitHub.
Instead of drawing the remote control from scratch with vector graphics, a useful technique is to import a jpeg or png image of your remote control into the vector-graphics program. When it asks you whether to embed the image or link to it, choose “embed”. Then draw rectangles or circles on top of each button and set the attributes of these shapes as described above. These shapes must be in the foreground layer, and they must have a solid fill. However you can set the alpha channel of the fill colour to 1% which makes it effectively invisible (so that you can see the original button in the bitmap image underneath).