stbt_rig Command-Line Tool#

stbt_rig is a command-line tool for interacting with the Stb-tester Portal’s REST API. Developers run stbt_rig from their own PCs (or from a Continuous Integration server) to run tests, generate reports in JUnit XML format, and download test artifacts.

stbt_rig has several sub-commands:

  • stbt_rig snapshot pushes a snapshot of your current test-pack (git repo) to a branch called “@YOUR_USERNAME’s snapshot” on the Stb-tester Portal. Typically you bind this to your IDE’s “Save” button to test your changes without having to make git commits each time — see Development Environment Setup.

  • stbt_rig run runs one or more tests, waits for them to finish, and optionally downloads artifacts and generates test-results in JUnit XML format (which is commonly understood by third-party test-reporting tools). You can use this to run tests directly from your IDE (see Development Environment Setup) or from your CI server (see Continuous Integration).

  • stbt_rig download downloads artifacts (log files, screenshots, etc) from previous test-runs.

  • stbt_rig screenshot downloads a live screenshot from the specified Stb-tester Node.

  • stbt_rig setup sets up your local development environment.

Installation#

stbt_rig.py is committed to your Stb-tester test-pack (git repo). You can also install it with pip install stbt_rig or download the latest version from <https://github.com/stb-tester/stbt-rig>.

stbt_rig will run on Windows, MacOS or Linux. It requires Python, Git, and the following Python packages:

  • requests.

  • keyring (optional, for storing your authentication token — see “Authentication” below).

  • tzlocal (optional, used to display logs in your own timezone instead of UTC).

stbt_rig is released under the MIT license so you can copy it into your own git repositories.

Authentication#

stbt_rig needs to authenticate to your Stb-tester Portal, so you need to generate an access token: Log into the Stb-tester Portal using your browser, click on the user menu in the top right corner, and select “Generate new access token”.

_images/generate-new-access-token.png

You will need to enter the access token the first time you run stbt_rig. If you have the Python “keyring” package installed, stbt_rig will save the token in the secure password storage provided by your operating system, so that you don’t have to type it again.

If that doesn’t work, you can save the access token into a file and pass it on the stbt_rig command-line with --portal-auth-file=FILENAME (don’t commit this file to git!) or you can put the access token in the environment variable $STBT_AUTH_TOKEN.

In Jenkins you can use the Credentials Binding plugin to pass the access token in an environment variable. See Jenkins integration below.

Global options#

-h, --help#

Show a help message and exit.

-C PATH#

Change to directory PATH before doing anything else.

--portal-url https://COMPANYNAME.stb-tester.com#

Base URL of your Stb-tester Portal. Defaults to the value specified in .stbt.conf in the current directory (see Configuration Reference).

--portal-auth-file FILENAME#

File containing the HTTP REST API access token. See Authentication above.

--node-id stb-tester-abcdef123456#

Which Stb-tester Node to execute the command on. The Node ID is labelled on the physical Stb-tester Node, and it is also shown in the Stb-tester Portal. Defaults to the environment variable $STBT_NODE_ID.

--git-remote NAME#

Which git remote to push to. Defaults to “origin” (this is the default name that git creates when you did the original “git clone” of the test-pack repository). This is only used by the commands that need to push temporary snapshots to git: that is, “run” (when --mode=interactive) and “snapshot”.

--mode {auto,bamboo,interactive,jenkins}#

See the sections Interactive mode, Jenkins integration, and Bamboo integration below. This defaults to “auto”, which detects if it is being run inside Jenkins or Bamboo.

--csv FILENAME#

Write test-results in CSV format to the specified file.

-v, --verbose#

Specify once to enable INFO logging, twice for DEBUG.

stbt_rig snapshot#

Usage:

stbt_rig snapshot

Push a snapshot of your current test-pack (git repo) to a branch called “@YOUR_USERNAME’s snapshot” on the Stb-tester Portal. Typically you bind this to your IDE’s “Save” button to test your changes without having to make git commits each time — see Development Environment Setup.

This doesn’t modify your current git checkout (it doesn’t create any local git commits). After you have tested the snapshot, you still need to make a git commit and do a git push to share your changes with the rest of your team (see Git Workflow).

The snapshot only contains files known to git; use git add to stage new files.

Note that the “stbt_rig run” command does a snapshot automatically when it’s in interactive mode.

stbt_rig run#

Run the specified testcases on the specified Stb-tester Node. In interactive mode (the default mode if not running inside a Jenkins job) it also pushes a snapshot of your current test-pack (see stbt_rig snapshot above) so that you don’t have to make lots of temporary git commits to debug your test scripts.

Usage:

stbt_rig --node-id=stb-tester-abcdef123456 run
  [--help] [--force] [--test-pack-revision GIT_SHA] [--remote-control NAME]
  [--category NAME] [--soak] [--shuffle] [--tag NAME=VALUE]
  [--artifacts GLOB] [--artifacts-dest PATH] [--junit-xml FILENAME]
  TESTCASE [TESTCASE ...]

Positional arguments:

TESTCASE#

One or more tests to run. Test names have the form FILENAME::FUNCTION_NAME where FILENAME is given relative to the root of the test-pack repository and FUNCTION_NAME identifies a Python function within that file; for example “tests/my_test.py::test_that_blah_dee_blah”.

Optional arguments:

--force#

Stop an existing job first (otherwise the command will fail if the Stb-tester Node is busy).

--test-pack-revision GIT_SHA#

Git commit SHA in the test-pack repository identifying the version of the tests to run. Can also be the name of a git branch or tag. In interactive mode this defaults to a snapshot of your current working directory. In jenkins mode this defaults to “master”.

--remote-control NAME#

The remote control infrared configuration to use when running the tests. This should match the name of a remote control configuration file in your test-pack git repository. For example if your test-pack has “config/remote-control/roku.lircd.conf” then you should specify “roku”. If not specified here, you must specify “test_pack.default_remote_control” in the test-pack’s .stbt.conf file (see Configuration Reference).

--category NAME#

Category to save the test-results in. When you are viewing test results in the Stb-tester Portal you can filter by this string. In interactive mode this defaults to the branch name. In jenkins mode this defaults to the Jenkins job name.

--soak#

Run the tests forever until you interrupt them by pressing Control-C.

--shuffle#

Randomise the order in which the tests are run. If --soak is also specified, this will prefer to run the faster testcases more often.

-t NAME=VALUE, --tag NAME=VALUE#

Tags are recorded in the test results database, and can be accessed from the Python test script with stbt.get_config("result.tags", "NAME"). --tag can be specified more than once.

--artifacts GLOB#

Download the specified artifacts. This is a filename glob. Set to * for all artifacts. This argument can be specified multiple times.

--artifacts-dest PATH#

Download the artifacts (specified by --artifacts) to PATH. You can use the placeholders {result_id}, {filename}, and {basename}. Defaults to {result_id}/artifacts/{filename}. Directories will be created as required.

--junit-xml FILENAME#

Save JUnit style XML file with results to this path. In jenkins or bamboo mode this is enabled by default with the filename “stbt-results.xml”.

Interactive mode#

In interactive mode (the default mode if not running inside a Jenkins job) the “run” command takes a snapshot of your current directory and pushes it to the branch “@<your username>’s snapshot” on the Stb-tester Portal, so that you don’t have to make lots of temporary git commits to debug your test code.

Jenkins integration#

stbt_rig detects if it is running inside a Jenkins job. If so, it enables the following behaviours:

  • Record various Jenkins parameters as tags in the Stb-tester results:

    • jenkins/BUILD_ID

    • jenkins/BUILD_URL

    • jenkins/GIT_COMMIT

    • jenkins/JOB_NAME

    • jenkins/SVN_REVISION

  • Write test results in JUnit format to “stbt-results.xml” for the Jenkins JUnit plugin.

  • Stop the tests if you press the “stop” button in Jenkins.

For instructions on how to configure your Jenkins job see Continuous Integration.

Bamboo integration#

Similarly, stbt_rig detects if it is running from Bamboo (Atlassian’s continuous integration server):

  • Read the access token from bamboo.STBT_AUTH_PASSWORD variable.

  • Record the following Bamboo variables as tags in the Stb-tester results:

    • bamboo.buildPlanName

    • bamboo.buildResultKey

    • bamboo.buildResultsUrl

    • bamboo.planRepository.branchName

    • bamboo.planRepository.revision

  • Write test results in JUnit format to “stbt-results.xml” suitable for Bamboo’s “JUnit Parser” task.

  • Stop the tests if you press “Stop build” in Bamboo.

For instructions on how to configure your Bamboo job see Continuous Integration.

stbt_rig download#

Usage:

stbt_rig download [--artifacts GLOB] [--artifacts-dest PATH] FILTER

Download artifacts (log files, screenshots, etc) from previous test-runs.

Positional arguments:

FILTER#

A search filter to identify the desired test-runs, as given to the REST API (/api/v2/results) or the interactive test-results view in the Stb-tester Portal – see Filter in the User Interface Reference.

For example “node:stb-tester-abcdef123456 date:>2021-04-20” would match all test-runs on that node since that date.

Optional arguments:

--artifacts GLOB#

Download the specified artifacts. This is a filename glob. Defaults to * (all artifacts). This argument can be specified multiple times.

--artifacts-dest PATH#

Download the artifacts to PATH. You can use the placeholders {result_id}, {filename}, and {basename}. Defaults to {result_id}/artifacts/{filename}. Directories will be created as required.

stbt_rig screenshot#

Usage:

stbt_rig --node-id=stb-tester-abcdef123456 screenshot [FILENAME]

Take a screenshot from the specified Stb-tester Node and save it to disk.

Positional arguments:

FILENAME#

Output filename. Defaults to “screenshot.png”.

stbt_rig setup#

Usage:

stbt_rig setup [--vscode]

Set up local development environment. This creates a virtual environment, installs stb-tester, ensures authentication and git are working and optionally configures VSCode. See Development Environment Setup.

The virtual environment will be created under .venv/ in your test-pack directory.

Optional arguments:

--vscode#

Create/update .vscode/settings.json to enable linting, running tests and snapshots from within the VSCode.