stbt_rig

stbt_rig is a command-line tool for interacting with the Stb-tester Portal’s REST API. Developers run stbt_rig from their own PCs (or from a Continuous Integration server) to run tests, generate reports in JUnit XML format, and download test artifacts.

stbt_rig has several sub-commands:

  • stbt_rig snapshot pushes a snapshot of your current test-pack (git repo) to a branch called “@YOUR_USERNAME’s snapshot” on the Stb-tester Portal. Typically you bind this to your IDE’s “Save” button to test your changes without having to make git commits each time — see IDE Configuration.
  • stbt_rig run runs one or more tests, waits for them to finish, and optionally downloads artifacts and generates test-results in JUnit XML format (which is commonly understood by third-party test-reporting tools). You can use this to run tests directly from your IDE (see IDE Configuration) or from your CI server (see Continuous Integration).
  • stbt_rig screenshot downloads a screenshot from the specified Stb-tester Node.

Installation

stbt_rig.py is committed to your Stb-tester test-pack (git repo). You can also install it with pip install stbt_rig or download the latest version from <https://github.com/stb-tester/stbt-rig>.

stbt_rig will run on Windows, MacOS or Linux. It requires Python, Git, python-requests, and optionally python-keyring for storing your access token.

stbt_rig is released under the MIT license so you can copy it into your own git repositories.

Authentication

stbt_rig needs to authenticate to your Stb-tester Portal, so you need to generate an access token: Log into the Stb-tester Portal using your browser, click on the user menu in the top right corner, and select “Generate new access token”.

_images/generate-new-access-token.png

You will need to enter the access token the first time you run stbt_rig. If you have the Python “keyring” package installed, stbt_rig will save the token in the secure password storage provided by your operating system, so that you don’t have to type it again.

If that doesn’t work, you can save the access token into a file and pass it on the stbt_rig command-line with --portal-auth-file=FILENAME (don’t commit this file to git!) or you can put the access token in the environment variable $STBT_AUTH_TOKEN.

In Jenkins you can use the Credentials Binding plugin to pass the access token in an environment variable. See Jenkins integration below.

Global options

-h, --help

Show a help message and exit.

-C PATH

Change to directory PATH before doing anything else.

--portal-url https://COMPANYNAME.stb-tester.com

Base URL of your Stb-tester Portal. Defaults to the value specified in .stbt.conf in the current directory (see Configuration Reference).

--portal-auth-file FILENAME

File containing the HTTP REST API access token. See Authentication above.

--node-id stb-tester-abcdef123456

Which Stb-tester Node to execute the command on. The Node ID is labelled on the physical Stb-tester Node, and it is also shown in the Stb-tester Portal. Defaults to the environment variable $STBT_NODE_ID.

--git-remote NAME

Which git remote to push to. Defaults to “origin” (this is the default name that git creates when you did the original “git clone” of the test-pack repository). This is only used by the commands that need to push temporary snapshots to git: that is, “run” (when --mode=interactive) and “snapshot”.

--mode {auto,bamboo,interactive,jenkins}

See the sections Interactive mode, Jenkins integration, and Bamboo integration below. This defaults to “auto”, which detects if it is being run inside Jenkins or Bamboo.

--csv FILENAME

Write test-results in CSV format to the specified file.

-v, --verbose

Specify once to enable INFO logging, twice for DEBUG.

stbt_rig snapshot

Usage:

stbt_rig snapshot

Push a snapshot of your current test-pack (git repo) to a branch called “@YOUR_USERNAME’s snapshot” on the Stb-tester Portal. Typically you bind this to your IDE’s “Save” button to test your changes without having to make git commits each time — see IDE Configuration.

This doesn’t modify your current git checkout (it doesn’t create any local git commits). After you have tested the snapshot, you still need to make a git commit and do a git push to share your changes with the rest of your team.

The snapshot only contains files known to git; use git add to stage new files.

Note that the “stbt_rig run” command does a snapshot automatically when it’s in interactive mode.

stbt_rig run

Run the specified testcases on the specified Stb-tester Node. In interactive mode (the default mode if not running inside a Jenkins job) it also pushes a snapshot of your current test-pack (see stbt_rig snapshot above) so that you don’t have to make lots of temporary git commits to debug your test scripts.

Usage:

stbt_rig --node-id=stb-tester-abcdef123456 run
  [--help] [--force] [--test-pack-revision GIT_SHA] [--remote-control NAME]
  [--category NAME] [--soak] [--shuffle] [--tag NAME=VALUE]
  [--artifacts GLOB] [--artifacts-dest PATH] [--junit-xml FILENAME]
  TESTCASE [TESTCASE ...]

Positional arguments:

TESTCASE

One or more tests to run. Test names have the form FILENAME::FUNCTION_NAME where FILENAME is given relative to the root of the test-pack repository and FUNCTION_NAME identifies a Python function within that file; for example “tests/my_test.py::test_that_blah_dee_blah”.

Optional arguments:

--force

Stop an existing job first (otherwise the command will fail if the Stb-tester Node is busy).

--test-pack-revision GIT_SHA

Git commit SHA in the test-pack repository identifying the version of the tests to run. Can also be the name of a git branch or tag. In interactive mode this defaults to a snapshot of your current working directory. In jenkins mode this defaults to “master”.

--remote-control NAME

The remote control infrared configuration to use when running the tests. This should match the name of a remote control configuration file in your test-pack git repository. For example if your test-pack has “config/remote-control/roku.lircd.conf” then you should specify “roku”. If not specified here, you must specify “test_pack.default_remote_control” in the test-pack’s .stbt.conf file (see Configuration Reference).

--category NAME

Category to save the test-results in. When you are viewing test results in the Stb-tester Portal you can filter by this string. In interactive mode this defaults to the branch name. In jenkins mode this defaults to the Jenkins job name.

--soak

Run the tests forever until you interrupt them by pressing Control-C.

--shuffle

Randomise the order in which the tests are run. If --soak is also specified, this will prefer to run the faster testcases more often.

-t NAME=VALUE, --tag NAME=VALUE

Tags are recorded in the test results database, and can be accessed from the Python test script with stbt.get_config("result.tags", "NAME"). --tag can be specified more than once.

--artifacts GLOB

Select artifacts to be downloaded. This is a filename glob. Set it to * for all artifacts. This argument can be specified multiple times.

--artifacts-dest PATH

Artifacts will be downloaded to here. You can include the placeholders {result_id}, {filename} and {basename}. Defaults to {result_id}/artifacts/{filename}. Directories will be created as required.

--junit-xml FILENAME

Save JUnit style XML file with results to this path. In jenkins or bamboo mode this is enabled by default with the filename “stbt-results.xml”.

Interactive mode

In interactive mode (the default mode if not running inside a Jenkins job) the “run” command takes a snapshot of your current directory and pushes it to the branch “@<your username>’s snapshot” on the Stb-tester Portal, so that you don’t have to make lots of temporary git commits to debug your test code.

Jenkins integration

stbt_rig detects if it is running inside a Jenkins job. If so, it enables the following behaviours:

  • Record various Jenkins parameters as tags in the Stb-tester results:
    • jenkins/BUILD_ID
    • jenkins/BUILD_URL
    • jenkins/GIT_COMMIT
    • jenkins/JOB_NAME
    • jenkins/SVN_REVISION
  • Write test results in JUnit format to “stbt-results.xml” for the Jenkins JUnit plugin.
  • Stop the tests if you press the “stop” button in Jenkins.

For instructions on how to configure your Jenkins job see Continuous Integration.

Bamboo integration

Similarly, stbt_rig detects if it is running from Bamboo (Atlassian’s continuous integration server):

  • Read the access token from bamboo.STBT_AUTH_PASSWORD variable.
  • Record the following Bamboo variables as tags in the Stb-tester results:
    • bamboo.buildPlanName
    • bamboo.buildResultKey
    • bamboo.buildResultsUrl
    • bamboo.planRepository.branchName
    • bamboo.planRepository.revision
  • Write test results in JUnit format to “stbt-results.xml” suitable for Bamboo’s “JUnit Parser” task.
  • Stop the tests if you press “Stop build” in Bamboo.

For instructions on how to configure your Bamboo job see Continuous Integration.

stbt_rig screenshot

Usage:

stbt_rig --node-id=stb-tester-abcdef123456 screenshot [filename]

Take a screenshot from the specified Stb-tester Node and save it to disk.

Positional arguments:

filename

Output filename. Defaults to “screenshot.png”.