User Interface Reference

Test farm

Click on the device name at the top left corner of the screen to access a menu where you can quickly switch between your devices:

images/devices-dropdown.png

By default this displays the ID of each Stb-tester node (which based on its MAC address). You can configure a “friendly name” to display: See node.friendly_name in the Configuration Reference.

From this menu click “monitor” to access a “video wall” that allows you to monitor all the devices in your test farm:

images/video-wall.png

Tags

When running tests, you can specify a set of tags (name and value, or just name). This allows you to change the test-script behaviour at run-time. For example, if you have a test that searches for a programme in your video-on-demand search screen, you might want to specify the name of the programme in a tag.

Tags are passed to the test script as an automatic configuration key and are recorded in the test-results. Tags set here can be retrived in Python during the test-run with stbt.get_config("result.tags", "tagname"). Tags can be retrieved after the test run with the REST API endpoint GET /api/v2/results/(result_id)/ and viewed in the result details pane of the Results View.

Valid tag names:

  • Must not be blank (you can pass a tag without a value, but not a value without a tag).
  • Must not be the same as another tag name.
  • Must be shorter than 500 characters.
  • Must not contain :, =, $, <, >, newline, null, or other unprintable characters.
  • Must not begin with /, -, ., or _.

Valid tag values:

  • Must be shorter than 500 characters.

Job category

images/run-tests.png

This is simply a way of categorising your test results into different “folders”. When viewing test results, you can filter by this category so that you only see the relevant results.

For example, you might use the system-under-test’s version as the job category. Or if you are testing changes you’ve made to the test scripts, you might use your own name or the name of the git branch, so that your results don’t pollute the “real” results.

Soak-testing options

images/Soak-testing_options.png

Run once

Each test selected in the Testcases list will run once in the order in which they are selected in the list. Once all the testcases have been run the test job will finish.

Run forever (until stopped manually)

The testcases selected in the Testcases list will be run in the order in which they are selected in the list. Once all the testcases have been run the tests will start again from the beginning of the testcase list.

The testcases will continue to be run in a loop until the “Stop” button is clicked.

Run forever in random order (until stopped manually)

This option runs the given testcases in a random order, until the “Stop” button is clicked.

This can be useful if you have structured your test-pack as a large number of short targeted tests. You can then select many different tests to attempt a random walk of different journeys though your set-top-box UI. This can be particularly effective at finding hard to trigger bugs and get more value out of the testcases you have written.

Some tests may take much longer than other tests, which will then use up a disproportionate amount of your soaking time. To work around this we measure how long each test takes the first time it runs, and then we use that as a weighting when choosing the next test to run. Overall, we try to spend the same amount of time in each testcase, so we run the shorter testcases more frequently.

This makes it reasonable to include tests that take 10 seconds, and tests that take a few minutes or hours, in the same random soak.

Test results

images/results-view.png

Filter

The search box at the top of the page allows you to filter the results that are shown.

  • Usually you can just type the words that you want to match. This will show results that contain all of the words you typed (in any order). It will search in the testcase name, the result / failure reason, and the job category (a label that you specify when you run a test job). All searches are case insensitive.

  • Use quotes to search for a specific phrase with spaces: “app crashed” or ‘app crashed’

  • Limit the search to specific fields: testcase:settings result:fail will show failing testruns from any testcases with the word “settings” in the testcase name. Note that these field names match the column headings of the results table.

    • branch: matches the git branch of the test-pack.

      • Note: For test jobs initiated via the REST API, the git branch is only recorded if it was specified by name, not by sha.
    • category: matches the job category (a label that you specify when you run a test job).

    • date: matches the start timestamp of a testrun.

      • date:>”2015-10-21 13:27:00” shows testruns that started after the specified time.
      • date:>2015-10-21 – if you leave out the time, it defaults to midnight (the start of the day). Similarly, you can leave out the seconds, or the minutes.
      • date:<”2015-10-21 13:27:00” shows testruns that started before the specified time.
      • date:[2015-10-21 13:00 to 2015-10-21 06:30] shows testruns that started within the specified range – from 13:00:00 to 06:30:59.
    • duration: matches how long a testrun took, in seconds.

      • You can use ranges similar to the date syntax above: duration:>90 shows testruns that took more than 90 seconds to complete.
    • job: matches the specified job, for example job:/stb-tester-eca86bfd3b76/1jkw/4. You can find the job uid in the result details pane on the right, or via the REST API.

    • node: matches tests than ran on a specific stb-tester node, for example node:stb-tester-eca86bfd3b76.

    • result: matches the result or failure reason.

      • result:pass shows passing results.
      • result:fail shows failing results.
      • result:error shows errored results (for example, the test did not complete because of a bug in the test script).
      • result:”some failure reason” shows failing or errored results with the specified failure reason (that is, the text of the exception raised by your test script).
    • sha: matches the git sha of the test-pack.

    • testcase: matches the testcase name.

    • user: matches the username of the person who started the test job.

  • Combine search terms with boolean operators (and, or, not). A space is the same as and (unless inside quotes).

    • testcase:settings and result:pass shows results from testcases with the word “settings” in their name, that passed.
    • testcase:settings result:pass is the same as the previous example.
    • testcase:settings or result:pass shows all results that passed, as well as all results for testcases with the word “settings” in their name.
    • (testcase:settings or testcase:menu) result:pass – use parentheses for precedence in complex boolean queries. Without parentheses, and has higher precedence than or.

REST API

The Stb-tester portal provides a REST API for accessing test results programmatically.

Live video formats

You can choose the format used for streaming live video to your browser by clicking the gear icon underneath the video:

images/video-format-selector.png

Choose Flash video:

  • For high quality video
  • If you have a high bandwidth link to the Stb-tester HDMI node
  • And you have the flash plugin installed

Choose JPEG thumbnails:

  • For low bandwidth video
  • If you don’t have the flash plugin installed
  • If you can tolerate low-framerate video