User Interface Reference

Job category

images/run-tests.png

This is simply a way of categorising your test results into different “folders”. When viewing test results, you can filter by this category so that you only see the relevant results.

For example, you might use the system-under-test’s version as the job category. Or if you are testing changes you’ve made to the test scripts, you might use your own name or the name of the git branch, so that your results don’t pollute the “real” results.

Soak-testing options

images/Soak-testing_options.png

Run once

Each test selected in the Testcases list will run once in the order in which they are selected in the list. Once all the testcases have been run the test job will finish.

Run forever (until stopped manually)

The testcases selected in the Testcases list will be run in the order in which they are selected in the list. Once all the testcases have been run the tests will start again from the beginning of the testcase list.

The testcases will continue to be run in a loop until the “Stop” button is clicked.

Run forever in random order (until stopped manually)

This option runs the given testcases in a random order, until the “Stop” button is clicked.

This can be useful if you have structured your test pack as a large number of short targeted tests. You can then select many different tests to attempt a random walk of different journeys though your set-top-box UI. This can be particularly effective at finding hard to trigger bugs and get more value out of the test-cases you have written.

Some tests may take much longer than other tests, which will then use up a disproportionate amount of your soaking time. To work around this we measure how long each test takes the first time it runs, and then we use that as a weighting when choosing the next test to run. Overall, we try to spend the same amount of time in each testcase, so we run the shorter testcases more frequently.

This makes it reasonable to include tests that take 10 seconds, and tests that take a few minutes or hours, in the same random soak.

Test results

images/results-view.png

Filter

The search box at the top of the page allows you to filter the results that are shown.

  • Usually you can just type the words that you want to match. This will show results that contain all of the words you typed (in any order). It will search in the testcase name, the result / failure reason, and the job category (which you specify when you run a test job). All searches are case insensitive.

  • Use quotes to search for a specific phrase with spaces: “app crashed” or ‘app crashed’

  • Limit the search to specific fields: testcase:settings result:fail will show failing testruns from any testcases with the word “settings” in the testcase name. Note that these field names match the column headings of the results table.

    • category: matches the job category (which you specify when you run a test job).

    • date: matches the start timestamp of a testrun.

      • date:>”2015-10-21 13:27:00” shows testruns that started after the specified time.
      • date:>2015-10-21 – if you leave out the time, it defaults to midnight (the start of the day). Similarly, you can leave out the seconds, or the minutes.
      • date:<”2015-10-21 13:27:00” shows testruns that started before the specified time.
      • date:[2015-10-21 13:00 to 2015-10-21 06:30] shows testruns that started within the specified range – from 13:00:00 to 06:30:59.
    • duration: matches how long a testrun took.

      • You can use ranges similar to the date syntax above: duration:>90 shows testruns that took more than 90 seconds to complete.
    • job: matches the specified job, for example job:/stb-tester-one-eca86bfd3b76/1jkw/4. You can find the job uid in the result details pane on the right, or via the REST API.

    • testcase: matches the testcase name.

    • result: matches the result or failure reason.

      • result:pass shows passing results.
      • result:fail shows failing results.
      • result:error shows errored results (for example, the test did not complete because of a bug in the test script).
      • result:”some failure reason” shows failing or errored results with the specified failure reason (that is, the text of the exception raised by your test script).
  • Combine search terms with boolean operators (and, or, not). A space is the same as and (unless inside quotes).

    • testcase:settings and result:pass shows results from testcases with the word “settings” in their name, that passed.
    • testcase:settings result:pass is the same as the previous example.
    • testcase:settings or result:pass shows all results that passed, as well as all results for testcases with the word “settings” in their name.
    • (testcase:settings or testcase:menu) result:pass – use parentheses for precedence in complex boolean queries. Without parentheses, and has higher precedence than or.

REST API

The stb-tester ONE provides a REST API for accessing test results programmatically.

Large test farms

If you have several stb-tester ONE devices you can quickly switch between your devices-under-test using the menu at the top left of the screen:

images/devices-dropdown.png

From this menu click “expand” to access a “video wall” that allows you to monitor all the devices in your test farm:

images/video-wall.png

See also:

Live video formats

You can choose the format used for streaming live video to your browser by clicking the gear icon underneath the video:

images/video-format-selector.png

Choose Flash video:

  • For high quality video
  • If you have a high bandwidth link to the stb-tester ONE
  • And you have the flash plugin installed

Choose JPEG thumbnails:

  • For low bandwidth video
  • If you don’t have the flash plugin installed
  • If you can tolerate low-quality, low-framerate video