31 Mar 2015.

Up until now, stb-tester testcases were Python scripts with one testcase per file, and you referred to your testcases by the filename.

When we were designing the stb-tester ONE we wanted a more expressive way of declaring and organising testcases. We took inspiration from standard Python testing frameworks like nose and pytest, where testcases are specified as Python functions:

import stbt

def test_that_logo_is_shown():
    stbt.wait_for_match("logo.png")

def test_that_menu_appears():
    stbt.press("KEY_MENU")
    stbt.wait_for_match("menu.png")

With the stb-tester v22 release, this functionality is also available to the stbt run and stbt batch run command-line tools:

stbt run test.py::test_that_logo_is_shown

Organising your testcases in this way has several advantages:

  • It encourages smaller, more descriptive testcases.

  • It makes it easier to factor out and share common code between testcases.

  • You can attach documentation or metadata to testcases (using docstrings, nose’s attr decorator, etc.) which can be queried by importing the test file and inspecting the functions.

    For example, you can attach metadata like “this testcase validates requirement ABC” or “this is a regression test for defect XYZ”; and you can write post-run scripts to update your defect-tracker after each testrun.

  • It brings stb-tester in line with standard Python testing frameworks like nose and pytest, which your Python programmers are already familiar with.

Drawing further inspiration from nose and pytest, stb-tester v22 also allows you to use assert in your testcases. We have also added a utility function called wait_until that allows a very flexible and compact way of expressing your testcases. See also this older blog post where we had discussed the design of wait_until as a way of making the stb-tester API more composable.