Pytest

Stb-tester uses pytest, a popular Python test-runner. Pytest provides powerful functionality such as assertion introspection and fixtures.

Assertion introspection

When an assert in your test fails, pytest prints additional information about the value that triggered the assertion. Without pytest the output looks like this:

    assert menu.selection == "Channels"
AssertionError

Pytest’s output looks like this:

>       assert menu.selection == "Channels"
E       AssertionError: assert 'Settings' == 'Channels'
E         - Channels
E         + Settings
tests/btsport/demo.py:13: AssertionError

Much more helpful, because it shows you the actual value that didn’t match the expected value.

To learn more, see pytest’s documentation about asserting.

Fixtures

Pytest’s fixtures allow you to define setup and teardown code that can be re-used in many tests. For example, the following fixture captures logs from the device-under-test while the test is running:

tests/conftest.py

import pytest

@pytest.fixture
def logs():
    # Code to start capturing logs goes here
    try:
        yield
    finally:
        # Code to stop capturing logs goes here

To use this fixture in any test, just pass it as an argument to the test:

def test_menu_navigation(logs):
    page = launch_my_app()
    page = page.navigate_to("Settings")
    assert page.selection == "Settings"

In the fixture, code before the yield will run before the test; code in the finally block after the yield will run after the test (even if the test raised an exception).

To learn more, see About fixtures in the pytest documentation.

For our recommended logging configuration using fixtures, see Capturing logs from the device-under-test.

Older Stb-tester versions

In Stb-tester v33, using pytest as the test-runner is optional. From Stb-tester v34, pytest is the default (and only) test-runner.

To enable pytest on v33, use the following configuration in your test-pack’s .stbt.conf file (you don’t need this on v34 or newer):

[test_pack]
stbt_version = 33
pytest = true

Changes to the stbt core Python API, and the test-run environment, are version-controlled. You can specify the version you want to use in your .stbt.conf file. This mechanism allows you to upgrade in a controlled manner, and to test the upgrade on a branch first.

For more details see the Stb-tester release notes.

Notes for pytest experts

We run each test-case in a separate pytest process. This means that there’s no difference between “session” fixtures and “function” fixtures — each test function is run in a separate session.

Support for parametrized tests is limited: All parametrized variants are run as a single test-run, so you get a single test-result (a single video, single logfile, and single pass/fail outcome) for all of them.

Both of these limitations may be removed in a future Stb-tester release.