New features in v33: Pytest support
03 Nov 2022.
Stb-tester v33 supports running your tests with pytest, a popular Python test-runner. Pytest provides powerful functionality such as assertion introspection and fixtures.
When an assert in your test fails, pytest prints additional information about the value that triggered the assertion. Without pytest the output looks like this:
assert menu.selection == "Channels" AssertionError
Pytest’s output looks like this:
> assert menu.selection == "Channels" E AssertionError: assert 'Settings' == 'Channels' E - Channels E + Settings tests/btsport/demo.py:13: AssertionError
Much more helpful!
To learn more, see pytest’s documentation about asserting.
Pytest’s fixtures allow you to define setup and teardown code that can be re-used in many tests. For example, the following fixture captures logs from the device-under-test while the test is running:
@pytest.fixture def logs(): start_capturing_logs("dut.log") try: yield finally: stop_capturing_logs()
(I’ll leave the implementation of start_capturing_logs() and stop_capturing_logs() up to you. 😁)
To use this fixture in any test, just pass it as an argument to the test:
def test_menu_navigation(logs): page = launch_my_app() page = page.navigate_to("Settings") assert page.selection == "Settings"
In the fixture, code before the yield will run before the test; code in the finally block after the yield will run after the test (even if the test raised an exception).
To learn more, see About fixtures in the pytest documentation.
Using pytest as the test-runner is optional. To use it, you must set the
following configuration in your test-pack’s
[test_pack] stbt_version = 33 pytest = true
Changes to the stbt core Python API, and the test-run environment, are
version-controlled. You can specify the version you want to use in your
.stbt.conf file. This mechanism allows you to upgrade in a controlled manner,
and to test the upgrade on a branch first.
For more details see the v33 release notes.
Pytest may become the default runner (or the only runner) in the next Stb-tester release.
Notes for pytest experts
We run each test-case in a separate pytest process. This means that there’s no difference between “session” fixtures and “function” fixtures — each test function is run in a separate session.
Support for parametrized tests is limited: All parametrized variants are run as a single test-run, so you get a single test-result (a single video, single logfile, and single pass/fail outcome) for all of them.
Both of these limitations may be removed in a future Stb-tester release.