IDE Configuration¶

Quick summary for Python experts:

Install stb-tester into your project’s virtualenv with pip install stb-tester. This enables code completion / IntelliSense, but it doesn’t allow running the test scripts locally (see the note in the first section below for details).
Add our custom linting checks by configuring pylint to run with pylint --load-plugins=stbt.pylint_plugin.
Install stbt_rig with pip install stbt_rig. This is a Python library for interacting with the Stb-tester Portal’s REST API & git repository.
Configure your IDE’s “Save” button to run python -m stbt_rig -v snapshot. This syncs your local code to the Stb-tester Portal on a special “snapshot” branch — without affecting your local git checkout.
Configure your IDE to sync & run a single test remotely (on your Stb-tester Node) when you click “Run Test” from the IDE.

For more details, read on.

IntelliSense for stbt APIs¶

Install the “stb-tester” package using pip:

pip install stb-tester

Note

This package doesn’t include video-capture infrastructure and other complex dependencies, so it won’t allow you to run your test scripts locally (on your PC). It does enable your IDE’s “IntelliSense” or “Code Insight” features such as code completion, type inference, inline documentation, and linting.

If you haven’t yet upgraded your test-pack to use the latest stbt version, you can tell pip to install a specific version like this:

pip install stb-tester~=31.0

Note that ~=31.0 will install the latest 31 package, but not 32 or greater.

Make sure you install the same version locally as the version specified in your test-pack’s configuration file (see test_pack.stbt_version in the Configuration Reference). The stb-tester pip package is only available for version 31 and newer.

Enable Stb-tester’s pylint plugin¶

Pylint is a static analysis tool for Python code. This means that it analyses your code to catch common mistakes, without actually running your code. This saves a lot of development time by catching the mistakes immediately —your IDE shows red squiggles under each mistake— instead of running the test and waiting for it to reach the line of code where the mistake is.

Stb-tester provides a Pylint plugin that teaches Pylint about some mistakes that are specific to Stb-tester APIs. For example, the frame parameter of stbt.match is usually optional, but it’s mandatory when called inside a FrameObject property:

_images/stbt-frame-object-missing-frame.png

PyCharm¶

PyCharm uses its own built-in linter; it doesn’t allow using external linters such as pylint. It will still catch common Python mistakes, but not the specific stbt checks like the example above.

VS Code¶

Run Python: Select Linter from the Command Palette (Control+Shift+P) and select pylint.

Next, add the following pylint args in .vscode/settings.json:

"python.linting.pylintArgs": ["--load-plugins=stbt.pylint_plugin"],

Disabling annoying pylint warnings¶

Some of Pylint’s built-in checks are related to coding style, not real problems. You can disable specific checks by listing them in the .pylintrc file in the root of your test-pack. For example:

[MESSAGES CONTROL]
disable=
  line-too-long,
  missing-docstring,

The name of each checker (such as “line-too-long”) is shown after the error message in your IDE.

Sync the code from your IDE to the Stb-tester portal¶

Bind the “Save” button of your IDE to run python -m stbt_rig -v snapshot after saving. This will push your local code to the Stb-tester Portal on a special branch called “<your username>’s snapshot” that is only visible to you. It doesn’t affect your local git checkout (it doesn’t make any commits on your checked-out branch) so after you have tested your changes & got them working, you still need to do “git commit” and “git push” to make it official. See Development Workflow.

Before setting this up in your IDE, make sure that you can run python -m stbt_rig -v snapshot from your terminal, to debug any authentication issues. See stbt_rig installation and stbt_rig authentication here.

PyCharm¶

In Preferences > Plugins install the File Watchers plugin.

Now in Preferences > Tools > File Watchers add a new watcher with the following settings:

Name: stbt_rig snapshot
File type: Any
Scope: Project Files
Program: python
Arguments: -m stbt_rig -v snapshot
Working directory: path to your test-pack checkout

VS Code¶

In VS Code’s Extensions Marketplace (Ctrl+Shift+X) install the Run on Save extension by pucelle (or any of the dozens of similar extensions if you have a favourite, but you’ll need to change the configuration below to match your chosen extension):

Next, add this to .vscode/settings.json:

"runOnSave.commands": [
    {
        "match": ".*\\.py$",
        "command": "${workspaceFolder}/.venv/bin/python -m stbt_rig -v snapshot",
        "runIn": "terminal",
        "runningStatusMessage": "Running stbt_rig snapshot...",
        "finishStatusMessage": "Snapshot complete"
    }
],

Run a test from your IDE¶

IDEs like PyCharm and Visual Studio Code can detect the test functions in a file and show a small button to run the test:

With the configuration below, clicking the button will sync your local code-changes and run the test remotely (on your Stb-tester Node). This is useful during test-development, when you are writing or debugging the test.

Before setting this up in your IDE, make sure that you can run python -m stbt_rig -v snapshot from your terminal, to debug any authentication issues. See stbt_rig installation and stbt_rig authentication here.

PyCharm¶

First go to File > Settings and select Tools > Python Integrated Tools. Under Testing: Default test runner select pytest. Click OK.

Next, go to Run > Edit Configurations and select Templates > Python tests > pytest. Fill in the following settings:

Target: Module name
Additional Arguments: -p stbt_rig --node-id=stb-tester-123456789abc
Add content roots to PYTHONPATH: True

(Change “stb-tester-123456789abc” to the actual ID of your Stb-tester Node.)

Note that you aren’t creating a new configuration (don’t click the little “+” icon on the left-hand pane). You are editing the Template from which PyCharm creates new configurations on the fly for each test.

The configuration should now look like:

You can now run the current test with Ctrl-Shift-F10 or by clicking the green play button in the margin on the left and selecting “Run ‘pytest for …’”:

VS Code¶

Run Python: Configure Tests from the Command Palette (Control+Shift+P) and select pytest. When it asks for the directory containing the tests, choose tests.

Now open .vscode/settings.json and change python.testing.pytestArgs to this:

"python.testing.pytestArgs": [
    "-p", "stbt_rig", "--node-id=stb-tester-123456789abc",
    "--override-ini=python_files=*.py",
    "--override-ini=python_functions=test_*",
    "--tb=no", "--capture=no",
    "tests"
],

(Change “stb-tester-123456789abc” to the actual ID of your Stb-tester Node.)

Now run Python: Discover Tests from the Command Palette (Control+Shift+P) to identify each test. This adds a little “Run Test” button just above each testcase:

After the test finishes, its output appears in the Output window: View > Output (Control+K Control+H) and select Python Test Log.

Troubleshooting¶

If any of the steps above don’t work, make sure that you have the latest version of stbt_rig (use pip install --upgrade stbt_rig).

Make sure that you can run python -m stbt_rig -v snapshot from your terminal. This will prompt for your access token and save it into your operating system’s keyring for future invocations. See stbt_rig.

VS Code¶

After following all the instructions above, your .vscode/settings.json should look like this:

{
    "python.linting.pylintEnabled": true,
    "python.linting.enabled": true,
    "python.linting.pylintArgs": ["--load-plugins=stbt.pylint_plugin"],

    "runOnSave.commands": [
        {
            "match": ".*\\.py$",
            "command": "${workspaceFolder}/.venv/bin/python -m stbt_rig -v snapshot",
            "runIn": "terminal",
            "runningStatusMessage": "Running stbt_rig snapshot...",
            "finishStatusMessage": "Snapshot complete"
        }
    ],

    "python.testing.pytestArgs": [
        "-p", "stbt_rig", "--node-id=stb-tester-123456789abc",
        "--override-ini=python_files=*.py",
        "--override-ini=python_functions=test_*",
        "--tb=no", "--capture=no",
        "tests"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true
}