IDE Configuration

Quick summary for Python experts:

  1. Install stb-tester into your project’s virtualenv with pip install stb-tester. This enables code completion / IntelliSense, but it doesn’t allow running the test scripts locally (see the note in the first section below for details).
  2. Add our custom linting checks by configuring pylint to run with pylint --load-plugins=stbt.pylint_plugin.
  3. Install stbt_rig with pip install stbt_rig. This is a Python library for interacting with the Stb-tester Portal’s REST API & git repository.
  4. Configure your IDE’s “Save” button to run python -m stbt_rig -v snapshot. This syncs your local code to the Stb-tester Portal on a special “snapshot” branch — without affecting your local git checkout.
  5. Configure your IDE to sync & run a single test remotely (on your Stb-tester Node) when you click “Run Test” from the IDE.

For more details, read on.

IntelliSense for stbt APIs

Install the “stb-tester” package using pip:

pip install stb-tester

Note

This package doesn’t include video-capture infrastructure and other complex dependencies, so it won’t allow you to run your test scripts locally (on your PC). It does enable your IDE’s “IntelliSense” or “Code Insight” features such as code completion, type inference, inline documentation, and linting.

If you haven’t yet upgraded your test-pack to use the latest stbt version, you can tell pip to install a specific version like this:

pip install stb-tester~=31.0

Note that ~=31.0 will install the latest 31 package, but not 32 or greater.

Make sure you install the same version locally as the version specified in your test-pack’s configuration file (see test_pack.stbt_version in the Configuration Reference). The stb-tester pip package is only available for version 31 and newer.

Enable Stb-tester’s pylint plugin

Pylint is a static analysis tool for Python code. This means that it analyses your code to catch common mistakes, without actually running your code. This saves a lot of development time by catching the mistakes immediately —your IDE shows red squiggles under each mistake— instead of running the test and waiting for it to reach the line of code where the mistake is.

Stb-tester provides a Pylint plugin that teaches Pylint about some mistakes that are specific to Stb-tester APIs. For example, the frame parameter of stbt.match is usually optional, but it’s mandatory when called inside a FrameObject property:

_images/stbt-frame-object-missing-frame.png

PyCharm

PyCharm uses its own built-in linter; it doesn’t allow using external linters such as pylint. It will still catch common Python mistakes, but not the specific stbt checks like the example above.

VS Code

Run Python: Select Linter from the Command Palette (Control+Shift+P) and select pylint.

Next, add the following pylint args in .vscode/settings.json:

"python.linting.pylintArgs": ["--load-plugins=stbt.pylint_plugin"],

Disabling annoying pylint warnings

Some of Pylint’s built-in checks are related to coding style, not real problems. You can disable specific checks by listing them in the .pylintrc file in the root of your test-pack. For example:

[MESSAGES CONTROL]
disable=
  line-too-long,
  missing-docstring,

The name of each checker (such as “line-too-long”) is shown after the error message in your IDE.

Sync the code from your IDE to the Stb-tester portal

Bind the “Save” button of your IDE to run python -m stbt_rig -v snapshot after saving. This will push your local code to the Stb-tester Portal on a special branch called “<your username>’s snapshot” that is only visible to you. It doesn’t affect your local git checkout (it doesn’t make any commits on your checked-out branch) so after you have tested your changes & got them working, you still need to do “git commit” and “git push” to make it official. See Development Workflow.

Before setting this up in your IDE, make sure that you can run python -m stbt_rig -v snapshot from your terminal, to debug any authentication issues. See stbt_rig installation and stbt_rig authentication here.

PyCharm

In Preferences > Plugins install the File Watchers plugin.

Now in Preferences > Tools > File Watchers add a new watcher with the following settings:

  • Name: stbt_rig snapshot
  • File type: Any
  • Scope: Project Files
  • Program: python
  • Arguments: -m stbt_rig -v snapshot
  • Working directory: path to your test-pack checkout
_images/file-watchers-configuration.png

VS Code

In VS Code’s Extensions Marketplace (Ctrl+Shift+X) install the Run on Save extension by pucelle (or any of the dozens of similar extensions if you have a favourite, but you’ll need to change the configuration below to match your chosen extension):

Next, add this to .vscode/settings.json:

"runOnSave.commands": [
    {
        "match": ".*\\.py$",
        "command": "${workspaceFolder}/.venv/bin/python -m stbt_rig -v snapshot",
        "runIn": "terminal",
        "runningStatusMessage": "Running stbt_rig snapshot...",
        "finishStatusMessage": "Snapshot complete"
    }
],

Run a test from your IDE

IDEs like PyCharm and Visual Studio Code can detect the test functions in a file and show a small button to run the test:

_images/vscode-run-test.png

With the configuration below, clicking the button will sync your local code-changes and run the test remotely (on your Stb-tester Node). This is useful during test-development, when you are writing or debugging the test.

Before setting this up in your IDE, make sure that you can run python -m stbt_rig -v snapshot from your terminal, to debug any authentication issues. See stbt_rig installation and stbt_rig authentication here.

PyCharm

First go to File > Settings and select Tools > Python Integrated Tools. Under Testing: Default test runner select pytest. Click OK.

Next, go to Run > Edit Configurations and select Templates > Python tests > pytest. Fill in the following settings:

  • Target: Module name
  • Additional Arguments: -p stbt_rig --node-id=stb-tester-123456789abc
  • Add content roots to PYTHONPATH: True

(Change “stb-tester-123456789abc” to the actual ID of your Stb-tester Node.)

Note that you aren’t creating a new configuration (don’t click the little “+” icon on the left-hand pane). You are editing the Template from which PyCharm creates new configurations on the fly for each test.

The configuration should now look like:

_images/configuration.png

You can now run the current test with Ctrl-Shift-F10 or by clicking the green play button in the margin on the left and selecting “Run ‘pytest for …’”:

_images/pycharm-run-test.png

VS Code

Run Python: Configure Tests from the Command Palette (Control+Shift+P) and select pytest. When it asks for the directory containing the tests, choose tests.

Now open .vscode/settings.json and change python.testing.pytestArgs to this:

"python.testing.pytestArgs": [
    "-p", "stbt_rig", "--node-id=stb-tester-123456789abc",
    "--override-ini=python_files=*.py",
    "--override-ini=python_functions=test_*",
    "--tb=no", "--capture=no",
    "tests"
],

(Change “stb-tester-123456789abc” to the actual ID of your Stb-tester Node.)

Now run Python: Discover Tests from the Command Palette (Control+Shift+P) to identify each test. This adds a little “Run Test” button just above each testcase:

_images/vscode-run-test.png

After the test finishes, its output appears in the Output window: View > Output (Control+K Control+H) and select Python Test Log.

Troubleshooting

If any of the steps above don’t work, make sure that you have the latest version of stbt_rig (use pip install --upgrade stbt_rig).

Make sure that you can run python -m stbt_rig -v snapshot from your terminal. This will prompt for your access token and save it into your operating system’s keyring for future invocations. See stbt_rig.

VS Code

After following all the instructions above, your .vscode/settings.json should look like this:

{
    "python.linting.pylintEnabled": true,
    "python.linting.enabled": true,
    "python.linting.pylintArgs": ["--load-plugins=stbt.pylint_plugin"],

    "runOnSave.commands": [
        {
            "match": ".*\\.py$",
            "command": "${workspaceFolder}/.venv/bin/python -m stbt_rig -v snapshot",
            "runIn": "terminal",
            "runningStatusMessage": "Running stbt_rig snapshot...",
            "finishStatusMessage": "Snapshot complete"
        }
    ],

    "python.testing.pytestArgs": [
        "-p", "stbt_rig", "--node-id=stb-tester-123456789abc",
        "--override-ini=python_files=*.py",
        "--override-ini=python_functions=test_*",
        "--tb=no", "--capture=no",
        "tests"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true
}