IDE Configuration

Quick summary for Python experts:

  1. Install stb-tester into your project’s virtualenv with pip install stb-tester. This enables code completion / IntelliSense, but it doesn’t allow running the test scripts locally (see the note in the first section below for details).
  2. Add stbt-specific linting checks by configuring pylint to run with pylint --load-plugins=stbt.pylint_plugin.
  3. Configure your IDE’s “Save” button to run python stbt_rig.py snapshot. This syncs your local code to the Stb-tester Portal on a branch called “<your username>/snapshot” — without affecting your local git checkout.
  4. Configure your IDE to sync & run a single test remotely (on your Stb-tester Node) when you click “Run Test” from the IDE.

For more details, read on.

IntelliSense for stbt APIs

Install the “stb-tester” package using pip:

pip install stb-tester

Note

This package doesn’t include video-capture infrastructure and other complex dependencies, so it won’t allow you to run your test scripts locally (on your PC). It does enable your IDE’s “IntelliSense” or “Code Insight” features such as code completion, type inference, inline documentation, and linting.

If you haven’t yet upgraded your test-pack to use the latest stbt version, you can tell pip to install a specific version like this:

pip install stb-tester~=31.0

Note that ~=31.0 will install the latest 31 package, but not 32 or greater.

Make sure you install the same version locally as the version specified in your test-pack’s configuration file (see test_pack.stbt_version in the Configuration Reference). The stb-tester pip package is only available for version 31 and newer.

Enable Stb-tester’s pylint plugin

Pylint is a static analysis tool for Python code. This means that it analyses your code to catch common mistakes, without actually running your code. This saves a lot of development time by catching the mistakes immediately —your IDE shows red squiggles under each mistake— instead of running the test and waiting for it to reach the line of code where the mistake is.

Stb-tester provides a Pylint plugin that teaches Pylint about some mistakes that are specific to Stb-tester APIs. For example, the frame parameter of stbt.match is usually optional, but it’s mandatory when called inside a FrameObject property:

images/stbt-frame-object-missing-frame.png

PyCharm

PyCharm uses its own built-in linter; it doesn’t allow using external linters such as pylint. It will still catch common Python mistakes, but not the specific stbt checks like the example above.

VS Code

Run Python: Select Linter from the Command Palette (Control+Shift+P) and select pylint.

Next, add the following pylint args in .vscode/settings.json:

{
    "python.linting.pylintArgs": ["--load-plugins=stbt.pylint_plugin"],

Disabling annoying pylint warnings

Some of Pylint’s built-in checks are related to coding style, not real problems. You can disable specific checks by listing them in the .pylintrc file in the root of your test-pack. For example:

[MESSAGES CONTROL]
disable=
  line-too-long,
  missing-docstring,

The name of each checker (such as “line-too-long”) is shown after the error message in your IDE.

Sync the code from your IDE to the Stb-tester portal

Bind the “Save” button of your IDE to run python stbt_rig.py snapshot after saving. This will push your local code to the Stb-tester Portal on a branch called “<your username>/snapshot” — without affecting your local git checkout.

(After you have tested your changes & got them working, you still need to do “git commit” and “git push” to make it official. See Development Workflow.)

Before setting this up in your IDE, make sure that you can run python stbt_rig.py snapshot from your terminal, to debug any authentication issues. See stbt-rig installation and stbt-rig authentication here.

PyCharm

In Preferences > Plugins install the File Watchers plugin.

Now in Preferences > Tools > File Watchers add a new watcher with the following settings:

  • Name: stbt_rig.py snapshot
  • File type: Any
  • Scope: Project Files
  • Program: python
  • Arguments: stbt_rig.py snapshot
  • Working directory: path to your test-pack checkout
images/file-watchers-configuration.png

VS Code

In VS Code’s Extensions Marketplace (Ctrl+Shift+X) install the Run on Save extension by pucelle (or any of the dozens of similar extensions if you have a favourite, but you’ll need to change the configuration below to match your chosen extension):

Next, edit .vscode/settings.json:

{
  "runOnSave.commands": [
      {
          "match": ".*\\.py$",
          "command": "${workspaceRoot}/.venv/bin/python ${workspaceRoot}/stbt_rig.py snapshot",
          "runIn": "terminal",
          "runningStatusMessage": "Running stbt_rig.py snapshot...",
          "finishStatusMessage": "Snapshot complete"
      }
  ]
}

Run a test from your IDE

IDEs like PyCharm and Visual Studio Code can detect the test functions in a file and show a small button to run the test:

images/vscode-run-test.png

With the configuration below, clicking the button will sync your local code-changes and run the test remotely (on your Stb-tester Node). This is useful during test-development, when you are writing or debugging the test.

Before setting this up in your IDE, make sure that you can run python stbt_rig.py snapshot from your terminal, to debug any authentication issues. See stbt-rig installation and stbt-rig authentication here.

PyCharm

First go to File > Settings and select Tools > Python Integrated Tools. Under Testing: Default test runner select pytest. Click OK.

Next, go to Run > Edit Configurations and select Templates > Python tests > pytest. Fill in the following settings:

  • Target: Module name
  • Additional Arguments: --node-id=stb-tester-123456789abc -p no:python -p stbt_rig
  • Add content roots to PYTHONPATH: True

(Change “stb-tester-123456789abc” to the actual ID of your Stb-tester Node.)

Note that you aren’t creating a new configuration (don’t click the little “+” icon on the left-hand pane). You are editing the Template from which PyCharm creates new configurations on the fly for each test.

The configuration should now look like:

images/configuration.png

You can now run the current test with Ctrl-Shift-F10 or by clicking the green play button in the margin on the left and selecting “Run ‘pytest for …’”:

images/pycharm-run-test.png

VS Code

Run Python: Configure Tests from the Command Palette (Control+Shift+P) and select pytest. When it asks for the directory containing the tests, choose tests.

Now open .vscode/settings.json and change python.testing.pytestArgs to this:

"python.testing.pytestArgs": [
    "-p", "stbt_rig", "--node-id=stb-tester-123456789abc",
    "-o", "python_files=*.py",
    "-o", "python_functions=test_*",
    "--tb=no", "-s",
    "tests"
]

(Change “stb-tester-123456789abc” to the actual ID of your Stb-tester Node.)

You also need to add the project’s root folder to PYTHONPATH so that pytest can find the stbt_rig module. Add this to the project’s .env file (create the file if it doesn’t exist):

PYTHONPATH=${workspaceFolder}:${PYTHONPATH}

Finally, run Python: Discover Tests from the Command Palette (Control+Shift+P) to identify each test. This adds a little “Run Test” button just above each testcase:

images/vscode-run-test.png

After the test finishes, its output appears in the Output window: View > Output (Control+K Control+H) and select Python Test Log.

Troubleshooting

If any of the steps above don’t work, make sure that you have the latest version of stbt_rig.py from <https://github.com/stb-tester/stbt-rig/raw/master/stbt_rig.py>.

Make sure that you have installed stbt_rig.py’s dependencies: See stbt-rig installation.

Make sure that you can run python stbt_rig.py snapshot from your terminal, which will prompt for your access token and save it into your operating system’s keyring for future invocations. See stbt-rig authentication.