Lint checks in pull requests

About pylint

Pylint is a static analysis tool for Python code. This means that it analyses your code to catch common mistakes, without actually running your code. This saves a lot of development time by catching the mistakes immediately —your IDE shows red squiggles under each mistake— instead of running the test and waiting for it to reach the line of code where the mistake is.

Stb-tester provides a Pylint plugin that catches some mistakes that are specific to Stb-tester APIs. For example, the frame parameter of stbt.match is usually optional, but it’s mandatory when called inside a FrameObject property:

VS Code showing an error caught by static analysis

Another common mistake is using a reference image that doesn’t exist on disk, or that hasn’t been committed to git:

Forgetting to commit the reference image is a frustrating mistake — but not if you catch it immediately!

Some Stb-tester APIs raise an exception on failure (for example stbt.wait_for_match) but others don’t (for example stbt.match and stbt.wait_until). This can be confusing, so we report an error if you forget to use the return value from the APIs that don’t raise:

If you don’t check the return value from stbt.wait_until, it’s probably a mistake

If you’re curious about the full list of checks that Stb-tester adds to Pylint, see the source code of our Pylint plugin here.

Lint checks in your IDE

To set up your IDE to catch all of these errors, see Development Environment Setup.

Lint checks in pull requests

When you push a git commit, Stb-tester uses any idle Node in your test-farm to run a lint check on all your code. We report the lint results using GitHub’s “checks” API, so the lint warnings appear directly in your pull request:

Lint failure annotation in GitHub pull request

The advantage of running lint on the Stb-tester Nodes is that we lint your code using exactly the same environment that we use to run your code when you run a test. This environment includes any third-party libraries that you install in your setup script, so Pylint will be able to check the way you call those libraries. For our customers who install in-house libraries from a server on a private network, this enables linting that isn’t possible on a cloud-based runner.

We run Pylint in the background on any idle Node. If you start running a test on a Node that was running Pylint, Pylint will be transparently re-scheduled onto a different Node so that it doesn’t interfere with your tests. All of this is completely automatic and requires no configuration.

Disabling Pylint warnings

In some rare cases you might want to ignore the return value for a valid reason. You can disable the warning for that specific line, by adding a comment like this:

Disable the warning on this line (but do warn me if I forget to check the return value in other places!)

Some of Pylint’s built-in checks are related to coding style, not real problems. If they get annoying you can disable specific checks by listing them in the .pylintrc file in the root of your test-pack. For example:

[MESSAGES CONTROL]
disable=
  line-too-long,
  missing-docstring,

The name of each checker (such as “line-too-long”) is shown after the error message in your IDE.