Updated 17 May 2017 with figures for our new hardware’s accuracy.

Some of our users care about frame-accurate performance measurements. For example, how many frames of video does it take a menu transition to complete? The Stb-tester HDMI provides the following guarantees:

• Your test scripts can process every frame of video, up to 60 frames per second, using the APIs provided by stb-tester (like match and wait_for_motion) or with custom image-processing using OpenCV.
• Stb-tester reports the timestamp of each frame.
• The timestamp is in wall-clock time (UTC), so you can compare it against log entries from the device-under-test, your head-end systems, etc.
• On the Stb-tester ONE the timestamp is accurate to within 35ms. At 60 frames per second, this is 2 frames of video. On our new Stb-tester HDMI hardware the timestamp is accurate to within 0.9ms (at 60 frames per second a single frame is 16ms, so stb-tester is providing frame-accurate timings by a wide margin).
• There is no long-term drift (we’ve tested every frame over 4 days of continuous test execution).

How we test & validate video-capture timestamps

Our test system consists of an stb-tester ONE capturing the video from a Raspberry Pi’s HDMI output:

The Raspberry Pi is generating a video on the fly. Each frame of video has a time drawn on it in binary — this is the time at which the frame will be sent over HDMI, according to the Raspberry Pi’s MMAL video subsystem. We have open-sourced the code that generates and displays this video: See stb-tester/latency-clock on github.

The receiver (an stb-tester Python script) decodes the time drawn on the frame, and compares it against the time reported by the video-capture process.

The Raspberry Pi uses a GPS-stabilised clock, by using a GPS expansion board. This keeps the Rasperry Pi’s system clock stable to ~20µs (as reported by ppstest). The Raspberry Pi also runs an NTP server; the stb-tester ONE runs an NTP client that synchronises against the Raspberry Pi’s clock, to an accuracy of ~300µs (as reported by ntpq). For instructions see Not quite 5 minute guide to making an NTP Server.