How we test the stb-tester ONE: Video-capture latency & accuracy
05 Jul 2016.
Updated 17 May 2017 with figures for our new hardware’s accuracy.
Some of our users care about frame-accurate performance measurements. For example, how many frames of video does it take a menu transition to complete? The Stb-tester HDMI provides the following guarantees:
- Your test scripts can process every frame of video, up to 60 frames per second, using the APIs provided by stb-tester (like match and wait_for_motion) or with custom image-processing using OpenCV.
- Stb-tester reports the timestamp of each frame.
- The timestamp is in wall-clock time (UTC), so you can compare it against log entries from the device-under-test, your head-end systems, etc.
On the Stb-tester ONE the timestamp is accurate to within 35ms. At 60 frames per second, this is 2 frames of video.On our new Stb-tester HDMI hardware the timestamp is accurate to within 0.9ms (at 60 frames per second a single frame is 16ms, so stb-tester is providing frame-accurate timings by a wide margin).
- There is no long-term drift (we’ve tested every frame over 4 days of continuous test execution).
How we test & validate video-capture timestamps
Our test system consists of an stb-tester ONE capturing the video from a Raspberry Pi’s HDMI output:
The Raspberry Pi is generating a video on the fly. Each frame of video has a time drawn on it in binary — this is the time at which the frame will be sent over HDMI, according to the Raspberry Pi’s MMAL video subsystem. We have open-sourced the code that generates and displays this video: See stb-tester/latency-clock on github.
The receiver (an stb-tester Python script) decodes the time drawn on the frame, and compares it against the time reported by the video-capture process.
The Raspberry Pi uses a GPS-stabilised clock, by using a GPS expansion board.
This keeps the Rasperry Pi’s system clock stable to ~20µs (as reported by
ppstest). The Raspberry Pi also runs an NTP server; the stb-tester ONE runs
an NTP client that synchronises against the Raspberry Pi’s clock, to an
accuracy of ~300µs (as reported by
ntpq). For instructions see
Not quite 5 minute guide to making an NTP Server.