There are several mature open source and proprietary tools for testing the user interface of mobile apps on Android and iOS devices, but there aren’t many solutions for internet service providers or video-on-demand streaming services who want to test video playback on real mobile devices to validate their end-to-end video delivery. In this article I’ll discuss various ways of getting video frames from a mobile device for automated testing, and the advantages and trade-offs of each approach.

Automated user-interface testing tools for mobile apps tend to test an internal representation of the UI — a hierarchical structure of nodes such as the example shown here for the “Now TV” app. (If you’re familiar with web development, this is similar to a web page’s DOM.) Tools like Appium or Calabash can check that a node with the ID “videoplayer” is present, but they can’t check whether the video player is actually working.

Capturing screenshots over USB

These tools do have the capability to capture screenshots, which you could analyse with Stb-tester’s image-processing APIs. Unfortunately, these screenshots may not include the video plane, depending on the app. For example, you don’t see video content with Netflix, but you do with Now TV:

App	iPhone	Android
Netflix		Returns invalid image data
Now TV

Presumably the difference is due to different DRM technologies used by the two apps.

Note that the white text in the Netflix screenshot is a subtitle, which is layered on top of the video by the Netflix app. Only the video itself is missing from the screenshots.

The frame rate (of frames delivered to the test script) is poor, because the mobile device is encoding and transferring large PNGs. Expect around 2 to 0.2 frames per second depending on the device and the complexity of the image in a particular frame. This might increase the load on the device-under-test, which could affect subtle timing conditions in your app — if your test is trying to reproduce some kind of intermittent defect, its results might not be representative.

The great advantage of this method is that it is simple, it doesn’t require any test hardware, and it is provided by many mobile testing tools. Stb-tester supports this method (we use ADB to capture screenshots from Android devices, and WebDriverAgent for iOS).

Capturing video over HDMI

Apple iPhones & iPads can output video to HDMI using a Lightning Digital AV Adapter. Similarly, some (but not all) Android devices can output HDMI using an MHL cable. See a full list of supported Android devices here.

Once again, it’s up to the app to support this. This time the situation is reversed: Netflix is happy to play the video over HDMI, whereas Now TV brings up a dialog saying “Sorry, we don’t support HDMI streaming to your TV from mobile devices”.

App	iPhone	Android
Netflix
Now TV

Note that the Netflix behaviour is slightly different across platforms. On Android the device’s screen is mirrored to the HDMI output, so UI elements like the playback controls appear on both screens. On iOS the video plays on the HDMI output while the device goes into “second screen remote control” mode: The device’s screen shows a static image plus the playback controls.

When you’re not playing video, all devices mirror the screen to the HDMI output. This works in all apps including the operating system home screen. When the device is in portrait mode you get some letterboxing on the sides (HDMI doesn’t support portrait resolutions).

A big disadvantage of this method is that it doesn’t leave a USB connection available, so the PC that is running the tests will have to find another way to send input events (simulated taps & swipes) to the mobile device. The Apple adapter and Android MHL cables all have an additional USB connector, but this is used exclusively to power the mobile device. On Android devices you can work around this by enabling ADB over TCP/IP, but this requires a USB connection to set up in the first place so it will require manual intervention if the device reboots or loses WiFi connection.

Capturing video with a camera

This is the most “black box” approach of all, so it works with any device and any app. Just point a camera at the device’s screen!

Our product, the Stb-tester CAMERA, performs automatic calibration on each video frame in real time so your test script sees only the mobile device’s screen, regardless of the camera’s position.

We have made the Stb-tester CAMERA extremely straight-forward to install and configure. Nevertheless, it does have some disadvantages: There is more hardware and it takes up more space. You need to cover the mobile device and camera with a box to eliminate reflections on the screen. The analogue nature of the process introduces some lossiness, and in particular some motion blur.

Summary of video-capture methods

The following table summarises the pros & cons of the various methods for capturing video frames from iOS & Android devices:

	USB	HDMI	CAMERA
Works on all devices	Yes.	No.	Yes.
Works on all apps	No.	No.	Yes.
Allows sending input events over USB	Yes.	No.	Yes.
Black box	No.	Yes.	Yes.
Frame rate	Slow	Full source frame rate	Camera’s frame rate
Picture quality	Lossless	Lossless	Lossy
Additional hardware required	None	HDMI video-capture	Camera

Our recommendations

If you don’t need to verify video playback and you just want to test user interaction within an app, we recommend a tool like Appium, which is tailored specifically for mobile UI testing. Services like Sauce Labs and Testdroid can even host the mobile devices for you, so you don’t have to deal with dozens of physical devices.

If you’re already using Stb-tester to test a set-top box (using HDMI video-capture) and you want to test the interaction between the set-top box and a “second screen” mobile app (remote control, TV guide, etc): Use USB screenshot capture from your existing Stb-tester scripts.

If you really do need a black-box view of your device’s screen (to verify video playback on any app & device, especially useful for monitoring teams): Use the Stb-tester CAMERA.

---