Handling UI elements of non-fixed size
25 Aug 2015. By William Manley.
Recently we have been writing tests involving extracting information from BBC iPlayer on some smart TVs.
When you just start playing content iPlayer shows an overlay providing some information about the content being played:
We wanted to read the title of the current piece of content so we can make some assertions about it. For example:
We need to know the location of the title on screen before we can perform OCR. For most static UIs we can hard-code it.
Naturally we use the page object pattern to improve the self-testability and clarity of our tests. The implementation for this screen looks like:
We test this against the screenshot we captured above and it works:
Great. But wait! not all iPlayer overlays are identical. Some are taller than others depending on the content and some are in a different position:
In the previous example we hard-coded the location of the overlay with
info_box_region = Region(x=194, y=437, width=763, height=173). This will
title helper function to produce the wrong title for this
Fortunately the iPlayer overlay is nonetheless fairly regular:
- It is always rectangular
- It is opaque and always has a constant background colour
- It is always near the bottom of the screen
- It is always roughly centred
We can take advantage of these facts to locate it.
Fortunately OpenCV (the library stb-tester uses for image processing under the
hood) provides a function for finding connected areas of constant colour:
cv2.floodFill. Among other things it takes an image and a point
and returns the bounding rectangle of the connected areas.
Using this we can create a helper function to integrate it with our test library:
We know that the pixel at
(640, 608) is always a background pixel within the
overlay so we will use that as the
point parameter. Our
IPlayerContentOverlay class is modified like so:
Note that we are now dynamically working out the location of the overlay with
info_box_region = find_contiguous_region((640, 608), self.frame).
And now it works for both screenshots:
We are currently considering adding
find_contiguous_region or some
generalisation of it to stb-tester proper.