Handling UI elements of non-fixed size
25 Aug 2015. By William Manley.
Recently we have been writing tests involving extracting information from BBC iPlayer on some smart TVs.
When you just start playing content iPlayer shows an overlay providing some information about the content being played:
We wanted to read the title of the current piece of content so we can make some assertions about it. For example:
assert overlay.title == "The One Show"
We need to know the location of the title on screen before we can perform OCR. For most static UIs we can hard-code it.
Naturally we use the page object pattern to improve the self-testability and clarity of our tests. The implementation for this screen looks like:
class IPlayerContentOverlay(object): def __init__(self, frame=None): if frame is None: frame = stbt.get_frame() self.frame = frame @property def title(self): info_box_region = stbt.Region(x=194, y=437, width=763, height=173) title_region = info_box_region.replace(height=36) return stbt.ocr(region=title_region, frame=self.frame)
We test this against the screenshot we captured above and it works:
>>> overlay = IPlayerContentOverlay(frame=screenshot1) >>> overlay.title The One Show
Great. But wait! not all iPlayer overlays are identical. Some are taller than others depending on the content and some are in a different position:
In the previous example we hard-coded the location of the overlay with
info_box_region = Region(x=194, y=437, width=763, height=173). This will
title helper function to produce the wrong title for this
Fortunately the iPlayer overlay is nonetheless fairly regular:
- It is always rectangular
- It is opaque and always has a constant background colour
- It is always near the bottom of the screen
- It is always roughly centred
We can take advantage of these facts to locate it.
Fortunately OpenCV (the library stb-tester uses for image processing under the
hood) provides a function for finding connected areas of constant colour:
cv2.floodFill. Among other things it takes an image and a point
and returns the bounding rectangle of the connected areas.
Using this we can create a helper function to integrate it with our test library:
def find_contiguous_region(point, frame=None): """ This function can be used to find contiguous regions of a single colour. This is useful when a UI element is in a constant position but has non-constant size. Args: frame (numpy.ndarray): An image point (tuple): A 2-tuple (x, y) coordinates of the location in the image within the region you want to find. Returns stbt.Region: The bounding box of contiguous colour from the given point. """ import cv2 if frame is None: frame = stbt.get_frame() _, rect = cv2.floodFill( frame, None, point, None, (1, 1, 1), (1, 1, 1), flags=cv2.FLOODFILL_FIXED_RANGE | cv2.FLOODFILL_MASK_ONLY) region = stbt.Region(*rect) stbt.debug("find_contiguous_region(..., %s) -> %s" % (point, str(region))) return region
We know that the pixel at
(640, 608) is always a background pixel within the
overlay so we will use that as the
point parameter. Our
IPlayerContentOverlay class is modified like so:
class IPlayerContentOverlay(object): def __init__(self, frame=None): if frame is None: frame = stbt.get_frame() self.frame = frame @property def title(self): info_box_region = find_contiguous_region((640, 608), self.frame) title_region = info_box_region.replace(height=36) return stbt.ocr(region=title_region, frame=self.frame)
Note that we are now dynamically working out the location of the overlay with
info_box_region = find_contiguous_region((640, 608), self.frame).
And now it works for both screenshots:
>>> overlay = IPlayerContentOverlay(frame=screenshot2) >>> overlay.title The One Show
We are currently considering adding
find_contiguous_region or some
generalisation of it to stb-tester proper.