Finding GUI elements by color#

stbt.find_regions_by_color finds contiguous regions of a particular color. It can be used to identify GUI elements such as the “selection” or “focus” on menus and carousels.

For example, let’s find the current focus in the menu at the top of this screenshot:

We’ll match the logo in the top-left corner to check if the menu is present, and we’ll use find_regions_by_color to find the position of the white underline or “focus”:

Here’s a Page Object that implements this:

import stbt

class Menu(stbt.FrameObject):

    @property
    def is_visible(self):
        return bool (stbt.match("BT SPORT.png", frame=self._frame)
                     and self.underline)

    @property
    def underline(self):
        r = stbt.find_regions_by_color(
            "#fff",
            frame=self._frame,
            mask=stbt.Region(x=0, y=80, right=1280, bottom=90),
            min_size=(20, 2))
        if len(r) == 1:
            return r[0]
        else:
            return None

find_regions_by_color takes a color in either of the following formats:

A string in the CSS hexadecimal color format: White is "#ffffff" or "#fff".
A tuple of 3 numbers in Blue, Green, Red order (note that this is the opposite of the CSS Red, Green, Blue order). White is (255, 255, 255).

It returns a list of regions. Here we are looking in the narrow band where we expect the white underline to be, so we only expect to find 1 region. If we find any more or any less, it’s probably a different page so we made is_visible return False.

Now we can use ocr to read the text above the underline:

class Menu(stbt.FrameObject):
    ...

    @property
    def focus(self):
        return stbt.ocr(
            frame=self._frame,
            region=self.underline
                       .above()
                       .extend(x=-10, right=10, bottom=-5))

We can test our Page Object in the Object Repository (simulated below). You can see that for each example screenshot, our underline property is finding the correct position (try hovering over the stbt.Region text in the table) and our focus property is reading the correct menu entry.

	channels.png	football.png
x	<Menu>	<Menu>
x.is_visible	True	True
x.focus	Channels	Football
x.underline	stbt.Region(x=205, y=83, right=285, bottom=85)	stbt.Region(x=323, y=83, right=397, bottom=85)