Detection of square binary glyphs

Previous tutorial: Capturing screen and windows

 

The 1.2.7 version of Computer Vision Sandbox includes a new module, which provides the Glyph Detector plug-in to detect square binary glyphs. Detection of these simple shapes can be used in a number of applications - some simple augmented reality, giving commands to a robot, helping robot to navigate in some environment, etc.

Before using the plug-in, let's have a look at what these square binary glyphs are (or just glyphs to keep it short). A glyph is a NxN grid of square cells, where each cell is filled with either white color or black color (or another dark enough color). To make them easier to be detected, each glyph has a row/column of black cells on its sides. So, a 3x3 glyph becomes a 5x5 structure, where all edge cells must be black and all inner cells represent content - white or black cells (the pattern to be recognized). Below is a sample of few 3x3 and 6x6 glyphs.

Each of these glyphs is printed on white paper and then can be used with the Glyph Detector plug-in to find them. When printing, make sure a glyph itself is surrounded by some white area, so that an edge detector could locate it. Here is how it may look when it is all done.

At this point the Glyph Detector plug-in is implemented as image processing plug-in and so can not be put directly into video processing graph from the user interface of sandbox wizard. Instead, it can be used from scripting only for now. However, this is easy to do. All we need is to call ProcessImage() method of the Glyph Detector plug-in and then get the information we need by querying properties of the plug-in. The code below does detection of glyphs and outputs the number of detected glyphs. 

local table = require "table"

-- Create instances of plug-ins to use
detectorPlugin = Host.CreatePluginInstance( 'GlyphDetector' )
drawingPlugin  = Host.CreatePluginInstance( 'ImageDrawing' )

-- Detect up to 7 glyphs of 3x3 size
detectorPlugin:SetProperty( 'glyphSize', 3 )
detectorPlugin:SetProperty( 'maxGlyphs', 7 )

-- Colors used to highlight corners of glyphs
cornerColors = { 'FF0000', '880000', 'FFAAAA', 'FF8800' }

function Main( )
    image = Host.GetImage( )

    -- Find glyphs in the current image
    detectorPlugin:ProcessImage( image )

    glyphsFound = detectorPlugin:GetProperty( 'glyphsFound' )
    glyphQuads  = detectorPlugin:GetProperty( 'glyphQuadrilaterals' )
    glyphCodes  = detectorPlugin:GetProperty( 'glyphCodes' )

    -- Tell how many glyphs are detected
    drawingPlugin:CallFunction( 'DrawText', image, 'Found glyphs: ' .. tostring( glyphsFound ),
                                { 5, 5 }, 'FF0000', '00000000' )
                                
    -- TODO
end

The above code does not do much - just the number of detected glyphs. Lets put some more into it - lets iterate through each detected glyph, highlight it with a polygon, highlight its four corners and finally output the code string of the pattern encoded by the glyph (the code replaces of the above 'TODO' comment).

    -- Highlight each detected glyph
    for i = 1, glyphsFound do
        quad = glyphQuads[i]
        rightMostPoint = quad[1]

        drawingPlugin:CallFunction( 'DrawPolygon', image, quad, 'FF0000' )

        -- Four corners of each rectangle
        for j = 1, 4 do
            drawingPlugin:CallFunction( 'FillRectangle', image,
                                        { quad[j][1] - 3, quad[j][2] - 3 },
                                        { quad[j][1] + 3, quad[j][2] + 3 },
                                        cornerColors[j] )

            if quad[j][1] > rightMostPoint[1] then
                rightMostPoint = quad[j]
            end
        end

        -- Tell the code of the glyph
        drawingPlugin:CallFunction( 'DrawText', image, glyphCodes[i],
                                    { rightMostPoint[1] + 4, rightMostPoint[2] },
                                    'FF0000', '00000000' )
    end

And here is how it may look like, when the above script is used with the Lua Scripting plug-in put into video processing graph:

Notice how glyph decoding works. The first glyph's corner provided by the plug-in is always the left most. All others then go in clockwise direction. Glyph decoding is then done line by line in the direction specified by the vector connecting first and second corners. Check the above picture and notice that '1' in glyph's code string corresponds to white cells, but '0' corresponds to black cells.

Since the Glyph Detector plug-in provides code string for each detected glyph, it is possible to distinguish them from each other (i.e. recognize them). For each code string it is possible to associate an image, which is first loaded with Png Importer plug-in (or Jpeg Importer) and then embedded into glyph's quadrilateral using the Embed Quadrilateral plug-in. The result may look like some simple 2D augmented reality.

As it was already mentioned before, the Glyph Detector plug-in can be used in some interesting detection/recognition applications. The one currently in progress here is a small robotics game, where a robot must be driven to find hidden glyphs within given amount of time. But, more on this later.

Note: here is complete script for detection of square binary glyphs.

 

Next tutorial: Detection of 1D linear bar codes