Blobs processing
Previous tutorial: Video repeaters
The version 1.2.2 of Computer Vision Sandbox comes with a new module containing set of plug-ins aimed for blobs processing. For example, there are plug-ins, which can find the biggest blob in an image, filter blobs by size/area, fill holes in them, etc. All of these plug-ins rely on the fact that image's background is filled with black color (zero values), while blobs (or objects) are represented with any other non-zero values. A single blob is simply a group of pixels connected with each other. Bellow are two sample images, which demonstrate the type of images expected by different blobs processing plug-ins. The image on the left has 3 blobs, while the image on the right has 7 blobs.
Images like the above usually don't come for free out of a video source. In most cases some sort of pre-processing is required to segment images into background and objects of interest. Below is the type of source image we'll be dealing with and we'll see how different blobs processing plug-ins could be used to locate red and blue objects in it.
To locate red and blue objects in the above image, we'll use nRGB color space this time. Using Extract nRGB Channel we can extract any of the nRGB channels out of a color image, which will be used then for further processing. However, before doing that we'll use Color Filter first to get rid of dark colors - this will reduce noise in the nRGB channels.
-- Create instances of plug-ins to use and configure those
colorFilter = Host.CreatePluginInstance( 'ColorFilter' )
colorFilter:SetProperty( 'redRange', { 0, 60 } )
colorFilter:SetProperty( 'greenRange', { 0, 60 } )
colorFilter:SetProperty( 'blueRange', { 0, 60 } )
colorFilter:SetProperty( 'fillOutside', false )
extractRedChannel = Host.CreatePluginInstance( 'ExtractNrgbChannel' )
extractRedChannel:SetProperty( 'channel', 0 )
extractBlueChannel = Host.CreatePluginInstance( 'ExtractNrgbChannel' )
extractBlueChannel:SetProperty( 'channel', 2 )
...
function Main( )
-- Get image to process
image = Host.GetImage( )
colorFilteredImage = colorFilter:ProcessImage( image )
-- Red channel
redChannel = extractRedChannel:ProcessImage( colorFilteredImage )
-- Blue channel
blueChannel = extractBlueChannel:ProcessImage( colorFilteredImage )
...
end
The above images demonstrate normalized red (left) and normalized blue (right) channels. As we can see, regions corresponding to red areas in the color image have higher intensities in the nR image. Similar about the nB image - its higher intensities tell us about something blue in the source color image. To prepare these images for further blobs processing, we need to use Threshold plug-in to segment objects of interest from the rest (from the background).
...
threshold = Host.CreatePluginInstance( 'Threshold' )
threshold:SetProperty( 'threshold', 100 )
function Main( )
...
threshold:ProcessImageInPlace( redChannel )
threshold:ProcessImageInPlace( blueChannel )
end
Now we have nice and clean images showing us red and blue areas. Well, these images are a bit noisy and contain some unwanted objects, which are either too small or not so red/blue as we may want them to be. But this is where blobs processing will help us.
The first blobs processing plug-in to try is Keep Biggest Blob. As the name suggests, this plug-in removes all blobs from an image except the biggest one. It finds the biggest blob either by the size of its bounding rectangle (width multiplied by height) or by blob's area.
...
biggestBlobFilter = Host.CreatePluginInstance( 'KeepBiggestBlob' )
-- Use blob's area as size criteria
biggestBlobFilter:SetProperty( 'sizeCriteria', 1 )
function Main( )
...
biggestBlobFilter:ProcessImageInPlace( redChannel )
biggestBlobFilter:ProcessImageInPlace( blueChannel )
end
In addition to removing blobs, the Keep Biggest Blob plug-in also provides position and area of the kept blob, which might be used for whatever purpose. For example, using functions of the Image Drawing plug-in it can be possible to highlight the blobs on the original image.
In some cases it is not actually required to perform removing other blobs just because we need to find position/area of the biggest blob. For example, to perform blobs' highlighting like it is demonstrated above, it is not needed. The Keep Biggest Blob plug-in is good in cases when it is required to get a mask image, for example, which later can be used to extract the biggest blob from the original image using Mask Image plug-in. However, like in the case above, the Find Biggest Blob plug-in could be used instead. The difference between these two is that Keep Biggest Blob is an image processing filter plug-in and so it supposed to change source image. But the Find Biggest Blob is an image processing plug-in (not filter; yes a bit confusing), which means it does not make any changes to the source image, but just calculates what in needs to calculate - blob's position and area. Not a big difference between these two plug-ins, but the later is faster, since it does not need to perform second pass over an image for removing blobs.
So we have images showing areas of biggest red and blue objects, which we may want to use as mask images. However, there is something to be done first with them. As it sometimes happens with color filtering, the obtained mask images may have holes in the found objects. Like it happened for us with the blue object, which got a bit too much light at the top and so has a patch looking less blue, but a bit more whiter. We could of play with threshold value applied to the normalized blue channel trying to improve result and get rid of the holes. However, this time we'll employ another blobs processing plug-in aimed for filling holes in objects - Fill Holes plug-in. All it does is filling all black holes inside found blobs with the specified color.
...
fillHoles = Host.CreatePluginInstance( 'FillHoles' )
function Main( )
...
fillHoles:ProcessImageInPlace( redChannel )
fillHoles:ProcessImageInPlace( blueChannel )
end
Now the above images look better without holes and can really serve as masks. Employing techniques described in another tutorial about two source image processing filters, these masks can be used to extract the objects from the original image or to extract everything else except objects, then either objects only or the rest can be processed and finally combined back to get a result like shown below.
Finding the biggest blob is nice, but is not enough in many cases. Sometimes it is required to find all blobs within certain size/area limits. For this purpose there is Filter Blobs By Size plug-in, which keeps blobs only within certain size limits and removes the rest. This allows us to find all red and blue objects of our interest.
...
blobFilter = Host.CreatePluginInstance( 'FilterBlobsBySize' )
-- Filter criteria is blobs' area
blobFilter:SetProperty( 'sizeCriteria', 1 )
-- Set the minimum allowed area to 250 pixels. Blobs smaller than that will be removed.
blobFilter:SetProperty( 'minArea', 250 )
function Main( )
...
blobFilter:ProcessImageInPlace( redChannel )
blobFilter:ProcessImageInPlace( blueChannel )
end
The Filter Blobs By Size plug-in also provides information about kept objects. The Blobs Removed and Blobs Left properties tell how many blobs were removed and how many were kept. For those, which were kept, there are Blobs Position, Blobs Size and Blobs Area properties, which tell about position and size of the kept blobs. As it was shown before, this information can be used to highlight all the found blobs.
The blobs processing plug-ins provide really nice techniques for finding blobs and getting information about them. Combined with different image processing filters, those can be used to achieve many interesting results.
All the above is only introduction to blobs processing, which demonstrates relatively simple image processing algorithms. Future versions of the blobs processing module may introduce additional plug-ins, which provide even more useful information about blobs and tools to work with them.
Next tutorial: Detection of circles and rectangles