Introduction to Scripting

Previous tutorial: Virtual video camera

 

The set of previous tutorials was talking about different sorts of image and video processing plug-ins and how to apply them to video sources running in Computer Vision Sandbox. Using different combinations of those plug-ins may allow achieving many different results. However there are certain limits to what can be done by just combining plug-ins into a video processing graph. The main limit so far was that once video processing steps are added and configured there is little can change during run time of the graph - all the properties stay like they were configured giving no variation between processed video frames. To brake these limits it was required to have some sorts of scripting capabilities, which would allow running scripts as part of video processing graph - scripts, which decide what to do and how to do at the moment they run. And so new plug-in type was added - Scripting Engine.

Version 1.2.0 of the Computer Vision Sandbox introduces support of the new Scripting Engine plug-in type and the first plug-in of this type is provided as well - Lua Scripting. The plug-in allows running scripts written in Lua programming language, which can interact with application through the special API and perform some custom actions. So far the scripting API is targeted to custom video processing done while running video processing graph. Similar to image and video processing plug-ins the Lua Scripting plug-in can be also put as a step into a video processing graph. However what this step does is completely up to the script executed by the scripting plug-in.

The Lua Scripting plug-in has a single property only, which is a file name of the Lua script to run. The file name can be either set using a file browser dialog or a built-in script editor can be used to create new or edit existing scripts.

Before diving to study the details of the API provided by the Lua Scripting plug-in, let's have a look at a simple script, which will not do much for now, but still more than can be achieved by running a plug-in as is from a video processing graph. All the script will do is keep changing Hue value of video frames. Just changing hue to some predefined value can be perfectly done as a video processing step based on Colorize plug-in, for example. However we would like to change the value at run time, which is where scripting comes handy. So here it is:

-- Create instance of Colorize plug-in
setHuePlugin = Host.CreatePluginInstance( 'Colorize' )
-- Set Saturation to maximum
setHuePlugin:SetProperty( 'saturation', 100 )
-- Start with Hue set to 0
hue = 0

-- Main function to be executed for every frame
function Main( )
    -- Get image to process
    image = Host.GetImage( )
    -- Set Hue value to set for the image
    setHuePlugin:SetProperty( 'hue', hue )
    -- Process the image
    setHuePlugin:ProcessImageInPlace( image )
    -- Move to the next Hue value
    hue = ( hue + 1 ) % 360
end

The above script is well commented, however lets go through it briefly. The initialization part of it uses host API to create an instance of an image processing plug-in to be used and sets one of it's properties to some value. Nothing new - same can be done easier using video processing graph with less typing. The Main function is then gets called for every video frame going through the processing graph and does the rest of the job - it uses host API to get the current image and processes it with the plug-in created during initialization. However what it does in between is setting another property to a value, which is not constant, but is calculated from the script at run time - something which is beyond capabilities of the preconfigured steps of a video processing graph.

Of course the above sample script is very simple. But it shows the main idea - by setting plug-ins' configuration at run time, making decisions of which plug-ins to use or not, changing execution flow, etc., it is possible to get unique results, which would be much harder to get in the absence of scripting capabilities. And while the number of available plug-ins is growing and the API gets extended, the application of scripting will spawn into many more different areas.

Although the Lua scripting API page guides through the details of its usage, there is one thing to note about getting information of plug-ins' and their properties names. The names of plug-ins and their properties used in scripting are very similar to those displayed in the user interface of Computer Vision Sandbox, however are not always exactly the same. To find those it is possible to use help system, where an option is added to show/hide scripting related information.

 

Further reading:

 

Next tutorial: Switching effects at run time