About MyScript Math SDK
MyScript Math SDK provides you with the API based on MyScript Math technology, enabling the recognition of handwritten equations and mathematical formulas.
In brief MyScript Math supplies:
Item  Description  Location 

API for C, Java, and .NET  Math API  edk/your_programming_language/ 
MyScriptMath.dll, MyScript2D.dll  The specific libraries  engine/ 
Resource files  These files contain specific data needed to perform math recognition.  resources/ 
MathRecognition  This code sample shows how to create an math document object, create a math recognizer, attach resources and a grammar to it, and get the parsed result, convert the parse tree to a LaTeX string and to a MathML format.  edk/your_programming_language/example/math/ 
Application blueprint
The figure below describes the MyScript Math SDK functioning.
A math recognition with MyScript Math SDK requires that:

you create a math recognizer. The math recognizer will analyze the spatial relationships between the parts of the math expression.

you attach resources to the recognizer. The data laid down in the resource files inform the recognizer on how to perform the segmentation of all the input ink parts.

you send the strokes and start the recognition that your application has gathered from user handwriting with an appropriate device (smartphone, tablet, interactive whiteboard, etc.).

you get back the recognition result in the form of a tree.
The following describes the main objects of MyScript Math environment in details.
Environment
Operations
MyScript Math supports the recognition of two main types of operations (each of which is handled with a specific grammar resource):
 Standard operations stand for common notations of mathematical expressions.
 Columnar operations refer to long divisions, multiplications in columns, additions in columns and subtractions in columns.
Additions  Subtractions  Multiplications 

Divisions 

Math recognizer
The MathRecognizer
is specialized for math recognition. The recognizer will process the input digital ink as a series of strokes.
Math Alphabet Knowledge
This is a special resource file used for the recognition of mathematical symbols. It provides “math alphabet knowledge” to the recognizer and makes it able to recognize the various elements constituting a mathematical expression. It is often referred to as AK.
Math Grammar Knowledge
The grammar knowledge is a resource file that you attached to the recognizer. It is a set of rules describing how to parse the mathematical expression. MyScript Math provides two types of grammar knowledge resources:

mathgrmstandard.res
, that is appropriate for recognizing standard mathematical expressions. 
mathgrmcolumnaroperations.res
, that is designed for recognizing columnar operations.
It is possible to create and compile custom grammar resources for the MyScript Math SDK.
Recognition result
The figure above shows a typical equation that could be recognized with MyScript Math SDK. The result is a tree which can be serialized to a LaTeX or MathML string. This tree is composed of different nodes.
Check section Nodes to learn more on nodes composing the resulting tree.
Check out the code sample provided with MyScript Math SDK to know how to map the result tree to LaTeX or MathML. It is located in the edk/[your_programming_language]/examples/
folder.
Running a recognition
A math recognition with MyScript Math SDK requires various steps hereby described:

Create a
MathRecognizer
, once you have set the MyScript environment through the creation of the engine object. You need a recognizer specialized for math recognition that is theMathRecognizer
. Use the methodcreateMathRecognizer
to do so. 
Load and attach the resources (AK and Grammar) to the recognizer. Go to this section to know more on the grammar resource.

Send the strokes that your application has gathered. You send the strokes into a
MathDocument
object using theaddStroke
method. 
Start the recognition using the method
process
. 
Get back the recognition result in the form of a tree using the method
getParseTree
. Learn how to parse the tree here.
Check also the Tutorial section or the code sample provided with MyScript Math SDK (in the edk/[your_programming_language]/examples/
folder) to learn in detail how to implement the whole Math recognition process.
Parsing the result
The recognizer outputs the recognition result in the form of a tree. You obtain this tree, after the recognition process, through the method getParseTree
. You can then explore the tree by using the methods from the different node types’ interfaces. Thus, to explore the result you need to:

Check the type of the node. Check whether the node is a
terminalNode
, anonTerminalNode
or aruleNode
with thevoGetType
method. 
Use the appropriate API to explore the components of a node. For example, if the node is a
ruleNode
, you use the methodsgetChildCount
,getChildAt
(among others) to get its components. This can return a node of any of the existing types. Check section Nodes to learn more on nodes’ APIs. 
Parse these nodes recursively .This means that you will need to reiterate from step 1 until you reach the end of the tree.
edk/
[language_name
]/src
.The structure of the tree is related to the relationship between the recognized symbols that compose the input ink. A tree is made of different nodes
. A node
describes a segment of the recognized mathematical expression. Inner segments of a single node
form new nodes that are attached to this parent node
. There are six types of nodes
:

terminalNode
. We callterminalNode
the elementary component of a mathematical expression. For instance, it can be a digit. 
nonTerminalNode
. We call anonTerminalNode
any group of symbols that is not elementary. Think of it as atoms and molecules. A molecule is made of several atoms like anonTerminalNode
is made of severalterminalNode
andruleNode
objects. For instance, the number42
is made of twoterminalNodes
:4
and2
.
cellNode
. AcellNode
is a particularnonTerminalNode
. It is used when dealing with columnar operations or matrices. 
borderNode
. AborderNode
is a particularnonTerminalNode
. It is used when dealing with columnar operations. AborderNode
models any type of border line (horizontal lines in columnar additions or vertical lines in columnar divisions, and so on.)


ruleNode
. AruleNode
describes the spatial relationship between groups ofnonTerminalNodes
andterminalNodes
.
tableNode
. AtableNode
is a particularruleNode
. It is used when dealing with columnar operations or matrices. ThetableNode
models the spatial relationship between every element of a columnar operation or in a matrix.

We refer to the whole mathematical expression as the root node.
The figure below illustrates the concept of nodes in a concrete example.
Nodes
This section details the node objects that you can find in the resulting tree.
TerminalNode
The terminalNode
interface provides the following methods:
Method  Description 

getName  returns the name of this terminalNode 
getInputRangeCount  returns the input range that this terminalNode covers 
getInputRangeAt  returns the input range at the specified index 
getCandidateCount  returns the number of candidates in this terminalNode candidate list 
getCandidateLabelAt  returns the label of the candidate at the specified index 
getCandidateNormalizedRecognitionScoreAt  returns the normalized recognition score of the candidate at the specified index 
getSelectedCandidateIndex  returns the index of the selected candidate in this terminalNode candidate list 
NonTerminalNode
The nonTerminalNode
interface provides the following methods:
Method  Description 

getName  returns the name of this terminalNode 
getInputRangeCount  returns the input range that this terminalNode covers 
getInputRangeAt  returns the input range at the specified index 
getCandidateCount  returns the number of candidates in this terminalNode candidate list 
getCandidateAt  returns the candidate at the specified index 
getSelectedCandidateIndex  returns the index of the selected candidate in this terminalNode candidate list 
RuleNode
The ruleNode
interface provides the following methods:
Method  Description 

getName  returns the name of this rule node 
getChildCount  returns the number of children this rule node contains 
getChildAt  returns the child at the specified index 
TableNode
A tableNode
object is an imaginary matrix that organizes every element of an operation (target operations being the matrices or the columnar operations). A tableNode
contains cellNode
objects and may contain borderNode
objects. The table below shows examples with a columnar operation and with a matrix.
Columnar operations  Matrices 

In addition to the methods of a standard ruleNode
, the tableNode
interface has the following method:
Method  Description 

getData  returns the data of this tableNode , i.e. number of rows and number of columns 
You can parse the contents of a tableNode
thanks to the getChildAt
method. Children of a tableNode
are either borderNode
or cellNode
objects. They are stored as a list and borderNode
objects, if any, are always on the top of this list.
CellNode
In addition to the methods of a standard nonTerminalNode
, the cellNode
interface has the following method:
Method  Description 

getData  returns the data of this cellNode 
A cellNode
can span one or more rows and columns. The getData
method returns:

rowStart
androwStop
, the row start and the row stop indices, 
columnStart
andcolumnStop
, the column start and column stop indices.
You can parse the contents of a cellNode
thanks to the getChildAt
method.
A cellNode
can contain carries. Carries are recognized and handled through:

overscript
orpresuperscript
rules for additions and multiplications, 
presuperscript
orpresubscript
rules for subtractions and divisions.
cellNode
contains is one carry. In this case, the cell contains a overscript
rule (or a underscript
rule) whose first child is NULL and second child is the overscript, i.e. the carry. See figure below.cellNode
objects (i.e. void cells, with no children) are not conveyed to the resulting tree.BorderNode
In addition to the methods of a standard nonTerminalNode
, the borderNode
interface has the following method:
Method  Description 

getData  returns the data of this borderNode 
A borderNode
can span one ore more rows and columns. The getData
method returns:

type
(horizontal or vertical), 
position
, the index of the row (or column if it is vertical) this border
belongs to, 
start
andstop
, the start and stop column (or row if it is horizontal)
indices that this border spans.
borderNode
is different than the indexing for cellNode
objects. The origin starts from the first axis of the imaginary table, for borderNode
, as depicted below.Using a grammar resource
mathgrmcolumnaroperations.res
resource to handle columnar operations recognition only. For other cases, when addressing standard equations, use the mathgrmstandard.res
grammar resource.The grammar resource indicates the way to segment this handwritten mathematical expression. The recognizer uses it to parse the handwritten mathematical expression properly. The grammar resource file specifies:

a limited set of terminal symbols
Terminal symbols are the elementary symbols. That is, they are basic symbols such as a, b, c, …, 0, 1, …, +, , ±, etc. that cannot be broken down into “smaller” recognized units. 
a limited set of nonterminal symbols
Nonterminal symbols describe groups of terminal symbols organized according torules
. 
a limited set of rules
Rules describe the way to parse digital ink. For instance, a fraction is a rule that contains a numerator, a fraction bar and a denominator. The recognizer expects to find these three elements to fit the fraction rule. 
a start symbol
This defines the general mathematical expression form that you expect to recognize, from the input digital ink.
To be efficient, you need to attach the grammar to the recognizer, in the form of a resource, such as resources/math/mathgrmstandard.res
, resources/math/mathgrmcolumnaroperations.res
or resources/math/mathgrmcalculator.res
. The source of these grammars can be found in the rdk/src/math/
folder, and you can open it and examine its internal structure.
Grammar syntax
Apart from the provided grammar resources, you can use and build your own MyScriptMathcompliant grammar resource. To do so, you must define the grammar in conformity with the syntax set out below and then compile it. You can also find examples of grammar definitions in the rdk/src/math/
directory.
You can add comments in a grammar resource as follows:
// This is the first way to start a whole comment line.
# This is the second way to start a whole comment line.
" This is the third way to start a whole comment line.
/* This is a block comment. */
Inline comments are not supported.
Your grammar resource will contain terminal symbol definitions such as the following:
my_terminal_name = 0 1 2 3 4 5 6 7 8 9

The terminal symbol name is defined as: [azAZ_ ][azAZ_09]*
The terminal symbol name is a character string which starts with a character from [azAZ_] and whose other characters are any of [azAZ_09]. In the example above, the terminal symbol name ismy_terminal_name
. 
The list of symbols referred to by the terminal name are defined as: ( ( !EOL . )+  EOL space+ )+
The definition of the symbols referred to by the terminal name do not start by a EOL (end of line). They can be any existing character separated by a space or EOL and a space. In other words, the definition of the symbols is allowed to span multiple lines, providing that the “continuation line” starts with a space. This means you can format long terminal symbol definitions in a more visually organized (“prettyformatted”) manner. In the example above, the list of symbols referred to bymy_terminal_name
is0 1 2 3 4 5 6 7 8 9
. 
Space is defined as ’ ’ or ‘\t’

EOL is defined as ‘\r\n’ or ‘\n’ or ‘\r’
You will also need to list non terminal symbol definitions. Here is an example:
my_non_terminal_name ::= fraction(my_terminal_name,my_terminal_name)

The nonterminal symbol name is defined as : [azAZ_ ][azAZ_09]* The nonterminal symbol name is a character string which starts with a character from [azAZ_] and whose other characters are any of [azAZ_09]. In the example above, the nonterminal symbol name is
my_non_terminal_name
. 
The nonterminal symbol is defined as:
non_terminal_name ::= rule ( rule)?
In the example above, the rule is fraction
. The example specifies that the numerator is a my_terminal_name
symbol and that the denominator is also a my_terminal_name
symbol.
Rule continuations allow you to “pretty format” rule definitions by avoiding a repetition of target nonterminal symbol names.
Finally, your grammar must include the start symbol definition:
start(my_non_terminal_name)
Finally, you must define the start symbol. In the example above, we specify that the general mathematical expression form that will be recognized is my_non_terminal_name
. In other words, we expect the input digital ink to represent a fraction of digits.
Here is an example of a custom grammar:
Supported rules
Here are the existing rules:
Syntax  Target rule  Rule representation  Written example  LaTex result 

identity(source)  Identity  
hpair(left, right)  Horizontal pair  
fence(source)  Fence  
fraction(numerator, denominator)  Fraction  
sqrt(source)  Square root  
subscript(source, index)  Subscript  
overscript(source, top)  Overscript  
superscript(source, exponent)  Superscript  
subsuperscript(source, index, exponent)  Subsuperscript  
underscript(source, bottom)  Underscript  
underoverscript(source, bottom, top)  Underoverscript  
vpair(top, bottom)  Vertical pair  
presuperscript(source, exponent)  Presuperscript  
presubscript(source, index)  Presubscript  
columnaroperation(digit, carry, decimalSeparator, cancel, operation)  Columnar operation  
overlap(source, cancel)  Crossed out digit  See warning below  
vlist(source)  Vertical list  
table(source)  Matrix 
overlap
is used internally by MyScript Math to label crossed out digits (typically the crossed out carries in columnar operations), so well that you may find it in the resulting tree. Though you may find this rule in the resulting tree, you should not add it in a custom grammar resource since it is implicitly included in the columnaroperation
rule.presubscript
is used internally by MyScript Math to label carries in columnar operations. It is implicitly included in the columnaroperation
rule but you can explicitly add it too in your custom grammar resource to handle standard operations.Compiling your own grammar
A tool has been supplied so that you can compile your own MyScriptMathcompliantgrammar resource.
MkMathGrammar is the command line tool designed to compile a grammar definition. This tool converts the grammar definition into a binary *.res
file format that can be used by MyScript Math SDK. It can be found in the rdk/tools folder in the MyScript root directory, in the bin folder corresponding to your system. If you have MyScript installed, you can use the rdkConsole.bat
that is in the rdk folder.
To convert a grammar definition file into a \*.res
file:

Execute the
rdkConsole.bat
file that is in the rdk folder. 
Give your grammar definition file a
.def
extension.
It must be in conformity with the grammar syntax given above. 
In the console, go to the required directory and use this command line:
% MkMathGrammar mygrammar.def
This will create a mygrammar.res
file in the same location (current directory).
If you want to give your file a different name, use this command line:
% MkMathGrammar o my_nice_grammar.res mygrammar.def
The tool has options that you can set. To see all the options, use MkMathGrammar help
in the command line.
Advanced use
Adding digital symbols
You can avoid performing recognition redundantly by importing digital mathematical symbols. It is possible to use the addCharacter
function, when you want to retrieve an existing result. It is a way of incorporating strings into the result.
Then, you can add extra ink strokes, to achieve a final result. By doing this, you can retrieve an existing result and spare redundant recognition.
The figure briefly shows the application blueprint on this case.
Recognizing edition gestures
The scratchout feature enables to erase several close strokes with a scribbleout gesture, as illustrated on the right.
ScratchOutDetectionSensitivity
controls the scratchout detection sensitivity. You can adjust the scratchout feature by modifying the value of ScratchOutDetectionSensitivity
. This determines how the engine differentiates between real symbols and scratchouts. 0.0f disables detection of scratchouts. 1.0f means maximum supported sensitivity (so it will be more inclined to label strokes as a scratchout, in case of doubt).
It is possible to access the ink ranges of both the scratchout strokes and the erased strokes.
Recognition example
This section provides you with an example of recognition using a custom grammar.
Here is an example of expression:
Now, imagine that you are going to recognize it, using the following grammar:
In the grammar above, symbol
is a terminal symbol and expression
is a nonterminal symbol. identity
, hpair
, fraction
, sqrt
and superscript
are rules. The term expression
is defined as the start symbol of the grammar.
Here is a visual representation of these elements, so you can see what they represent, spatially. Note that this is just a sample set  the real grammars supplied with the Math SDK are more elaborate. This grammar is simpler, for demonstration purposes.
The resulting digital expression is:
Now, let’s examine the parse tree that would be produced by recognizing the mathematical expression with the grammar set out above. This is a visual example to show you the relationships between the various elements.