About MyScript Math SDK

MyScript Math SDK provides you with the API based on MyScript Math technology, enabling the recognition of handwritten equations and mathematical formulas.

In brief MyScript Math supplies:

Item Description  Location
API for C, Java, and .NET Math API edk/your_programming_language/
MyScriptMath.dll, MyScript2D.dll The specific libraries engine/
Resource files These files contain specific data needed to perform math recognition. resources/
MathRecognition This code sample shows how to create an math document object, create a math recognizer, attach resources and a grammar to it, and get the parsed result, convert the parse tree to a LaTeX string and to a MathML format. edk/your_programming_language/example/math/

Application blueprint

The figure below describes the MyScript Math SDK functioning.

Recognizer Input (digital ink) Result Resources
Overview of MyScript Math SDK application blueprint

A math recognition with MyScript Math SDK requires that:

  1. you create a math recognizer. The math recognizer will analyze the spatial relationships between the parts of the math expression.

  2. you attach resources to the recognizer. The data laid down in the resource files inform the recognizer on how to perform the segmentation of all the input ink parts.

  3. you send the strokes and start the recognition that your application has gathered from user handwriting with an appropriate device (smartphone, tablet, interactive whiteboard, etc.).

  4. you get back the recognition result in the form of a tree.

The following describes the main objects of MyScript Math environment in details.

Environment

Operations

MyScript Math supports the recognition of two main types of operations (each of which is handled with a specific grammar resource):

  • Standard operations stand for common notations of mathematical expressions.
A list of standard operations
  • Columnar operations refer to long divisions, multiplications in columns, additions in columns and subtractions in columns.
Additions Subtractions Multiplications
Notation of long divisions varies depending on the country. MyScript Math supports Dutch, German, Indian, US and United Kingdom notations.
Divisions
France Germany India The Netherlands United States, United Kingdom

Math recognizer

The MathRecognizer is specialized for math recognition. The recognizer will process the input digital ink as a series of strokes.

Math Alphabet Knowledge

This is a special resource file used for the recognition of mathematical symbols. It provides “math alphabet knowledge” to the recognizer and makes it able to recognize the various elements constituting a mathematical expression. It is often referred to as AK.

Math Grammar Knowledge

The grammar knowledge is a resource file that you attached to the recognizer. It is a set of rules describing how to parse the mathematical expression. MyScript Math provides two types of grammar knowledge resources:

  • math-grm-standard.res, that is appropriate for recognizing standard mathematical expressions.

  • math-grm-columnaroperations.res, that is designed for recognizing columnar operations.

It is possible to create and compile custom grammar resources for the MyScript Math SDK.

Recognition result

Mapped result Math Document Tree
MyScript Math SDK structures the recognition result as a tree

The figure above shows a typical equation that could be recognized with MyScript Math SDK. The result is a tree which can be serialized to a LaTeX or MathML string. This tree is composed of different nodes.

Check section Nodes to learn more on nodes composing the resulting tree.

Check out the code sample provided with MyScript Math SDK to know how to map the result tree to LaTeX or MathML. It is located in the edk/[your_programming_language]/examples/ folder.

Running a recognition

Input (digital ink) Result AK Recognizer Math document 2 cos = a ƒ ¾ b ± attach() Grammar addStroke() process() attach() getParseTree() 1 2 3 4 5
Overview of MyScript Math SDK application blueprint

A math recognition with MyScript Math SDK requires various steps hereby described:

  1. Create a MathRecognizer, once you have set the MyScript environment through the creation of the engine object. You need a recognizer specialized for math recognition that is the MathRecognizer. Use the method createMathRecognizer to do so.

  2. Load and attach the resources (AK and Grammar) to the recognizer. Go to this section to know more on the grammar resource.

  3. Send the strokes that your application has gathered. You send the strokes into a MathDocument object using the addStroke method.

  4. Start the recognition using the method process.

  5. Get back the recognition result in the form of a tree using the method getParseTree. Learn how to parse the tree here.

Check also the Tutorial section or the code sample provided with MyScript Math SDK (in the edk/[your_programming_language]/examples/ folder) to learn in detail how to implement the whole Math recognition process.

Parsing the result

The recognizer outputs the recognition result in the form of a tree. You obtain this tree, after the recognition process, through the method getParseTree. You can then explore the tree by using the methods from the different node types’ interfaces. Thus, to explore the result you need to:

  1. Check the type of the node. Check whether the node is aterminalNode, a nonTerminalNode or a ruleNode with the voGetType method.

  2. Use the appropriate API to explore the components of a node. For example, if the node is a ruleNode, you use the methods getChildCount, getChildAt (among others) to get its components. This can return a node of any of the existing types. Check section Nodes to learn more on nodes’ APIs.

  3. Parse these nodes recursively .This means that you will need to reiterate from step 1 until you reach the end of the tree.

You can convert the tree into a more usable form. We provide a code sample to perform the conversion into a LaTeX or MathML format. Check in the folder edk/[language_name]/src.

The structure of the tree is related to the relationship between the recognized symbols that compose the input ink. A tree is made of different nodes. A node describes a segment of the recognized mathematical expression. Inner segments of a single node form new nodes that are attached to this parent node. There are six types of nodes:

node ruleNode terminalNode nonTerminalNode cellNode tableNode borderNode
Types of nodes
  • terminalNode. We call terminalNode the elementary component of a mathematical expression. For instance, it can be a digit.

  • nonTerminalNode. We call a nonTerminalNode any group of symbols that is not elementary. Think of it as atoms and molecules. A molecule is made of several atoms like a nonTerminalNode is made of several terminalNode and ruleNode objects. For instance, the number 42 is made of two terminalNodes: 4 and 2.

    • cellNode. A cellNode is a particular nonTerminalNode. It is used when dealing with columnar operations or matrices.

    • borderNode. A borderNode is a particular nonTerminalNode. It is used when dealing with columnar operations. A borderNode models any type of border line (horizontal lines in columnar additions or vertical lines in columnar divisions, and so on.)

  • ruleNode. A ruleNode describes the spatial relationship between groups of nonTerminalNodes and terminalNodes.

    • tableNode. A tableNode is a particular ruleNode. It is used when dealing with columnar operations or matrices. The tableNode models the spatial relationship between every element of a columnar operation or in a matrix.

We refer to the whole mathematical expression as the root node.

The figure below illustrates the concept of nodes in a concrete example.

root node nonTerminalNode terminalNode ruleNode Legend
Example of tree structure

Nodes

This section details the node objects that you can find in the resulting tree.

TerminalNode

The terminalNode interface provides the following methods:

Method Description 
getName returns the name of this terminalNode
getInputRangeCount returns the input range that this terminalNode covers
getInputRangeAt returns the input range at the specified index 
getCandidateCount returns the number of candidates in this terminalNode candidate list
getCandidateLabelAt returns the label of the candidate at the specified index
getCandidateNormalizedRecognitionScoreAt returns the normalized recognition score of the candidate at the specified index
getSelectedCandidateIndex returns the index of the selected candidate in this terminalNode candidate list

NonTerminalNode

The nonTerminalNode interface provides the following methods:

Method Description 
getName returns the name of this terminalNode
getInputRangeCount returns the input range that this terminalNode covers
getInputRangeAt returns the input range at the specified index 
getCandidateCount returns the number of candidates in this terminalNode candidate list
getCandidateAt returns the candidate at the specified index
getSelectedCandidateIndex returns the index of the selected candidate in this terminalNode candidate list

RuleNode

The ruleNode interface provides the following methods:

Method Description 
getName returns the name of this rule node
getChildCount returns the number of children this rule node contains 
getChildAt returns the child at the specified index 

TableNode

A tableNode object is an imaginary matrix that organizes every element of an operation (target operations being the matrices or the columnar operations). A tableNode contains cellNode objects and may contain borderNode objects. The table below shows examples with a columnar operation and with a matrix.

Columnar operations Matrices
tableNode cellNode MyScript Mathprocessing: borderNode tableNode cellNode MyScript Mathprocessing:

In addition to the methods of a standard ruleNode, the tableNode interface has the following method:

Method Description 
getData returns the data of this tableNode, i.e. number of rows and number of columns

You can parse the contents of a tableNode thanks to the getChildAt method. Children of a tableNode are either borderNode or cellNode objects. They are stored as a list and borderNode objects, if any, are always on the top of this list.

CellNode

In addition to the methods of a standard nonTerminalNode, the cellNode interface has the following method:

Method Description 
getData returns the data of this cellNode

A cellNode can span one or more rows and columns. The getData method returns:

  • rowStart and rowStop, the row start and the row stop indices,

  • columnStart and columnStop, the column start and column stop indices.

cellNode 0 1 2 3 4 0 1 2 3 column indices row indices columnStop: columnStart: rowStop: rowStart: 0 3 0 3 cellData
CellNode and its data

You can parse the contents of a cellNode thanks to the getChildAt method.

A cellNode can contain carries. Carries are recognized and handled through:

  • overscript or presuperscript rules for additions and multiplications,

  • presuperscript or presubscript rules for subtractions and divisions.

overscript presuperscript presubscript
Carries
It is possible that the only element that a cellNode contains is one carry. In this case, the cell contains a overscript rule (or a underscript rule) whose first child is NULL and second child is the overscript, i.e. the carry. See figure below.
0 1 2 3 4 0 1 2 3 column indices row indices NULL cellNode with one carry only:
CellNode with
one carry
Null cellNode objects (i.e. void cells, with no children) are not conveyed to the resulting tree.

BorderNode

In addition to the methods of a standard nonTerminalNode, the borderNode interface has the following method:

Method Description 
getData returns the data of this borderNode

A borderNode can span one ore more rows and columns. The getData method returns:

  • type (horizontal or vertical),

  • position, the index of the row (or column if it is vertical) this border
    belongs to,

  • start and stop, the start and stop column (or row if it is horizontal)
    indices that this border spans.

Note that the indexing for borderNode is different than the indexing for cellNode objects. The origin starts from the first axis of the imaginary table, for borderNode, as depicted below.
borderNode 0 1 2 3 4 0 1 2 3 column indices row indices stop: start: position: type: horizontal 0 3 4 borderData 5 4
BorderNode and its data

Using a grammar resource

Columnar operation recognition is enabled through the math-grm-columnaroperations.res resource to handle columnar operations recognition only. For other cases, when addressing standard equations, use the math-grm-standard.res grammar resource.

The grammar resource indicates the way to segment this handwritten mathematical expression. The recognizer uses it to parse the handwritten mathematical expression properly. The grammar resource file specifies:

  • a limited set of terminal symbols
    Terminal symbols are the elementary symbols. That is, they are basic symbols such as a, b, c, …, 0, 1, …, +, -, ±, etc. that cannot be broken down into “smaller” recognized units.

  • a limited set of non-terminal symbols
    Non-terminal symbols describe groups of terminal symbols organized according to rules.

  • a limited set of rules
    Rules describe the way to parse digital ink. For instance, a fraction is a rule that contains a numerator, a fraction bar and a denominator. The recognizer expects to find these three elements to fit the fraction rule.

  • a start symbol
    This defines the general mathematical expression form that you expect to recognize, from the input digital ink.

To be efficient, you need to attach the grammar to the recognizer, in the form of a resource, such as resources/math/math-grm-standard.res, resources/math/math-grm-columnaroperations.res or resources/math/math-grm-calculator.res. The source of these grammars can be found in the rdk/src/math/ folder, and you can open it and examine its internal structure.

Grammar syntax

Apart from the provided grammar resources, you can use and build your own MyScript-Math-compliant grammar resource. To do so, you must define the grammar in conformity with the syntax set out below and then compile it. You can also find examples of grammar definitions in the rdk/src/math/ directory.

You can add comments in a grammar resource as follows:

  // This is the first way to start a whole comment line.
  #  This is the second way to start a whole comment line.
  "  This is the third way to start a whole comment line.
  /* This is a block comment. */

Inline comments are not supported.

Your grammar resource will contain terminal symbol definitions such as the following:

  my_terminal_name = 0 1 2 3 4 5 6 7 8 9
  • The terminal symbol name is defined as: [-a-zA-Z_ ][-a-zA-Z_0-9]*
    The terminal symbol name is a character string which starts with a character from [-a-zA-Z_] and whose other characters are any of [-a-zA-Z_0-9]. In the example above, the terminal symbol name is my_terminal_name.

  • The list of symbols referred to by the terminal name are defined as: ( ( !EOL . )+ | EOL space+ )+
    The definition of the symbols referred to by the terminal name do not start by a EOL (end of line). They can be any existing character separated by a space or EOL and a space. In other words, the definition of the symbols is allowed to span multiple lines, providing that the “continuation line” starts with a space. This means you can format long terminal symbol definitions in a more visually organized (“pretty-formatted”) manner. In the example above, the list of symbols referred to by my_terminal_name is 0 1 2 3 4 5 6 7 8 9.

  • Space is defined as ’ ’ or ‘\t’

  • EOL is defined as ‘\r\n’ or ‘\n’ or ‘\r’

You will also need to list non terminal symbol definitions. Here is an example:

  my_non_terminal_name ::= fraction(my_terminal_name,my_terminal_name)
  • The non-terminal symbol name is defined as : [-a-zA-Z_ ][-a-zA-Z_0-9]* The non-terminal symbol name is a character string which starts with a character from [-a-zA-Z_] and whose other characters are any of [-a-zA-Z_0-9]. In the example above, the non-terminal symbol name is my_non_terminal_name.

  • The non-terminal symbol is defined as: non_terminal_name ::= rule (| rule)?

In the example above, the rule is fraction. The example specifies that the numerator is a my_terminal_name symbol and that the denominator is also a my_terminal_name symbol.

Putting this example aside, note that MyScript Math also offers the possibility to recognize “partial” fraction, i.e. with one denominator only or with one numerator only. This feature is available by default, so that you do not need to add a non-terminal entry to recognize “partial” fractions.

Rule continuations allow you to “pretty format” rule definitions by avoiding a repetition of target non-terminal symbol names.

Finally, your grammar must include the start symbol definition:

  start(my_non_terminal_name)

Finally, you must define the start symbol. In the example above, we specify that the general mathematical expression form that will be recognized is my_non_terminal_name. In other words, we expect the input digital ink to represent a fraction of digits.

Here is an example of a custom grammar:

symbol expression start(expression) = ::= 0 1 2 3 4 5 6 7 8 9 a b c d e f g h i j k l mn o p q r s t u v w x y z A B C D E F G H I JK L M N O P Q R S T U V W X Y Z + - = / × ÷ ± √ identity (symbol) | hpair (expression, expression)| fraction (expression, expression)| sqrt (expression)| superscript (symbol, expression) Terminal symbol Non-terminal symbol Rule Start symbol

Supported rules

Here are the existing rules:

Syntax Target rule Rule representation Written example LaTex result
identity(source) Identity      
hpair(left, right) Horizontal pair
fence(source) Fence { } [ ] ( )
fraction(numerator, denominator) Fraction
sqrt(source) Square root
subscript(source, index) Subscript
overscript(source, top) Overscript
superscript(source, exponent) Superscript
subsuperscript(source, index, exponent) Subsuperscript
underscript(source, bottom) Underscript
underoverscript(source, bottom, top) Underoverscript
vpair(top, bottom) Vertical pair 57
presuperscript(source, exponent) Presuperscript
presubscript(source, index) Presubscript
columnaroperation(digit, carry, decimalSeparator, cancel, operation) Columnar operation
overlap(source, cancel) Crossed out digit   See warning below  
vlist(source) Vertical list
table(source) Matrix
The rule overlap is used internally by MyScript Math to label crossed out digits (typically the crossed out carries in columnar operations), so well that you may find it in the resulting tree. Though you may find this rule in the resulting tree, you should not add it in a custom grammar resource since it is implicitly included in the columnaroperation rule.
The rule presubscript is used internally by MyScript Math to label carries in columnar operations. It is implicitly included in the columnaroperation rule but you can explicitly add it too in your custom grammar resource to handle standard operations.

Compiling your own grammar

A tool has been supplied so that you can compile your own MyScript-Math-compliant-grammar resource.

MkMathGrammar is the command line tool designed to compile a grammar definition. This tool converts the grammar definition into a binary *.res file format that can be used by MyScript Math SDK. It can be found in the rdk/tools folder in the MyScript root directory, in the bin folder corresponding to your system. If you have MyScript installed, you can use the rdkConsole.bat that is in the rdk folder.

To convert a grammar definition file into a \*.res file:

  1. Execute the rdkConsole.bat file that is in the rdk folder.

  2. Give your grammar definition file a .def extension.
    It must be in conformity with the grammar syntax given above.

  3. In the console, go to the required directory and use this command line:

    % MkMathGrammar mygrammar.def

This will create a mygrammar.res file in the same location (current directory).

If you want to give your file a different name, use this command line:

  % MkMathGrammar -o my_nice_grammar.res mygrammar.def

The tool has options that you can set. To see all the options, use MkMathGrammar --help in the command line.

Advanced use

Adding digital symbols

You can avoid performing recognition redundantly by importing digital mathematical symbols. It is possible to use the addCharacter function, when you want to retrieve an existing result. It is a way of incorporating strings into the result.

Then, you can add extra ink strokes, to achieve a final result. By doing this, you can retrieve an existing result and spare redundant recognition.

The figure briefly shows the application blueprint on this case.

Input (digital ink) Result AK Recognizer Math document 2 cos = a ƒ ¾ b ± attach() Grammar addStroke() process() attach() getParseTree() addCharacter()

Recognizing edition gestures

Original ink Recognition result
MyScript Math enables scratching out

The scratch-out feature enables to erase several close strokes with a scribble-out gesture, as illustrated on the right.

ScratchOutDetectionSensitivity controls the scratch-out detection sensitivity. You can adjust the scratch-out feature by modifying the value of ScratchOutDetectionSensitivity. This determines how the engine differentiates between real symbols and scratch-outs. 0.0f disables detection of scratch-outs. 1.0f means maximum supported sensitivity (so it will be more inclined to label strokes as a scratch-out, in case of doubt).

It is possible to access the ink ranges of both the scratch-out strokes and the erased strokes.

Recognition example

This section provides you with an example of recognition using a custom grammar.

Here is an example of expression:

Now, imagine that you are going to recognize it, using the following grammar:

symbol expression start(expression) = ::= 0 1 2 3 4 5 6 7 8 9 a b c d e f g h i j k l mn o p q r s t u v w x y z A B C D E F G H I JK L M N O P Q R S T U V W X Y Z + - = / × ÷ ± √ identity (symbol) | hpair (expression, expression)| fraction (expression, expression)| sqrt (expression)| superscript (symbol, expression) Terminal symbol Non-terminal symbol Rule Start symbol

In the grammar above, symbol is a terminal symbol and expression is a non-terminal symbol. identity, hpair, fraction, sqrt and superscript are rules. The term expression is defined as the start symbol of the grammar.

Here is a visual representation of these elements, so you can see what they represent, spatially. Note that this is just a sample set - the real grammars supplied with the Math SDK are more elaborate. This grammar is simpler, for demonstration purposes.

hpair fraction sqrt superscript identity
Visual representation of rules

The resulting digital expression is:

Resulting expression

Now, let’s examine the parse tree that would be produced by recognizing the mathematical expression with the grammar set out above. This is a visual example to show you the relationships between the various elements.

Resulting tree