Using rulesets

This topic describes how to use rulesets to tune or normalize text input.

A ruleset is a set of match-and-replace rules that the engine uses to change strings of input text when converting to TTS. For example, a ruleset might expand an abbreviation (from “PIN” to “personal information number”) or to find all uses of a currency symbol, and replace it with words ("dollars", "euros" and so on) regardless of the amounts.

The match specification finds a string in the input.
The replacement specification substitutes a different string.

Whereas user dictionaries only support search and replace for literal strings—that is, complete words or tagged multi-word fragments—rulesets support any search pattern that can be expressed using regular expressions. You can use them to search for multiple words, part of a word, or a repeated pattern.

Loading

You can load any number of rulesets for runtime. The mechanism you use determines whether the ruleset is global or has a narrow scope. The load order determines the processing order: first loaded rulesets get processed before last loaded rulesets.

Mechanisms:

Load with a the <default_rulesets> XML configuration parameter in Management Station (global, applies to the entire input text).
Load with the SSML <lexicon> element in the input text (typed, only effects the input text in the same scope as the element). Use this techinque to narrow the scope of text replacements to specific text fragments. These are called typed rulesets because their matching text fragments are labelled as a particular data type by the SSML <say-as> or the native <ESC>\tn\ control sequence. The typed rulesets augment or override the Vocalizer built-in text normalization types.

Precedence and processing

Vocalizer applies a ruleset when the active language matches the language in the ruleset header. To perform the substitutions, it uses the Regular Expression Text-To-Text (RETTT) engine, a subcomponent of each text-to-text engine instance.

Text normalization sequence:

If you provide the input text via the input text callback mechanism, Vocalizer collects the entire input.
Vocalizer transcodes the input to UTF-16.
Vocalizer expands the SSML markup to native control sequences, applies rulesets in order, and then applies any other text normalizations (for example, before applying orthographic user dictionary entries).

For typed rulesets, the SSML <say-as> or <ESC>\tn\ type gets stripped by the engine while processing the typed ruleset (even if the typed ruleset does not make changes to the contained text fragment). This allows typed rulesets to override Vocalizer built-in text normalization types. Optionally, you can delegate portions of your typed ruleset processing to Vocalizer built-in text normalization types by adding <ESC>\tn\ wrappers around the text. (You must use the native control sequence because SSML processing is already done before ruleset processing.).

The engine processes rulesets as follows:

The engine processes rulesets in the order they were loaded and completes the typed rulesets before starting the global rulesets. This strategy allows you to define a sequential chain of rulesets that progressively refine and normalize the input text.
In each ruleset, rules are processed in document order.
Regardless of the effect of a rule, the engine applies the next rule, and so on. A later rule may change an input string that has already been transformed by a previous rule.
The engine continues processing rulesets from the first- to last-loaded. It stops after the last rule of the most recently-loaded ruleset.

Ruleset format

In general, a user-defined is a UTF-8 text file with a header followed by data. Use the pound (#) symbol for comments.

Short example ruleset:

[header]

language = ENU

charset = "utf-8"

type = financial:stocks

# The language is American English

[data]

/NUAN/ --> "Nuance"

/\x{20ac}(\d+)\.(\d{2})\d*/ --> "$1 euro $2 cents"

Data section

The data contains one or more rules, with one rule per line:

[data]

/NUAN/ --> "Nuance"

/\x{20ac}(\d+)\.(\d{2})\d*/ --> "$1 euro $2 cents"

A rule has the following syntax:

 search-specification "-->" replacement-specification

Comments can appear on separate lines, or be inserted at the end of a rule. A comment starts with a '#' character and span until the end of the line.

The search-specification and the replacement-specification must use syntax and semantics that comply with the PCRE v5.0 regular expression library—that is, they must work as Perl 5 expressions. These can be individual strings, or more complex Perl 5 regular expressions.

For the Perl 5 regular expression syntax, please refer to Perl regular expressions main page.
For a description of PCRE, a free regular expression library, see http://www.pcre.org/. For a more detailed description, see the pcrepattern.html document in the PCRE distribution package.

If markup is used in the search and/or replacement pattern, it must be in the native Vocalizer markup format.

Note that special characters and characters with a special meaning need to be escaped:

In the search pattern: non-alphanumerical characters with a specificationial meaning like dot (.), asterisk (*), dollar ($), backslash (\), and so on, need to be preceded with a backslash when used literally in a context where they can have a special meaning (for example, use \* for *). In the replacement specification this applies to characters like dollar ($), backslash (\) and double quote (").
Control characters like \t (tab), \n (newline), \r (return), and so on.
Character codes: \xhh (hh is the hexadecimal character code, for example, \x1b for Escape), \ooo (ooo is the octal notation, for example, \033 for Escape).
Perl5 also predefines some patterns like "\s" (white space) and "\d" (numeric). (For a full description, please refer to the Perl5 man pages.)

Search specification

In general the format of the search specification is:

 delimiter regular-expression delimiter modifiers

The delimiter is usually a forward slash ("/"), but can be any non-whitespace character except digits, backslash ("\"), or a hashtag ("#"). This facilitates the specification of a regular expression that contains a forward slash "/", because it eliminates the need to escape the "/".
The regular-expression is an expression that conforms to Perl5 syntax.

The modifiers are options that can include any or all of the following:

Modifier	Description
i	Search is case-insensitive.
m	Let "^" and "$" match even at embedded newline characters.
s	Let the "." pattern match even at embedded newline characters. By default "." matches any arbitrary character, except for a newline.
x	Allow for regular expression extensions like inserting white space and comments in <regular-expression>.

Some rule examples

Here are examples of rules and the resulting replacements:

Rule	Original text	Replacement text
/<NUAN>/ --> "Nuance Communications"	<NUAN>	Nuance Communications
/(Quack)/ --> ($1)	Quack	(Quack)
/(Quack)/ --> ($2)	Quack	()
/(\s):-\)(\s)/ --> "$1ha ha$2"	Where "\s" matches any white space character, $1 corresponds with the matched leading white space character and $2 corresponds with the matched trailing white space character. For example, this rule rewrites " :-) " into " ha ha ".
/(\r?\n)-{3,} Begin included message -{3,}(\r?\n)/ --> "$1Start of included message:$2"	---- Begin included message ---- (For example)	Start of included message:
/\x80 ?(\d+)\.(\d{2})\d*/ --> "$1 euro $2 cents"	€9.751 (For example)	9 euro 75 cents

Effect of rulesets on location markers

Markers on input text have source position fields that represent positions after the ruleset transformations (and not the original text positions).

Effect of rulesets on performance

The loading of rulesets can affect synthesis processing performance, increasing latency (time to first audio) and overall CPU use. It is important to test the system to see how the rulesets affect its performance, and to ensure that it is acceptable.

This makes it important to consider pattern efficiency carefully while writing rulesets. Certain regular expression patterns are more efficient than others. For example, a character class (such as "[aeiou]") is more efficient than the equivalent set of alternatives (such as "(a|e|i|o|u)").