NLEtokst—Tokenization start
Logged by the Nuance Text Processing Engine at the start of a Tokenize command. Indicates the beginning of the tokenization of the input to the Natural Language Engine Interpret command. Includes the input to the Tokenize command, the URI to the application linguistic model (ALM) to use to perform the tokenization, and the versions of MREC and TextProc used.
The input to the Tokenize command is an array. For text input, a single text string is specified. For audio input, multiple text strings are specified, one per n-best entry.
There is one such event per Interpret command.
In addition to the Tokens used for every event, this event has the following tokens:
| Token | Meaning |
|---|---|
| ALM |
URI of the application linguistic model (ALM) specified in the Tokenize command. Note: URI may be a local file ( |
| INPUT(x) |
Textual representation of each input to the Tokenize command. Includes encrypted literals from MREC tokens (only the literals in the written\sense\spoken form) for audio input and text for text input. |
| LANG | ALM language code used when creating the session; for example, "eng-USA". |
| MRECVER | MREC version. |
| TPVER | TextProc version. |
Example:
EVNT=NLEtokst|ALM=https://mt-myhost-c01:9091/media/models/74aadd65-f690-47aa-8441-ee67cd0c123a/alm.zip| INPUT(0)=pay thirty dollars to my visa from checking on February first two thousand eighteen| LANG=eng-USA|MRECVER=1.39.100.18359|TPVER=15.04.100.01668
