NLEtoknd—Tokenization end
Logged by the Nuance Text Processing Engine at the end of a Tokenize command. Indicates the end of the tokenization of input to the Natural Language Engine Interpret command. Contains the tokenizer output; that is, the minimally formatted and tokenized forms of each input.
There is one such event per Interpret command.
In addition to the Tokens used for every event, this event has the following tokens:
| Token | Meaning |
|---|---|
| MFF(x) |
The minimally formatted form (MFF) of the x-th input text. Indices are 0-based. For text input, there will only ever be one MFF. For audio input, there is one MFF per corresponding n-best entry in the Krypton result (KRrslt). |
| TOKT(x)(y) |
The tokenized form of the y-th token of the MFF of the x-th input text. Indices are 0-based. For text input, there will only ever be one tokenized form. For audio input, there is one tokenized form per corresponding n-best entry in the Krypton result. For each tokenized form per n-best entry, each one is an array of tokens, effectively making it a matrix (x is the n-best index and y is the token index). |
| TOKTIME | Clock-time in milliseconds to perform the tokenization. |
Example:
EVNT=NLEtoknd|MFF(0)=pay thirty dollars to my visa from checking on February first two thousand eighteen|SCPU=0|TOKT(0)(0)=pay|TOKT(0)(1)=thirty|TOKT(0)(10)=first|TOKT(0)(11)=two|TOKT(0)(12)=thousand|TOKT(0)(13)=eighteen|TOKT(0)(2)=dollars|TOKT(0)(3)=to|TOKT(0)(4)=my|TOKT(0)(5)=visa|TOKT(0)(6)=from|TOKT(0)(7)=checking|TOKT(0)(8)=on|TOKT(0)(9)=February|
