Dragon Voice recognition flow

Protocol messages between the Speech Server, the Natural Language Processing service, and Dragon Voice engines are conveyed over a WebSocket connection. WebSockets provide full-duplex communication channels over a single TCP connection. Text and binary frames are supported. The initial handshake is compatible with HTTP.

First, the engines register their capabilities with the Nuance Resource Manager:

Dragon Voice engine Capabilities registered for allocation by resource manager

Krypton recognition engine

Language, data pack topic, and data pack version.

Nuance Text Processing Engine (NTpE)

 Same as above: language including data pack topic, and data pack version. (This engine is not used or needed for Krypton-only recognition.)

Natural Language Engine (NLE)

Semantic pipeline name and version, as well as IDs of any semantic models loaded. (This engine is not used or needed for Krypton-only recognition.)

This figure shows allocation of recognition and interpretation resources:

  1. Nuance Speech Server connects to the NLP service and sends a CreateSession message to create a new session. The message contains information such as the unique session ID, company name, application name and version, session parameters such as timeouts, and recognition parameters such as the audio format (incoming audio mime-type).
  2. The voice browser reads the VoiceXML request and sends MRCP DEFINE-GRAMMAR messages to Speech Server. The DEFINE-GRAMMAR messages identify the recognition and artifacts to load into the current session. Supported objects:
    • Domain Language Models ("domain LM"): Optional. Customized statistical language models trained from material specific to a given context (an entire application or a certain dialog turn within an application). Used to improve recognition of application-specific content. When not specified, the factory or base language model the Krypton engine loads on startup is used for recognition.
    • Semantic model: Required for interpretation. Conceptual data model that describes meaning.
    • Wordsets: Optional. Lists of words to be added to the vocabulary at runtime, in JSON file format. Used for both recognition and interpretation.

    Tip: For more information on these artifacts, see Triggering the Dragon Voice recognizer.

  3. Once Speech Server receives a successful response (CreateSession complete), it sends a Load request to the NLP service containing information about the data to load.

    This information is conveyed via the URL string received from Speech Server in the form of URL parameters (key-value pairs): specifically, the NLP service uses the value of the nlptype key to determine to which engine—Krypton or NLE—to target the Load request.

  4. NLP service uses the above information to fetch from the application server the associated manifest or configuration file, which must be named nuance_package.json. The location of nuance_package.json is the base URL of the model provided in a Load request. The NLP service, together with the Resource Manager, uses the information contained in this file to allocate Krypton and NLE resources.(See Resource manager features.)
  5. NLP service then sends a request to the Resource Manager to allocate a suitable Krypton or NLE resource.
  6. Resource Manager returns a response identifying the selected engine resource.
  7. NLP service connects to the reserved engine.

    For Krypton:

    1. NLP service sends a createSession request to Krypton based on the previous CreateSession it received from Speech Server, adding to it information about the data pack (language, topic, and version).
    2. NLP service sends Load-in-progress to Speech Server while it waits for the Krypton CreateSession complete response.
    3. When CreateSession complete response is received, NLP service sends a Load request to Krypton for additional objects that may be required such as one or more domain language models and/or wordsets.
    4. NLP service waits for the Krypton Load complete response.
    5. NLP service sends a Load complete response message back to Speech Server.

    For NLE:

    1. NLP service sends a Load request to NLE for the required semantic model.

      Note: NLP service will use the same model ID string for the semantic model object in all Interpret requests it sends to NLE.

    2. NLE fetches, processes, and loads the semantic model file and contacts the Resource Manager to add this semantic model as a capability. When NLE performs tokenization, it uses the NTpE data pack name and version stored in the semantic model to query the Resource Manager for a suitable NTpE instance to perform the tokenization.
    3. NLP service waits for the NLE Load complete response before sending a Load complete response back to Speech Server.

This figure shows the recognition-interpretation-tokenize flow:

  1. Speech Server sends a Recognize request to NLP service.
  2. NLP service sends the Recognize request to Krypton.
  3. NLP service sends the Recognize in-progress response to Speech Server.
  4. Speech Server sends audio to NLP service. NLP service forwards the audio to Krypton.
  5. NLP service waits for the FinalResult response from Krypton.
Notes:
  1. On the FinalResult response from Krypton, NLP service sends an Interpret request to NLE.
  2. NLE receives the Krypton result, selects the top value, and sends it to NTpE to format tokens into text (to serve as the literal for semantic interpretation).
  3. Once the formatting is complete, NTpE returns the result to NLE for further semantic processing.
  4. On the NLE Interpret complete response, NLP service sends a Recognize complete response to Speech Server along with the interpretation result. The final result, therefore, is derived from the combined efforts of Krypton, NLE, and NTpE, and provided in the result format requested.
  5. NLP service returns call log events that are included in Krypton/NLE events (reformatted for Speech Server).
Notes:

The Recognize request includes:

  • For secure context, "sensitive" flag set to either "off" or "suppress".

  • "responseType" to specify the interpretation result format returned by NLE .

  • Changes to the weight of domain language models. See Triggering the Dragon Voice recognizer.

When all recognition-interpretation requests have been processed (final result received), Speech Server ends the session:

  1. NLP service disconnects from NLE. NLE releases its NTpE resource.
  2. NLP service ends the session that exists and is still alive on the receiving Krypton process.
  3. Once NLP service receives the End session complete message from Krypton, NLP service disconnects and sends End session complete to Speech Server.
  4. Speech Server disconnects from NLP service.
  5. Released Krypton and NLE (and NTpE) resources re-register with the Resource Manager with their updated capabilities.