Handle user input externally

Self-hosted environments: Using the asraas_result field requires version 1.4.0 (or later) of the Dialog service This corresponds to engine pack 2.3 for Speech Suite deployments and engine pack 3.11 for self-hosted Mix deployments.

When an external system or the client application itself is responsible for processing user input, performing NLU interpretation, ASR recognition, or both, or when it collects an entity value via an interaction, the DLGaaS client application needs to send the results of this in one of the following fields in an Execute request payload user_input message:

• For simple interpretations that include entities with string values only, use the interpretation field, including the intent and entities to use for this interaction. The Dialog proceeds using the specified intent.
• For interpretations that include complex entities, use the nluaas_interpretation field. This field expects the interpretation in the format used by the NLUaaS engine. See the NLUaaS InterpretResult documentation for details. Note that DLGaaS supports single intent interpretations only. Once again, the Dialog proceeds using the specified intent.
• For ASRaaS speech recognition results, use the asraas_result field. This field expects recognition results in the format used by the ASRaaS engine. See the ASRaaS Result documentation for details. Dialog will then continue as it would if the ASR result came from an orchestrated call to ASRaaS. The ASR recognition result is then sent to NLUaaS for interpretation.
• For selection of an NLU entity value from an interactive item, use the selected_item field. Dialog proceeds based on the entity value that is filled in.

An Execute request message containing one of these types of input can be send as part of either an Execute() or ExecuteStream() call.

If you are using the TTS channel in your application and want to receive streaming TTS speech audio, you need to send the input using one ExecuteStream() call that includes configuration parameters for TTS. If you do not require TTS, then send instead using an Execute() call.

Simple interpretation example

Here is an example of interpretation performed by an external system passed in as a simple format:

"payload": {
  "user_input": {
    "interpretation": {
      "confidence": 1.0,
      "utterance": "I want a large americano",
      "data": {
        "INTENT": "ORDER_COFFEE",
        "COFFEE_SIZE": "LG",
        "COFFEE_TYPE": "americano"
      },
      "slot_literals": {
        "COFFEE_SIZE": "large",
        "COFFEE_TYPE": "americano"
      }
    }
  }
}

Here:

• utterance is a string of the input typed by the user.
• INTENT in data is the inferred intent of the user input in terms of intents that are part of the Mix NLU model associated with your Dialog model. In this case, the ORDER_COFFEE intent has been detected.
• COFFEE_SIZE and COFFEE_TYPE in data hold detected values for entity slots associated with the intent.

NLUaaS interpretation format example

Here is an example of interpretation performed by an external system passed in using the NLUaaS format:

"payload": {
  "user_input": {
    "nluaas_interpretation": {
      "literal": "i want a double espresso",
      "interpretations": [{
          "single_intent_interpretation": {
            "intent": "ORDER_COFFEE",
            "confidence": 1,
            "origin": "GRAMMAR",
            "entities": {
              "COFFEE_SIZE": {
                "entities": [{
                    "text_range": {
                      "start_index": 9,
                      "end_index": 15
                    },
                    "confidence": 1,
                    "origin": "GRAMMAR",
                    "string_value": "lg"
                  }
                ]
              },
              "COFFEE_TYPE": {
                "entities": [{
                    "text_range": {
                      "start_index": 16,
                      "end_index": 24
                    },
                    "confidence": 1,
                    "origin": "GRAMMAR",
                    "string_value": "espresso"
                  }
                ]
              }
            }
          }
        }
      ]
    }     
  }
}

Problems handling user input

When you encounter a problem collecting or interpreting the user input externally, you can inform Dialog by sending in a Dialog event.