Using confidence scores
The confidence score assigned to each recognition (the value assigned to the lastresult$[i].confidence shadow variable) indicates how closely the caller speech matches the recognition result. The higher the score, the more likely it is that the recognition result matches what the caller said. When the score is low (close to zero), there is a significant chance that the recognition result is incorrect.
You can use the confidence score to guide the behavior of your application. For example, you can compare the confidence score against high and low thresholds to determine whether to accept the recognition result, to reject it, or to ask the caller to confirm it.
The confidencelevel property
The confidencelevel parameter represents the minimum confidence score that will be acceptable for a recognition result. The recognizer automatically rejects any recognition whose confidence score is below this value.
You can set the confidencelevel parameter in one of the following places:
- In a parameter grammar
- In a VoiceXML file
- In a <meta> element in a GrXML grammar
- in the “default” language section of a user configuration file
The Voice Platform recognizer accepts confidencelevel values from 0 to 1000. If you set the parameter in a parameter grammar, grammar <meta> element, or in a user configuration file, you must use this range.
However, when you set the confidencelevel in a VoiceXML file, you must use a range of 0.0 to 1.0 to conform to the VoiceXML specification. Voice Platform scales this value to a corresponding value from 0 to 1000 before sending it to the recognizer.
Note that the confidence score that is returned with a recognition is converted the 0.0 to 1.0 range, even when the applicable confidencelevel parameter used the 0 to 1000 range.
Setting the confidencelevel in a parameter grammar
You can also use confidencelevel within a parameter grammar:
<?xml version='1.0' encoding='ISO-8859-1'?>
<SWIparameter version="1.0" id="myParams1" precedence="6" >
<parameter name="confidencelevel">
<value>600</value>
</parameter>
...
</SWIparameter>
The possible values range from 0 to 1000, just as in a GrXML grammar <meta> element.
Since a parameter grammar takes precedence over VoiceXML, a confidencelevel specified in a parameter grammar will override a confidencelevel specified in VoiceXML, GrXML, or a user configuration file. For information on parameter grammars, see the Speech Suite documentation.
Setting the confidencelevel in VoiceXML
The confidencelevel is subject to the rules of precedence in VoiceXML. For example, the pizza.vxml root document of the PizzaTalk application sets a default confidencelevel of 0.5:
<property name="confidencelevel" value=".50"/>
However, you could override this within a form by supplying a different value to be used within that form only. For example, suppose you want to be very certain to avoid a false acceptance when recognizing a caller’s unique password. You could set the confidencelevel property within that field only:
<form id="security">
<property name="confidencelevel" value="0.75"/>
<field id="password" >
<prompt>What is your unique password?</prompt>
<grammar src="../grammars/passwords.grxml">
</field>
...
</form>
A new default confidencelevel of 0.75 would override the previous default value of 0.5 for all fields within the security form. After exiting the form, the default confidencelevel would revert to the previous value of 0.5.
Similarly, if you set a different confidencelevel within one of the security form’s fields, it would take precedence within that field:
<form id="security">
<property name="confidencelevel" value="0.75"/>
...
<field id="name" >
<property name="confidencelevel" value="0.6"/>
<prompt>For the record, please state your name.</prompt>
<grammar src="../grammars/names.grxml">
</field>
...
</form>
Here the value of 0.75 is used by default for all fields within the security form except for the name field, which uses a confidencelevel of 0.6 instead.
Setting the confidencelevel in a grammar
The confidencelevel can also be used within a grammar by using the <meta> element in GrXML. When you use this method, the range is from 0 to 1000:
<?xml version="1.0" encoding="ISO-8859-1" ?>
<grammar xmlns="http://www.w3.org/2001/06/grammar"
version="1.0" xml:lang="en-US" root="YesNo"
tag-format="swi-semantics/1.0">
<meta name="confidencelevel" content="600"/>
<rule id="YesNo" scope="public">
...
</grammar>
In this example, the confidencelevel of 600 applies whenever the grammar is used.
However, if a confidencelevel is specified in the VoiceXML file, this will override the value specified in the grammar. Similarly, the value may be ignored if a parameter grammar applies, or if the grammar is imported by a different grammar.
Setting the confidencelevel in a user configuration file
You can set the confidencelevel in a user configuration file as follows:
<?xml version="1.0"?>
<SWIrecConfig version="1.0.0">
<lang name="default">
<param name="confidencelevel">
<value>600</value>
</param>
...
</lang>
</SWIrecConfig>
Using confidence thresholds to guide confirmation
You can compare the confidence score for a given recognition result against high and low thresholds, and use this comparison to decide whether to confirm the caller’s answer:
...
<property name="confidencelevel" value=".50"/>
...
<form id="backgroundinfo">
<field name="birthdate" type="date">
<prompt>What is your birthday?</prompt>
<filled>
<if cond="lastresult$.confidence <= 0.75">
<prompt>I heard <value expr="birthdate"/>.
Did I get that right?</prompt>
<goto next="Confirmation.vxml"/>
<else/>
<goto next="#nextField"/>
</if>
</filled>
</field>
In this example, the application automatically rejects any recognition result that receives a confidence score below 0.5, because confidencelevel of 0.5 is set in the <property> element in the header. If the result is accepted, however, the application makes a further comparison against the threshold of 0.75 defined in the <if> condition.
- If the result has a confidence score from 0.5 to 0.75, the application plays back the result and asks for a confirmation. In this example, we assume that the confirmation is handled in the Confirmation.vxml document (<goto next=”Confirmation.vxml”/>).
- If the result has a confidence score above 0.75, the application immediately transitions to the nextField form in the current document (<goto next=”#nextField”/>).
This strategy saves time, as the application automatically rejects recognition results that are clearly wrong, accepts results that are clearly correct, and only asks for a confirmation when the accuracy of the result falls between these two extremes.
Of course false acceptances are always possible, even when the automatic acceptance threshold is high. It is therefore recommended that you ask for explicit confirmation before actually using caller input. However, judicious use of confidence thresholds lets you confirm or change individual field items immediately when necessary, but still confirm all the caller’s responses at the end of a form.
