Synthesizer gRPC API for Neural TTSaaS

The Synthesizer gRPC API contains methods for requesting speech synthesis from Neural TTSaaS, using Microsoft neural voices.

Tip:

Try out this API using the Sample synthesis client for Neural TTSaaS that you can download and run.

Proto file structure

The Synthesizer API is defined in the synthesizer.proto file.

└── nuance
    └── tts
        └── v1 
            └── synthesizer.proto

Neural TTSaaS does not support all fields in synthesizer.proto. See Supported fields and defaults.

For Neural TTSaaS, the proto file defines a Synthesizer service with two RPC methods: GetVoices and Synthesize.

Sequence flow

The essential tasks are illustrated in the following high-level sequence flow of a client application at runtime:

  Runtime sequence flow  

Synthesizer

The Synthesizer service offers these functionalities:

• GetVoices: Queries the list of available voices, with filters to reduce the search space.
• Synthesize: Synthesizes audio from input text and parameters, and returns an audio stream.

GetVoicesRequest

Input message for the GetVoices method, to query voices available to the client. For more examples, see Sample synthesis client for Neural TTSaaS: Get voices and Voice filters.

For information on the Microsoft voices returned by GetVoices, see the Microsoft documentation: Language and voice support for the Speech service  .

The GetVoicesRequest message includes:

GetVoicesRequest
  voice (Voice)
    name
    language
    gender (EnumGender)
    sample_rate_hz

For example, this requests information about all female American English voices:

GetVoicesRequest (
    voice = Voice (
        language = "en-US",
        gender = EnumGender.FEMALE
    )
)

This asks about one named voice:

GetVoicesRequest (
    voice = Voice (
        name = "en-US-JennyNeural"
    )
)

Voice

Input or output message for voices. Different fields are supported depending on the method.

• In SynthesisRequest:

  For plain text, it specifies the voice to use with the mandatory name field.

  For SSML, it optionally specifies the voice to use with the name field. The voice may instead be set with <voice> in the SSML input. See SSML input.

• In GetVoicesRequest, it filters the list of available voices, with optional fields name, language, gender, foreign_languages, styles, and sample_rate_hz. See Voice filters for more examples.

• In GetVoicesResponse, it returns the list of available voices, with name, model, language, gender, sample_rate_hz. It includes foreign_languages and/or styles when available for the voice.

The Voice message includes different fields depending on the context:

GetVoicesRequest
  voice (Voice)
    name
    language
    gender (EnumGender)
    sample_rate_hz
 
GetVoicesResponse
  voice (Voice)
    name
    model
    language
    gender (EnumGender)
    sample_rate_hz
    foreign_languages
    styles
 
SynthesisRequest
  voice (Voice)
    name

EnumGender

Input field for GetVoicesRequest or output field for GetVoicesResponse, specifying gender for voices that support multiple genders. Included in Voice.

GetVoicesResponse

Output message in response to GetVoicesRequest. Contains information about the voices that match the input criteria, if any, and includes foreign languages and styles for voices that support them.

To use the styles in a synthesis request, see Input to synthesize: SSML elements: Voice style.

For example, this is the response to GetVoices for American English voices. Notice that voice styles are included for voices that support them, and foreign languages are listed for the Jenny multilingual voice.

2022-10-24 15:56:27,265 (140266945111872) DEBUG [voice {
    language: "en-US"
}
]
2022-10-24 15:56:27,265 (140266945111872) INFO  Sending GetVoices request
2022-10-24 15:56:27,405 (140266945111872) INFO  voices {
    name: "en-US-JennyNeural"
    model: "neural"
    language: "en-US"
    gender: FEMALE
    sample_rate_hz: 24000
    styles: "assistant"
    styles: "chat"
    styles: "customerservice"
    styles: "newscast"
    styles: "angry"
    styles: "cheerful"
    styles: "sad"
    styles: "excited"
    styles: "friendly"
    styles: "terrified"
    styles: "shouting"
    styles: "unfriendly"
    styles: "whispering"
    styles: "hopeful"
}
voices {
    name: "en-US-JennyMultilingualNeural"
    model: "neural"
    language: "en-US"
    gender: FEMALE
    sample_rate_hz: 24000
    foreign_languages: "de-DE"
    foreign_languages: "en-AU"
    foreign_languages: "en-CA"
    foreign_languages: "en-GB"
    foreign_languages: "es-ES"
    foreign_languages: "es-MX"
    foreign_languages: "fr-CA"
    foreign_languages: "fr-FR"
    foreign_languages: "it-IT"
    foreign_languages: "ja-JP"
    foreign_languages: "ko-KR"
    foreign_languages: "pt-BR"
    foreign_languages: "zh-CN"
}
voices {
    name: "en-US-GuyNeural"
    model: "neural"
    language: "en-US"
    gender: MALE
    sample_rate_hz: 24000
    styles: "newscast"
    styles: "angry"
    styles: "cheerful"
    styles: "sad"
    styles: "excited"
    styles: "friendly"
    styles: "terrified"
    styles: "shouting"
    styles: "unfriendly"
    styles: "whispering"
    styles: "hopeful"
}
voices {
    name: "en-US-AmberNeural"
    model: "enhanced"
    language: "en-US"
    gender: FEMALE
    sample_rate_hz: 24000
}
voices {
    name: "en-US-AnaNeural"
    model: "enhanced"
    language: "en-US"
    gender: FEMALE
    sample_rate_hz: 24000
}
. . . voices omitted here . . . 
voices {
    name: "en-US-ZiraRUS"
    model: "enhanced"
    language: "en-US"
    gender: FEMALE
    sample_rate_hz: 24000
}

2022-10-24 15:56:27,405 (140266945111872) INFO  Done running file [flow.py]
2022-10-24 15:56:27,407 (140266945111872) INFO  Iteration #1 complete
2022-10-24 15:56:27,407 (140266945111872) INFO  Done

SynthesisRequest

Input message for the Synthesize method. Specifies input text, audio parameters, and events to subscribe to, in exchange for synthesized audio. See Supported fields and defaults.

For more examples, see Sample synthesis client for Neural TTSaaS > Synthesize text input and Synthesize SSML input.

The SynthesisRequest message includes:

SynthesisRequest
  voice (Voice)
    name
  audio_params (AudioParameters)
    audio_format (AudioFormat)
  input (Input)
    text (Text)
    ssml (SSML)
  event_params (EventParameters)
    send_bookmark_marker_events
    send_visemes
    suppress_input
  client_data
  user_id

This synthesis request includes most fields:

SynthesisRequest(
    voice = Voice(
        name = "en-US-JennyNeural"
    ),
    audio_params = AudioParameters(
        audio_format = AudioFormat(
            pcm = PCM(sample_rate_hz = 22050) 
        )
    ),
    input = Input(
        text = Text(
            text = "Your coffee will be ready in 5 minutes"
        )
    ),
    event_params = EventParameters(
        send_visemes = True  
    ),
    client_data = {"company":"Aardvark Coffee","user":"Leslie"},
    user_id = "leslie.somebody@aardvark.com"
)

This is a minimal synthesis request, using all defaults:

SynthesisRequest(
    voice = Voice(
        name = "en-US-JennyNeural"
    ),
    input = Input(
        text = Text(
            text = "Your coffee will be ready in 5 minutes."
        )
    )
)

The API voice field is optional in SSML input. A voice may instead by provided in the <voice> element in the SSML input.

SynthesisRequest(
    input = Input(
        ssml = SSML(
            text = '''<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US">
            <voice name="en-US-JennyNeural">Your coffee will be ready in 5 minutes.</voice>
            </speak>'''
        )
    )
)

AudioParameters

Input message for audio-related parameters during synthesis, including encoding, volume, and audio length. Included in SynthesisRequest.

The AudioParameters message includes:

SynthesisRequest
  voice (Voice)
  audio_params (AudioParameters)
    audio_format (AudioFormat)
      pcm (PCM)
      alaw (Alaw)
      ulaw (Ulaw)
      ogg_opus (OggOpus)
      opus (Opus)

AudioFormat

Input message for audio encoding of synthesized text. Included in AudioParameters.

The AudioFormat message includes:

SynthesisRequest
  voice (Voice)
  audio_params (AudioParameters)
    audio_format (AudioFormat)
      pcm (PCM)
        sample_rate_hz
      alaw (Alaw)
      ulaw (Ulaw)
      ogg_opus (OggOpus)
        sample_rate_hz
      opus (Opus)
        sample_rate_hz
        bit_rate_bps

The PCM audio format is shown, with alternatives in commented lines:

SynthesisRequest(
    voice = Voice(
        name = "en-US-JennyNeural"
    ),
    audio_params = AudioParameters(
        audio_format = AudioFormat(
            pcm = PCM(sample_rate_hz = 22050)
#           alaw = ALaw()
#           ulaw = ULaw()
#           ogg_opus = OggOpus(sample_rate_hz = 16000)
#           opus = Opus(sample_rate_hz = 16000, bit_rate_bps = 30000)
        )
    )
)

PCM

Input message defining PCM sample rate. Included in AudioFormat.

ALaw

Input message defining A-law audio format. Included in AudioFormat. G.711 audio formats are set to 8kHz.

ULaw

Input message defining Mu-law audio format. Included in AudioFormat. G.711 audio formats are set to 8kHz.

OggOpus

Input message defining Ogg Opus output rate. Included in AudioFormat.

Opus

Input message defining Opus output rate. Included in AudioFormat.

Input

Input message containing text to synthesize and synthesis parameters, including tuning data, etc. Included in SynthesisRequest. The type of input may be plain text or SSML. See Input to synthesize for examples.

The Input message includes:

SynthesisRequest
  voice (Voice)
  input (Input)
    text (Text)
      text
    ssml (SSML)
      text

Text

Input message for synthesizing plain text. The encoding must be UTF-8. For plain text input, a voice field is required.

For example, this is plain text input:

SynthesisRequest(
    voice = Voice(
        name = "en-US-JennyNeural"
    ),
    input = Input(
        text = Text(
            text = "Your coffee will be ready in 5 minutes"
        )
    )
)

SSML

Input message for synthesizing SSML input. See SSML elements for supported elements and examples.

For example, this is SSML input. The SynthesisRequest voice field is ignored and may be omitted because the voice is set in the <voice> element in the SSML.

SynthesisRequest(
    input = Input(
        ssml = SSML(
            text = '''<speak  xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US" version="1.0">
            <voice name = "en-US-JennyNeural">Your coffee will be ready in 5 minutes.</voice>
            </speak>'''
        ) 
    )
)

EventParameters

Input message that defines event subscription parameters. Included in SynthesisRequest. Events that are requested are sent throughout the SynthesisResponse stream as they are generated.

Log events are produced throughout a synthesis request for events such as a voice loaded by the server or an audio chunk being ready to send.

The EventParameters message includes:

SynthesisRequest
  voice (Voice)
  input (Input)
  event_params (EventParameters)
    send_bookmark_marker_events
    send_visemes
    suppress_input

Event parameters in SynthesisRequest

SynthesisRequest(
    voice = Voice(
    name = "en-US-JennyNeural"
    ),
    input = Input(
        text = Text(
            text = "Your coffee will be ready in 5 minutes."
        )
    ),
    event_params = EventParameters(
        send_visemes = True
    )
)

SynthesisResponse

Output message in response to a SynthesisRequest, consisting of a stream of SynthesisResponse responses. Each response contains one of:

• A status response, indicating completion or failure of the request. This is received only once and signifies the end of a Synthesize call.
• A list of events the client has requested. This can be received many times. See EventParameters for details.
• An audio buffer. This may be received many times.

The SynthesisResponse message includes:

SynthesisResponse
  status (Status)
    code
    message
    details
  events (Events)
    event (Event)
      name
      values
  audio

Status

Output message containing a status response, indicating completion or failure of a Synthesize call. Included in SynthesisResponse.

Events

Output message defining a container for a list of events. This container is needed because oneof does not allow repeated parameters in Protobuf. Included in SynthesisResponse.

Event

Output message defining an event message. Included in Events. See EventParameters for details.

Scalar value types