Streaming API: Message Sequence Breakdown
For a description of each message field, refer to our Turn object explanation.
Understanding transcript vs utterance
Before walking through the message sequence, it’s important to understand the difference between the transcript and utterance fields:
transcript— The running transcript of the current turn. Updated with each Turn message, it accumulates finalized words progressively. Use this field for displaying live captions or getting the full turn text at any point.utterance— Populated when the system detects a pause in speech (based on a combination of silence duration and model confidence). Contains the finalized text of the current utterance at that moment. On all other Turn messages,utteranceis an empty string"". This enables eager LLM inference, allowing you to start processing text before the turn officially ends. Note that the message whereutteranceis populated may or may not also be anend_of_turn: truemessage, depending on whether the turn ends at the same time.
Key takeaway: To get the complete text of a turn, always read
transcript. The utterance field is an optimization for low-latency voice
agent pipelines — it lets you start processing text as soon as an utterance
boundary is detected, before the turn officially ends.
For this example, we are going to walk through a user saying:
Hi my name is Sonny. I am a voice agent.
Session initialization
When the session begins you will get a message indicating the session ID and expiration time.
Initial utterance and turn
Start of utterance (Partial transcript), and Start of turn
The user begins saying Hi my name is Sonny. Each message return finalizes words as they come through. Finalized words appear in the transcript field.
End of utterance (Final transcript), and End of turn
When the utterance is completed, the turn may or may not be predicted to end (this depends entirely on the end_of_turn_confidence value). Since the utterance is complete, we push the utterance value to the utterance key so you can process it for pre-emptive generation. This is especially useful to get the fastest possible return from our streaming STT model.
In this case, the turn also ends "end_of_turn":True since end_of_turn_confidence is greater than 0.5. If the turn confidence was higher, the turn will end later - see End of utterance (Final Transcript), not End of turn section below.
Because the utterance end and turn end occur on the same message here,
both utterance and transcript contain the finalized text. When the
utterance ends before the turn (see
below), utterance is
populated on the earlier message and will be empty on the subsequent
end_of_turn: true message.
End of turn formatting
Once the turn ends, if formatting is enabled there will be an additional message where "turn_is_formatted":True comes back. This will format the final message.
Since LLMs can parse unformatted text, waiting for this message is NOT recommended for voice agents as it adds additional latency. It is recommended for other use cases like closed captioning where the transcript will be displayed to the end user.
Additional utterance(s) and turn(s)
Start of a new utterance (Partial transcript), and Start of a new turn
In this case, the user is not actually done speaking yet. Normally you would have a VAD set up to cancel the end of turn above and let the user keep speaking even though End of Turn was predicted. The user now continues to speak I am a voice agent.
End of utterance (Final Transcript), not End of turn
In many cases, the turn may not end but we will have finalized an utterance. For this reason, we supply the utterance field to finalize each utterance as fast as possible for downstream LLM pre-emptive generation.
End of turn, after End of utterance
Once the confidence score of 0.5 is passed for End of Turn, we will end the turn.
Notice that utterance is an empty string in this message. The utterance
field was already populated on the previous
message when the
utterance boundary was first detected. On the end_of_turn: true message, use
transcript to get the complete turn text.
End of turn formatting
Lastly we will format the transcript if formatting is enabled. Again this is only recommended for use cases where the end user will be shown the transcript as it adds some latency for voice agents.
Best practices
Voice agents
- Grab the
utteranceparameter as soon as it is available for the fastest STT model response. This is particularly useful if you are using an external Turn Detection model. You can read more about configuring for third-party turn detection here.- If you are using AssemblyAI’s turn detection, you can simply configure your settings to be Aggressive, Balanced, or Conservative.
- Avoid using
format_turnsas it will significantly increase latency. LLMs don’t need formatting so you can just pass raw text as soon as its ready. - Utilize the
end_of_turnparameter to determine when a user is likely to end their turn. Combined with a VAD you can determine if the user is continuing to speak, or if the turn has ended and you can safely interrupt with your voice agent.- Note that AssemblyAI has a built-in VAD to the model. If you modify
min_turn_silencelatency will increase but you can substitute the model entirely for a VAD.
- Note that AssemblyAI has a built-in VAD to the model. If you modify
Notetakers and Closed captioning
- Utilize
format_turnssince text will be displayed to the end user and this will make it human-readable. The added latency will be less noticeable to the human eye in these applications. - Utilize
keyterms_promptto increase the accuracy of rare words and entities. Again since the text will be shown to the end user this will improve their perception of accuracy.
Session termination
When the session ends (either by sending a Terminate message or when the session expires), the server sends a Termination message with the total audio and session durations.
After receiving the Termination message, no further messages will be sent and the WebSocket connection will be closed.
