Streaming Audio

Quickstart

In this quick guide you will learn how to use AssemblyAI’s Streaming Speech-to-Text feature to transcribe audio from your microphone.

To run this quickstart you will need:

  • Python or JavaScript installed
  • A valid AssemblyAI API key

To run the quickstart:

1

Create a new Python file (for example, main.py) and paste the code provided below inside.

2

Insert your API key to line 17.

3

Install the necessary libraries

$pip install assemblyai pyaudio
4

Run with python main.py

1import logging
2from typing import Type
3
4import assemblyai as aai
5from assemblyai.streaming.v3 import (
6 BeginEvent,
7 StreamingClient,
8 StreamingClientOptions,
9 StreamingError,
10 StreamingEvents,
11 StreamingParameters,
12 StreamingSessionParameters,
13 TerminationEvent,
14 TurnEvent,
15)
16
17api_key = "<YOUR_API_KEY>"
18
19logging.basicConfig(level=logging.INFO)
20logger = logging.getLogger(__name__)
21
22
23def on_begin(self: Type[StreamingClient], event: BeginEvent):
24 print(f"Session started: {event.id}")
25
26
27def on_turn(self: Type[StreamingClient], event: TurnEvent):
28 print(f"{event.transcript} ({event.end_of_turn})")
29
30 if event.end_of_turn and not event.turn_is_formatted:
31 params = StreamingSessionParameters(
32 format_turns=True,
33 )
34
35 self.set_params(params)
36
37
38def on_terminated(self: Type[StreamingClient], event: TerminationEvent):
39 print(
40 f"Session terminated: {event.audio_duration_seconds} seconds of audio processed"
41 )
42
43
44def on_error(self: Type[StreamingClient], error: StreamingError):
45 print(f"Error occurred: {error}")
46
47
48def main():
49 client = StreamingClient(
50 StreamingClientOptions(
51 api_key=api_key,
52 api_host="streaming.assemblyai.com",
53 )
54 )
55
56 client.on(StreamingEvents.Begin, on_begin)
57 client.on(StreamingEvents.Turn, on_turn)
58 client.on(StreamingEvents.Termination, on_terminated)
59 client.on(StreamingEvents.Error, on_error)
60
61 client.connect(
62 StreamingParameters(
63 sample_rate=16000,
64 format_turns=True,
65 )
66 )
67
68 try:
69 client.stream(
70 aai.extras.MicrophoneStream(sample_rate=16000)
71 )
72 finally:
73 client.disconnect(terminate=True)
74
75
76if __name__ == "__main__":
77 main()

Core concepts

Streaming API: Message Sequence Breakdown

Universal-Streaming is built based upon two core concepts: Turn objects and immutable transcriptions.

Turn object

A Turn object is intended to correspond to a speaking turn in the context of voice agent applications, and therefore it roughly corresponds to an utterance in a broader context. We assign a unique ID to each Turn object, which is included in our response. Specifically, the Universal-Streaming response is formatted as follows:

1{
2 "turn_order": 1,
3 "turn_is_formatted": false,
4 "end_of_turn": false,
5 "transcript": "modern medicine is",
6 "end_of_turn_confidence": 0.7,
7 "words": [
8 { "text": "modern", "word_is_final": true, ... },
9 { "text": "medicine", "word_is_final": true, ... },
10 { "text": "is", "word_is_final": true, ... },
11 { "text": "amazing", "word_is_final": false, ... }
12 ]
13}
  • turn_order: Integer that increments with each new turn
  • turn_is_formatted: Boolean indicating if the text in the transcript field is formatted. Text formatting is enabled when format_turns is set to true. It adds punctuation as well as performs casing and inverse text normalization to display various entities, such as dates, times, and phone numbers, in a human-friendly format
  • end_of_turn: Boolean indicating if this is the end of the current turn
  • transcript: String containing only finalized words
  • end_of_turn_confidence: Floating number (0-1) representing the confidence that the current turn has finished, i.e., the current speaker has completed their turn
  • words: List of Word objects with individual metadata

Each Word object in the words array includes:

  • text: The string representation of the word
  • word_is_final: Boolean indicating if the word is finalized, where a finalized word means the word won’t be altered in future transcription responses
  • start: Timestamp for word start
  • end: Timestamp for word end
  • confidence: Confidence score for the word

Immutable transcription

AssemblyAI’s streaming system receives audio in a streaming fashion, it returns transcription responses in real-time using the format specified above. Unlike many other streaming speech-to-text models that implement the concept of partial/variable transcriptions to show transcripts in an ongoing manner, Universal-Streaming transcriptions are immutable. In other words, the text that has already been produced will not be overwritten in future transcription responses. Therefore, with Universal-Streaming, the transcriptions will be delivered in the following way:

1→ hello my na
2→ hello my name
3→ hello my name
4→ hello my name is
5→ hello my name is zac
6→ hello my name is zack

When an end of the current turn is detected, you then receive a message with end_of_turn being true. Additionally, if you enable text formatting, you will also receive a transcription response with turn_is_formatted being true.

1→ hello my name is zack (unformatted)
2→ Hello my name is Zack. (formatted)

In this example, you may have noticed that the last word of each transcript may occasionally be a subword (“zac” in the example shown above). Each Word object has the word_is_final field to indicate whether the model is confident that the last word is a completed word. Note that, except for the last word, word_is_final is always true.

Recommendations

  • Use an audio chunk size of 50ms. Larger chunk sizes are workable, but may result in latency fluctuations.

Voice agent use case

To optimize for latency, we recommend using the unformatted transcript as it’s received more quickly than the formatted version. In typical voice agent applications involving large language models (LLMs), the lack of formatting makes little impact on the subsequent LLM processing.

Possible implementation strategy

Since all our transcripts are immutable, the data is immediately ready to be sent in the voice agent pipeline. Here’s one way to handle the conversation flow:

  1. When you receive a transcription response with the end_of_turn value being true but your Voice Agent (i.e., your own turn detection logic) hasn’t detected end of turn, save this data in a variable (let’s call it running_transcript).
  2. When the voice agent detects end of turn, combine the running_transcript with the latest partial transcript and send it to the LLM.
  3. Clear the running_transcript after sending and be sure to ignore the next transcription with end_of_turn of true, that will eventually arrive for the latest partial you used. This prevents duplicate information from being processed in future turns.

What you send to the voice agent should look like: running_transcript + ’ ’ + latest_partial

Example flow

1→ hello my na
2→ hello my name
3→ hello my name
4→ hello my name is
5→ hello my name is son
6→ hello my name is sonny (final – added to running_transcript)
7
8→ I
9→ I work at
10→ I work at assembly ai (final – added to running_transcript)
11
12→ how
13→ how can
14→ how can I help
15→ how can I help you today (latest partial, final not yet received)
16
17<END_OF_TURN_DETECTED>
18"hello my name is sonny I work at assembly ai how can I help you today" → sent to LLM
19
20<running_transcript cleared>
21<final for latest_partial not added to running_transcript>

Utilizing our ongoing transcriptions in this manner will allow you to achieve the fastest possible latency for this step of your Voice Agent. Please reach out to the AssemblyAI team with any questions.

Instead of building your own logic for conversation flow handling, you may use AssemblyAI via integrations with tools like LiveKit and Pipecat. See the next section of our docs for more information on using these orchestrators.

Voice agent orchestrators

Reference

Connection parameters

token
string

Authenticate the session using a generated temporary token.

sample_rate
intRequired

The sample rate of the audio stream.

encoding
stringRequired

The encoding of the audio stream. Allowed values: pcm_s16le, pcm_mulaw

format_turns
booleanDefaults to False

Whether to return formatted final transcripts.

If enabled, formatted final transcripts will be emitted shortly following an end-of-turn detection.

end_of_turn_confidence_threshold
floatDefaults to 0.7

The confidence threshold to use when determining if the end of a turn has been reached.

min_end_of_turn_silence_when_confident
intDefaults to 160

The minimum amount of silence required to detect end of turn when confident.

max_turn_silence
intDefaults to 2400

The maximum amount of silence allowed in a turn before end of turn is triggered.

Audio requirements

The audio format must conform to the following requirements:

  • PCM16 or Mu-law encoding (See Specify the encoding)
  • A sample rate that matches the value of the sample_rate parameter
  • Single-channel
  • 50 milliseconds of audio per message (recommended)

Message types

You send:

1"UklGRtjIAABXQVZFZ"
1{
2 "type": "UpdateConfiguration",
3 "end_of_turn_confidence_threshold": 0.5
4}
1{ "type": "Terminate" }
1{ "type": "ForceEndpoint" }

You receive:

1{
2 "type": "Begin",
3 "id": "cfd280c7-5a9b-4dd6-8c05-235ccfa3c97f",
4 "expires_at": 1745483367
5}
1{
2 "turn_order": 0,
3 "turn_is_formatted": true,
4 "end_of_turn": true,
5 "transcript": "Hi, my name is Sonny.",
6 "end_of_turn_confidence": 0.8095446228981018,
7 "words":
8 [
9 {
10 "start": 1440,
11 "end": 1520,
12 "text": "Hi,",
13 "confidence": 0.9967870712280273,
14 "word_is_final": true
15 },
16 {
17 "start": 1600,
18 "end": 1680,
19 "text": "my",
20 "confidence": 0.999546468257904,
21 "word_is_final": true
22 },
23 {
24 "start": 1600,
25 "end": 1680,
26 "text": "name",
27 "confidence": 0.9597182273864746,
28 "word_is_final": true
29 },
30 {
31 "start": 1680,
32 "end": 1760,
33 "text": "is",
34 "confidence": 0.8261497616767883,
35 "word_is_final": true
36 },
37 {
38 "start": 2320,
39 "end": 3040,
40 "text": "Sonny.",
41 "confidence": 0.5737350583076477,
42 "word_is_final": true
43 }
44 ],
45 "type": "Turn"
46}

For the full breakdown of the message sequence for a turn, see the Message sequence breakdown guide.

1{
2 "type": "Termination",
3 "audio_duration_seconds": 2000,
4 "session_duration_seconds": 2000
5}