Streaming

AssemblyAI’s Streaming Speech-to-Text (STT) allows you to transcribe live audio streams with high accuracy and low latency. By streaming your audio data to our secure WebSocket API, you can receive transcripts back within a few hundred milliseconds.

Getting started

Check out our getting started guides:

If your programming language isn’t supported yet, see the WebSocket API:

• Streaming API reference

Audio requirements

The audio format must conform to the following requirements:

• PCM16 or Mu-law encoding (See Specify the encoding)
• A sample rate that matches the value of the supplied sample_rate parameter
• Single-channel
• 100 to 2000 milliseconds of audio per message

Specify the encoding

1
transcriber = aai.RealtimeTranscriber(
2
    ...,
3
    encoding=aai.AudioEncoding.pcm_mulaw
4
)

Add custom vocabulary

You can add up to 2500 characters of custom vocabulary to boost their transcription probability.

1
transcriber = aai.RealtimeTranscriber(
2
    ...,
3
    word_boost=["aws", "azure", "google cloud"]
4
)

Authenticate with a temporary token

If you need to authenticate on the client, you can avoid exposing your API key by using temporary authentication tokens. You should generate this token on your server and pass it to the client.

1
token = aai.RealtimeTranscriber.create_temporary_token(
2
    expires_in=60
3
)

1
transcriber = aai.RealtimeTranscriber(
2
    ...,
3
    token=token
4
)

Manually end current utterance

1
transcriber.force_end_utterance()

Manually ending an utterance immediately produces a final transcript.

Configure the threshold for automatic utterance detection

You can configure the threshold for how long to wait before ending an utterance.

1
transcriber = aai.RealtimeTranscriber(
2
    ...,
3
    end_utterance_silence_threshold=500
4
)
5
6
# after connecting
7
transcriber.configure_end_utterance_silence_threshold(300)

Disable partial transcripts

If you’re only using the final transcript, you can disable partial transcripts to reduce network traffic.

1
transcriber = aai.RealtimeTranscriber(
2
    ...,
3
    disable_partial_transcripts=True
4
)

Enable extra session information

1
# Define a callback to handle the session information message
2
def on_extra_session_information(data: aai.RealtimeSessionInformation):
3
    print(data.audio_duration_seconds)
4
5
# Configure the RealtimeTranscriber
6
transcriber = aai.RealtimeTranscriber(
7
    ...,
8
    on_extra_session_information=on_extra_session_information,
9
)

Best practices

Here are some best practices to get the best results from Streaming Speech-to-Text:

Audio quality

• Use a sample rate of at least 16000 Hz for better transcription accuracy. Higher quality audio input generally leads to better results.
• For noisy environments, see our Cookbook example for implementing noise reduction in your streaming pipeline.

Connection management

• Keep the WebSocket connection open for the entire duration of a session instead of frequently reconnecting. Reconnecting adds latency overhead and can impact real-time performance.
• For client-side implementations of our Streaming API, we recommend first creating a temporary authentication token using a temporary token to avoid exposing your API key.

Optimizing for latency

You can optimize for lower latency in several ways:

• Use partial transcripts instead of waiting for final transcripts. Partial transcripts provide faster feedback but may be less accurate. See our Cookbook example for more information.
• Configure the threshold for silence between speaker turns using end_utterance_silence_threshold to control when utterances are finalized.
• If you don’t need real-time feedback, you can disable partial transcripts to reduce network traffic.

Learn more

To learn about using Streaming Speech-to-Text, see the following resources:

• Blog post: Automatically Transcribe Zoom Calls in Real Time
• Blog post: Transcribe Twilio Phone Calls
• Blog post: Real Time Speech Recognition with Python and PyAudio
• GitHub: End-to-end examples
• GitHub: Cookbook examples
• GitHub: Use Express.js for Streaming Speech-to-Text
• Streaming API reference