ModelsStreaming Speech-to-text

Model selection

The speech_model connection parameter lets you specify which model to use for streaming transcription.

Available models

NameParameterDescription
Universal-3 Pro"speech_model": "u3-rt-pro"Our highest accuracy model with native multilingual code switching, entity accuracy, performance across varying audio, and prompting support.
Universal-Streaming English (default)"speech_model": "universal-streaming-english"Our low-latency streaming model optimized for real-time English transcription.
Universal-Streaming Multilingual"speech_model": "universal-streaming-multilingual"Our low-latency streaming model optimized for real-time multilingual transcription in English, Spanish, French, German, Italian, and Portuguese.
Whisper Streaming"speech_model": "whisper-rt"Transcribe audio streams in 99 languages using the WhisperLiveKit model with automatic language detection.

Choosing a model

FeatureUniversal-3-Pro StreamingUniversal-Streaming EnglishUniversal-Streaming MultilingualWhisper Streaming
LatencyFastFastestFastModerate
Partial transcriptsYesYesYesYes
MultilingualNative Code SwitchingNoPer Turn99 languages (auto-detected)
Entity accuracyBestOkayOkayOkay
Disfluencies & filler wordsYesNoNoNo
Language detectionYesNoYesYes (with confidence scores)
Non-speech tagsNoNoNoYes ([Silence], [Music], etc.)
CustomizationKeyterms prompting (known context) + Native prompting (unknown context)Keyterms prompting (known context)Keyterms prompting (known context)No

For detailed setup and configuration of Universal-3-Pro streaming, see the Universal-3 Pro page. For prompting guidance, see the Prompting guide.

For detailed setup and configuration of Whisper streaming, see the Multilingual streaming page.

End-to-end example

You can select a model by setting the speech_model connection parameter when connecting to the streaming API:

1import logging
2from typing import Type
3
4import assemblyai as aai
5from assemblyai.streaming.v3 import (
6    BeginEvent,
7    StreamingClient,
8    StreamingClientOptions,
9    StreamingError,
10    StreamingEvents,
11    StreamingParameters,
12    TurnEvent,
13    TerminationEvent,
14)
15
16api_key = "<YOUR_API_KEY>"
17
18logging.basicConfig(level=logging.INFO)
19logger = logging.getLogger(__name__)
20
21def on_begin(self: Type[StreamingClient], event: BeginEvent):
22    print(f"Session started: {event.id}")
23
24def on_turn(self: Type[StreamingClient], event: TurnEvent):
25    print(f"{event.transcript} ({event.end_of_turn})")
26
27def on_terminated(self: Type[StreamingClient], event: TerminationEvent):
28    print(
29        f"Session terminated: {event.audio_duration_seconds} seconds of audio processed"
30    )
31
32def on_error(self: Type[StreamingClient], error: StreamingError):
33    print(f"Error occurred: {error}")
34
35def main():
36    client = StreamingClient(
37        StreamingClientOptions(
38            api_key=api_key,
39            api_host="streaming.assemblyai.com",
40        )
41    )
42
43    client.on(StreamingEvents.Begin, on_begin)
44    client.on(StreamingEvents.Turn, on_turn)
45    client.on(StreamingEvents.Termination, on_terminated)
46    client.on(StreamingEvents.Error, on_error)
47
48    client.connect(
49        StreamingParameters(
50            sample_rate=16000,
51            speech_model="u3-rt-pro",  # or "universal-streaming-english", "universal-streaming-multilingual", "whisper-rt"
52            min_turn_silence=100,
53            max_turn_silence=1200,
54            # format_turns=True,  # Whether to return formatted final transcripts (not applicable to u3-rt-pro)
55        )
56    )
57
58    try:
59        client.stream(
60            aai.extras.MicrophoneStream(sample_rate=16000)
61        )
62    finally:
63        client.disconnect(terminate=True)
64
65if __name__ == "__main__":
66    main()