> ## Documentation Index > Fetch the complete documentation index at: https://assemblyai.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Streaming WebSocket API > Stream audio and receive real-time transcription results. ### Endpoints | WebSocket URL | Description | | --- | --- | | `wss://streaming.assemblyai.com/v3/ws` | **Global (default).** Latency-optimized — automatically routes each connection to the nearest region. | | `wss://streaming.us.assemblyai.com/v3/ws` | **US data residency.** Audio and transcription data never leaves the US. | | `wss://streaming.eu.assemblyai.com/v3/ws` | **EU data residency.** Audio and transcription data never leaves the EU. | Use a data residency endpoint for compliance requirements; see [Cloud endpoints & data residency](https://www.assemblyai.com/docs/streaming/endpoints-and-data-zones). ### Headers | Header | Description | | --- | --- | | `Authorization` | **Required.** Your AssemblyAI API key (no `Bearer` prefix). Required on every connection unless you authenticate with the `token` query parameter instead. | | `AssemblyAI-Version` | **Optional.** API version pin. Defaults to the latest version. | Browsers cannot set headers on a WebSocket connection. In that case, [generate a temporary token](https://www.assemblyai.com/docs/streaming/authenticate-with-a-temporary-token) server-side and pass it via the `token` query parameter instead. Never expose your permanent API key in a URL or in client-side code. ### Query parameters All query parameters below are optional — a connection with only authentication set uses the default speech model (`universal-3-5-pro`) at 16 kHz `pcm_s16le`. Parameters apply to every model unless their description says otherwise. ## AsyncAPI ````yaml api-reference/specs/streaming.yaml streaming id: streaming title: Streaming description: > Stream audio and receive real-time transcription results. ### Endpoints | WebSocket URL | Description | | --- | --- | | `wss://streaming.assemblyai.com/v3/ws` | **Global (default).** Latency-optimized — automatically routes each connection to the nearest region. | | `wss://streaming.us.assemblyai.com/v3/ws` | **US data residency.** Audio and transcription data never leaves the US. | | `wss://streaming.eu.assemblyai.com/v3/ws` | **EU data residency.** Audio and transcription data never leaves the EU. | Use a data residency endpoint for compliance requirements; see [Cloud endpoints & data residency](https://www.assemblyai.com/docs/streaming/endpoints-and-data-zones). ### Headers | Header | Description | | --- | --- | | `Authorization` | **Required.** Your AssemblyAI API key (no `Bearer` prefix). Required on every connection unless you authenticate with the `token` query parameter instead. | | `AssemblyAI-Version` | **Optional.** API version pin. Defaults to the latest version. | Browsers cannot set headers on a WebSocket connection. In that case, [generate a temporary token](https://www.assemblyai.com/docs/streaming/authenticate-with-a-temporary-token) server-side and pass it via the `token` query parameter instead. Never expose your permanent API key in a URL or in client-side code. ### Query parameters All query parameters below are optional — a connection with only authentication set uses the default speech model (`universal-3-5-pro`) at 16 kHz `pcm_s16le`. Parameters apply to every model unless their description says otherwise. servers: - id: production protocol: wss host: streaming.assemblyai.com bindings: [] variables: [] - id: usDataResidency protocol: wss host: streaming.us.assemblyai.com bindings: [] variables: [] - id: euDataResidency protocol: wss host: streaming.eu.assemblyai.com bindings: [] variables: [] address: /v3/ws parameters: - id: token jsonSchema: type: string description: >- Authenticate with a [temporary token](/api-reference/streaming-api/generate-streaming-token) via query parameter. Use this where request headers cannot be set — for example, browser WebSocket connections — and prefer the `Authorization` header everywhere else. Generate the temporary token server-side; never expose your permanent API key in a URL. description: >- Authenticate with a [temporary token](/api-reference/streaming-api/generate-streaming-token) via query parameter. Use this where request headers cannot be set — for example, browser WebSocket connections — and prefer the `Authorization` header everywhere else. Generate the temporary token server-side; never expose your permanent API key in a URL. type: string required: true deprecated: false - id: speech_model jsonSchema: type: string description: >- The speech model to use for the session. See [Select the speech model](https://www.assemblyai.com/docs/streaming/select-the-speech-model) for the differences between models. enum: - universal-3-5-pro - universal-streaming-english - universal-streaming-multilingual description: >- The speech model to use for the session. See [Select the speech model](https://www.assemblyai.com/docs/streaming/select-the-speech-model) for the differences between models. type: string required: true deprecated: false - id: language_codes jsonSchema: type: string description: >- Steers transcription toward a set of languages by biasing output toward them on a per-token basis while still allowing native code-switching among them. Pass a list of language codes for the languages you expect (for example, `["en", "es"]`), or a single-element list (for example, `["es"]`) for a monolingual session. When unset, no steering is applied and the model code-switches natively across all of its supported languages. This is distinct from `language_detection`, which only controls whether the detected language is reported. Accepted codes: `en`, `es`, `fr`, `de`, `it`, `pt`, `tr`, `nl`, `sv`, `no`, `da`, `fi`, `hi`, `vi`, `ar`, `he`, `ja`, `zh`. Universal-3.5 Pro Streaming only. examples: - '["en", "es"]' description: >- Steers transcription toward a set of languages by biasing output toward them on a per-token basis while still allowing native code-switching among them. Pass a list of language codes for the languages you expect (for example, `["en", "es"]`), or a single-element list (for example, `["es"]`) for a monolingual session. When unset, no steering is applied and the model code-switches natively across all of its supported languages. This is distinct from `language_detection`, which only controls whether the detected language is reported. Accepted codes: `en`, `es`, `fr`, `de`, `it`, `pt`, `tr`, `nl`, `sv`, `no`, `da`, `fi`, `hi`, `vi`, `ar`, `he`, `ja`, `zh`. Universal-3.5 Pro Streaming only. type: string required: true deprecated: false - id: language_detection jsonSchema: type: string description: >- Whether to return `language_code` and `language_confidence` in turn messages. Available on Universal-3.5 Pro Streaming (which natively code-switches across all of its supported languages by default) and on Universal Streaming Multilingual only. enum: - 'true' - 'false' default: 'false' description: >- Whether to return `language_code` and `language_confidence` in turn messages. Available on Universal-3.5 Pro Streaming (which natively code-switches across all of its supported languages by default) and on Universal Streaming Multilingual only. type: string required: true deprecated: false - id: domain jsonSchema: type: string description: >- Enable domain-specific transcription models to improve accuracy for specialized terminology. Set to `"medical-v1"` to enable [Medical Mode](https://www.assemblyai.com/docs/streaming/medical-mode) for improved accuracy of medical terms such as medications, procedures, conditions, and dosages. Supported languages: English (`en`), Spanish (`es`), German (`de`), French (`fr`). If used with an unsupported language, the parameter is ignored and a warning is returned. enum: - medical-v1 description: >- Enable domain-specific transcription models to improve accuracy for specialized terminology. Set to `"medical-v1"` to enable [Medical Mode](https://www.assemblyai.com/docs/streaming/medical-mode) for improved accuracy of medical terms such as medications, procedures, conditions, and dosages. Supported languages: English (`en`), Spanish (`es`), German (`de`), French (`fr`). If used with an unsupported language, the parameter is ignored and a warning is returned. type: string required: true deprecated: false - id: mode jsonSchema: type: string description: >- Latency and accuracy preset that controls the model's turn-detection and partial-emission defaults. `max_accuracy` favors transcription quality, `min_latency` favors speed, and `balanced` trades off between the two. When omitted, the server applies its own preset, which determines the defaults for the mode-dependent fields `min_turn_silence` and `interruption_delay`. Universal-3.5 Pro Streaming only. enum: - max_accuracy - min_latency - balanced description: >- Latency and accuracy preset that controls the model's turn-detection and partial-emission defaults. `max_accuracy` favors transcription quality, `min_latency` favors speed, and `balanced` trades off between the two. When omitted, the server applies its own preset, which determines the defaults for the mode-dependent fields `min_turn_silence` and `interruption_delay`. Universal-3.5 Pro Streaming only. type: string required: true deprecated: false - id: encoding jsonSchema: type: string description: >- Encoding of the audio stream. `pcm_s16le` and `pcm_mulaw` are raw PCM formats. `opus` is raw Opus packets, where each binary WebSocket message must contain exactly one Opus packet. `ogg_opus` is an Ogg-encapsulated Opus byte stream (the format produced by ffmpeg, gstreamer, opusenc, and browser MediaRecorder); binary WebSocket messages can be arbitrary chunks of the stream. For both Opus encodings, `sample_rate` is ignored — the stream is self-describing. enum: - pcm_s16le - pcm_mulaw - opus - ogg_opus default: pcm_s16le description: >- Encoding of the audio stream. `pcm_s16le` and `pcm_mulaw` are raw PCM formats. `opus` is raw Opus packets, where each binary WebSocket message must contain exactly one Opus packet. `ogg_opus` is an Ogg-encapsulated Opus byte stream (the format produced by ffmpeg, gstreamer, opusenc, and browser MediaRecorder); binary WebSocket messages can be arbitrary chunks of the stream. For both Opus encodings, `sample_rate` is ignored — the stream is self-describing. type: string required: true deprecated: false - id: sample_rate jsonSchema: type: string description: >- Sample rate of the audio stream in Hz. Accepts any integer from 8000 to 96000. Ignored for Opus encodings (`opus`, `ogg_opus`). default: '16000' examples: - '16000' - '48000' description: >- Sample rate of the audio stream in Hz. Accepts any integer from 8000 to 96000. Ignored for Opus encodings (`opus`, `ogg_opus`). type: string required: true deprecated: false - id: min_turn_silence jsonSchema: type: string description: >- Silence duration in milliseconds before a speculative end-of-turn check. On Universal-3.5 Pro Streaming the check is punctuation-based (if terminal punctuation is found the turn ends, otherwise a partial is emitted and the turn continues) and the default is mode-dependent, set by the `mode` preset. On Universal Streaming the check is confidence-based and the default is 400 ms. Clamped to the range 50 to 10000 ms. See [Configuring Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details. examples: - '100' - '200' description: >- Silence duration in milliseconds before a speculative end-of-turn check. On Universal-3.5 Pro Streaming the check is punctuation-based (if terminal punctuation is found the turn ends, otherwise a partial is emitted and the turn continues) and the default is mode-dependent, set by the `mode` preset. On Universal Streaming the check is confidence-based and the default is 400 ms. Clamped to the range 50 to 10000 ms. See [Configuring Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details. type: string required: true deprecated: false - id: max_turn_silence jsonSchema: type: string description: >- Maximum silence in milliseconds before the turn is forced to end, regardless of punctuation. Defaults are 1536 ms on Universal-3.5 Pro Streaming (768 ms when `speaker_labels` is enabled) and 1280 ms on Universal Streaming. See [Configuring Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details. examples: - '700' - '1000' description: >- Maximum silence in milliseconds before the turn is forced to end, regardless of punctuation. Defaults are 1536 ms on Universal-3.5 Pro Streaming (768 ms when `speaker_labels` is enabled) and 1280 ms on Universal Streaming. See [Configuring Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details. type: string required: true deprecated: false - id: end_of_turn_confidence_threshold jsonSchema: type: string description: > The confidence threshold (0.0 to 1.0) to use when determining if the end of a turn has been reached. See [Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details. Universal Streaming (English and Multilingual) only. default: '0.4' examples: - '0.4' - '0.7' description: > The confidence threshold (0.0 to 1.0) to use when determining if the end of a turn has been reached. See [Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details. Universal Streaming (English and Multilingual) only. type: string required: true deprecated: false - id: vad_threshold jsonSchema: type: string description: >- The confidence threshold (0.0 to 1.0) for classifying audio frames as silence. Frames with VAD confidence below this value are considered silent. Increase for noisy environments to reduce false speech detection. Defaults are 0.2 on Universal-3.5 Pro Streaming (0.5 when `speaker_labels` is enabled) and 0.4 on Universal Streaming. description: >- The confidence threshold (0.0 to 1.0) for classifying audio frames as silence. Frames with VAD confidence below this value are considered silent. Increase for noisy environments to reduce false speech detection. Defaults are 0.2 on Universal-3.5 Pro Streaming (0.5 when `speaker_labels` is enabled) and 0.4 on Universal Streaming. type: string required: true deprecated: false - id: interruption_delay jsonSchema: type: string description: >- How soon the first partial is emitted in milliseconds. Accepts a value from 0 to 1000. Useful for tuning voice agent barge-in responsiveness or allowing earlier partials for early LLM inference. Larger values are more confident on interruptions, smaller values result in faster time to first partial. The server adds a fixed 256 ms on top of this value, so `0` yields an effective delay of 256 ms and `500` yields 756 ms. The default is mode-dependent and set by the server based on the `mode` preset. Universal-3.5 Pro Streaming only. examples: - '0' - '500' description: >- How soon the first partial is emitted in milliseconds. Accepts a value from 0 to 1000. Useful for tuning voice agent barge-in responsiveness or allowing earlier partials for early LLM inference. Larger values are more confident on interruptions, smaller values result in faster time to first partial. The server adds a fixed 256 ms on top of this value, so `0` yields an effective delay of 256 ms and `500` yields 756 ms. The default is mode-dependent and set by the server based on the `mode` preset. Universal-3.5 Pro Streaming only. type: string required: true deprecated: false - id: continuous_partials jsonSchema: type: string description: >- Whether to emit additional partial transcripts during long turns at a steady ~3 second cadence. When enabled (default), additional partials covering the full turn transcript are emitted approximately every 3 seconds while speech continues. When disabled, only one early partial is emitted near turn start. The first partial (timed by `interruption_delay`) is unaffected. When `speaker_labels` is enabled the server disables continuous partials by default. Universal-3.5 Pro Streaming only. default: 'true' description: >- Whether to emit additional partial transcripts during long turns at a steady ~3 second cadence. When enabled (default), additional partials covering the full turn transcript are emitted approximately every 3 seconds while speech continues. When disabled, only one early partial is emitted near turn start. The first partial (timed by `interruption_delay`) is unaffected. When `speaker_labels` is enabled the server disables continuous partials by default. Universal-3.5 Pro Streaming only. type: string required: true deprecated: false - id: include_partial_turns jsonSchema: type: string description: >- Whether to emit partial transcripts during the turn. When enabled (default), partial transcripts are forwarded as speech is still in progress alongside final turns. When disabled, only final turns (with end_of_turn true) are sent. Defaults to false when redact_pii is enabled, to prevent unredacted partial transcripts from reaching the client; set explicitly to true to override. default: 'true' description: >- Whether to emit partial transcripts during the turn. When enabled (default), partial transcripts are forwarded as speech is still in progress alongside final turns. When disabled, only final turns (with end_of_turn true) are sent. Defaults to false when redact_pii is enabled, to prevent unredacted partial transcripts from reaching the client; set explicitly to true to override. type: string required: true deprecated: false - id: format_turns jsonSchema: type: string description: >- Whether to return formatted final transcripts. Universal Streaming (English and Multilingual) only. enum: - 'true' - 'false' default: 'false' description: >- Whether to return formatted final transcripts. Universal Streaming (English and Multilingual) only. type: string required: true deprecated: false - id: prompt jsonSchema: type: string description: >- A contextual prompt describing what the audio is about, such as its domain, scenario, or full conversation details. The model uses this context to better recognize the vocabulary it makes likely, for example priming medical terminology for a cardiology call. It carries context about your audio, not transcription instructions, so formatting or behavioral commands (such as punctuation rules) are not supported. Maximum 1750 characters. Universal-3.5 Pro Streaming only. See [Prompting and keyterms](https://www.assemblyai.com/docs/streaming/prompting-and-keyterms) for details. description: >- A contextual prompt describing what the audio is about, such as its domain, scenario, or full conversation details. The model uses this context to better recognize the vocabulary it makes likely, for example priming medical terminology for a cardiology call. It carries context about your audio, not transcription instructions, so formatting or behavioral commands (such as punctuation rules) are not supported. Maximum 1750 characters. Universal-3.5 Pro Streaming only. See [Prompting and keyterms](https://www.assemblyai.com/docs/streaming/prompting-and-keyterms) for details. type: string required: true deprecated: false - id: keyterms_prompt jsonSchema: type: string description: >- A list of words and phrases to improve recognition accuracy for. Maximum 100 terms. See [Keyterms Prompting](https://www.assemblyai.com/docs/streaming/keyterms-prompting) for more details. description: >- A list of words and phrases to improve recognition accuracy for. Maximum 100 terms. See [Keyterms Prompting](https://www.assemblyai.com/docs/streaming/keyterms-prompting) for more details. type: string required: true deprecated: false - id: agent_context jsonSchema: type: string description: >- Your voice agent's spoken text (TTS reply). The model uses this as context for the next user turn, which improves accuracy on short or ambiguous replies and on spelled-out entities like emails or IDs. Set at connection time to seed the model with your agent's opening greeting, and/or update mid-stream via `UpdateConfiguration` after each agent reply. Each `UpdateConfiguration` replaces the previously set value. Maximum 1750 characters per value. Universal-3.5 Pro Streaming only. See [Context Carryover](https://www.assemblyai.com/docs/streaming/universal-3-pro/context-carryover). examples: - >- Welcome to the Krusty Krab, home of the Krabby Patty, may I take your order? description: >- Your voice agent's spoken text (TTS reply). The model uses this as context for the next user turn, which improves accuracy on short or ambiguous replies and on spelled-out entities like emails or IDs. Set at connection time to seed the model with your agent's opening greeting, and/or update mid-stream via `UpdateConfiguration` after each agent reply. Each `UpdateConfiguration` replaces the previously set value. Maximum 1750 characters per value. Universal-3.5 Pro Streaming only. See [Context Carryover](https://www.assemblyai.com/docs/streaming/universal-3-pro/context-carryover). type: string required: true deprecated: false - id: previous_context_n_turns jsonSchema: type: string description: >- Advanced. Maximum number of prior conversation entries (user transcripts and any `agent_context` values) carried forward as context for each transcription. Range 0 to 100. Set to `0` to disable automatic context carryover entirely. When unset, the server default applies (currently 5 turns). Most integrations should leave this unset. Universal-3.5 Pro Streaming only. description: >- Advanced. Maximum number of prior conversation entries (user transcripts and any `agent_context` values) carried forward as context for each transcription. Range 0 to 100. Set to `0` to disable automatic context carryover entirely. When unset, the server default applies (currently 5 turns). Most integrations should leave this unset. Universal-3.5 Pro Streaming only. type: string required: true deprecated: false - id: speaker_labels jsonSchema: type: string description: >- Whether to enable [Streaming Speaker Diarization](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels). When enabled, each Turn event will include a `speaker_label` field and each final word in the `words` array will include a `speaker` field for word-level speaker attribution. enum: - 'true' - 'false' default: 'false' description: >- Whether to enable [Streaming Speaker Diarization](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels). When enabled, each Turn event will include a `speaker_label` field and each final word in the `words` array will include a `speaker` field for word-level speaker attribution. type: string required: true deprecated: false - id: max_speakers jsonSchema: type: string description: >- A hard cap on the number of speaker labels in the audio stream (integer, 1-10). This is a strict limit, not a hint — once it is reached, any additional speakers are merged into the closest existing label rather than given a new one. Give the model a little headroom above the number of speakers you expect; setting it too high can cause over-splitting and return more speakers than are actually present. Only used when `speaker_labels` is enabled. See [Streaming Diarization](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels) for more details. examples: - '2' - '4' description: >- A hard cap on the number of speaker labels in the audio stream (integer, 1-10). This is a strict limit, not a hint — once it is reached, any additional speakers are merged into the closest existing label rather than given a new one. Give the model a little headroom above the number of speakers you expect; setting it too high can cause over-splitting and return more speakers than are actually present. Only used when `speaker_labels` is enabled. See [Streaming Diarization](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels) for more details. type: string required: true deprecated: false - id: voice_focus jsonSchema: type: string description: >- Enable [Voice Focus](https://www.assemblyai.com/docs/streaming/voice-focus) to isolate the primary voice and suppress background noise before transcription. Set to `near-field` for close-talking microphones (for example, headsets or phones) or `far-field` for distant microphones (for example, conference rooms). When unset, Voice Focus is off. enum: - near-field - far-field description: >- Enable [Voice Focus](https://www.assemblyai.com/docs/streaming/voice-focus) to isolate the primary voice and suppress background noise before transcription. Set to `near-field` for close-talking microphones (for example, headsets or phones) or `far-field` for distant microphones (for example, conference rooms). When unset, Voice Focus is off. type: string required: true deprecated: false - id: voice_focus_threshold jsonSchema: type: string description: >- Controls how aggressively Voice Focus suppresses background audio, from `0.0` (least) to `1.0` (most). Higher values are more aggressive. Requires `voice_focus` to be set, otherwise a validation error is returned. default: '0.7' description: >- Controls how aggressively Voice Focus suppresses background audio, from `0.0` (least) to `1.0` (most). Higher values are more aggressive. Requires `voice_focus` to be set, otherwise a validation error is returned. type: string required: true deprecated: false - id: redact_pii jsonSchema: type: string description: >- Redact PII from the transcribed text using the Redact PII model, can be true or false. Only applies to final turns. See [PII Redaction](https://www.assemblyai.com/docs/streaming/pii-redaction) for more details. enum: - 'true' - 'false' default: 'false' description: >- Redact PII from the transcribed text using the Redact PII model, can be true or false. Only applies to final turns. See [PII Redaction](https://www.assemblyai.com/docs/streaming/pii-redaction) for more details. type: string required: true deprecated: false - id: redact_pii_policies jsonSchema: type: string description: >- The list of PII Redaction policies to enable. Requires `redact_pii` to be `true`. See [PII redaction](https://www.assemblyai.com/docs/streaming/pii-redaction) for more details. description: >- The list of PII Redaction policies to enable. Requires `redact_pii` to be `true`. See [PII redaction](https://www.assemblyai.com/docs/streaming/pii-redaction) for more details. type: string required: true deprecated: false - id: redact_pii_sub jsonSchema: type: string description: >- The replacement logic for detected PII, can be `entity_name` or `hash`. Requires `redact_pii` to be `true`. See [PII redaction](https://www.assemblyai.com/docs/streaming/pii-redaction) for more details. enum: - entity_name - hash default: hash description: >- The replacement logic for detected PII, can be `entity_name` or `hash`. Requires `redact_pii` to be `true`. See [PII redaction](https://www.assemblyai.com/docs/streaming/pii-redaction) for more details. type: string required: true deprecated: false - id: filter_profanity jsonSchema: type: string description: >- Filter profanity from the transcribed text, can be true or false. See [Profanity Filtering](https://www.assemblyai.com/docs/streaming/filter-profanity-from-transcripts) for more details. enum: - 'true' - 'false' default: 'false' description: >- Filter profanity from the transcribed text, can be true or false. See [Profanity Filtering](https://www.assemblyai.com/docs/streaming/filter-profanity-from-transcripts) for more details. type: string required: true deprecated: false - id: llm_gateway jsonSchema: type: string description: >- JSON-stringified LLM Gateway configuration that processes each finalized turn. Follows the same interface as the [Chat Completions](/llm-gateway/chat-completions) endpoint and accepts `model`, `messages`, `tools`, `tool_choice`, `post_processing_steps`, and `max_tokens`. See [Apply LLM Gateway to Streaming](https://www.assemblyai.com/docs/guides/real_time_llm_gateway) for the full schema and examples. description: >- JSON-stringified LLM Gateway configuration that processes each finalized turn. Follows the same interface as the [Chat Completions](/llm-gateway/chat-completions) endpoint and accepts `model`, `messages`, `tools`, `tool_choice`, `post_processing_steps`, and `max_tokens`. See [Apply LLM Gateway to Streaming](https://www.assemblyai.com/docs/guides/real_time_llm_gateway) for the full schema and examples. type: string required: true deprecated: false - id: inactivity_timeout jsonSchema: type: string description: >- Optional time in seconds of inactivity before session is terminated (integer, minimum 5, maximum 3600). If not set, no inactivity timeout is applied. examples: - '5' - '3600' description: >- Optional time in seconds of inactivity before session is terminated (integer, minimum 5, maximum 3600). If not set, no inactivity timeout is applied. type: string required: true deprecated: false bindings: [] operations: - &ref_1 id: audio title: Audio description: >- Send audio data chunks for transcription. The payload must be of type bytes and contain audio data between 50ms and 1000ms in length. When streaming from a pre-recorded file, pace the chunks at approximately real-time (for example, sleep for the chunk's duration between sends) — sending chunks in a tight loop can produce inconsistent Turn messages. See the [Streaming quickstart](https://www.assemblyai.com/docs/streaming/getting-started/transcribe-streaming-audio) to get started. type: receive messages: - &ref_12 id: audioChunk contentType: application/octet-stream payload: - type: string format: binary x-parser-schema-id: name: Audio Data Chunk description: Client sends audio data as raw binary. headers: [] jsonPayloadSchema: type: string format: binary x-parser-schema-id: title: Audio Data Chunk description: Client sends audio data as raw binary. example: >- "\\x10\\x00\\x20\\x00\\x30\\x00\\x40\\x00\\x30\\x00\\x20\\x00\\x10\\x00\\x00\\x00\\xf0\\xff\\xe0\\xff\\xd0\\xff\\xc0\\xff" bindings: [] extensions: - id: x-parser-unique-object-id value: audioChunk bindings: [] extensions: &ref_0 - id: x-parser-unique-object-id value: streaming - &ref_2 id: updateConfiguration title: Update configuration description: >- Update streaming configuration parameters during an active session. You can update `prompt`, `keyterms_prompt`, `min_turn_silence`, `max_turn_silence`, `end_of_turn_confidence_threshold`, `continuous_partials`, `vad_threshold`, `interruption_delay`, `mode`, `language_codes`, and `agent_context`. type: receive messages: - &ref_13 id: updateConfiguration contentType: application/json payload: - name: Update Streaming Configuration description: >- Client message to update streaming configuration parameters during an active session. type: object properties: - name: type type: string description: UpdateConfiguration required: true - name: prompt type: string description: >- A contextual prompt describing what the audio is about, such as its domain, scenario, or full conversation details. The model uses this context to better recognize the vocabulary it makes likely. It carries context about your audio, not transcription instructions, so formatting or behavioral commands are not supported. Maximum 1750 characters. See [Prompting and keyterms](https://www.assemblyai.com/docs/streaming/prompting-and-keyterms) for details. required: false - name: keyterms_prompt type: array description: >- A list of words and phrases to boost recognition for. See [Keyterms Prompting](https://www.assemblyai.com/docs/streaming/keyterms-prompting) for more details. required: false properties: - name: item type: string required: false - name: min_turn_silence type: integer description: >- Silence duration in milliseconds before a speculative end-of-turn check. See [Configuring Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details. required: false - name: max_turn_silence type: integer description: >- Maximum silence in milliseconds before the turn is forced to end, regardless of punctuation. See [Configuring Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details. required: false - name: continuous_partials type: boolean description: >- Whether to emit additional partial transcripts during long turns at a steady ~3 second cadence. When enabled (default), additional partials covering the full turn transcript are emitted approximately every 3 seconds while speech continues. When disabled, only one early partial is emitted near turn start. The first partial (timed by `interruption_delay`) is unaffected. required: false - name: vad_threshold type: number description: >- The confidence threshold (0.0 to 1.0) for classifying audio frames as silence. Frames with VAD confidence below this value are considered silent. Increase for noisy environments to reduce false speech detection. required: false - name: interruption_delay type: integer description: >- How soon the first partial is emitted in milliseconds. Useful for tuning voice agent barge-in responsiveness or allowing earlier partials for early LLM inference. Larger values are more confident on interruptions, smaller values result in faster time to first partial. required: false - name: agent_context type: string description: >- Your voice agent's most recent spoken reply (TTS text). The model uses this as context for the next user turn, which improves accuracy on short or ambiguous replies and on spelled-out entities like emails or IDs. Each `UpdateConfiguration` replaces the previously set value. Maximum 1750 characters. See [Context Carryover](https://www.assemblyai.com/docs/streaming/universal-3-pro/context-carryover#passing-your-agents-reply-as-context). required: false - name: mode type: string description: >- Switch the latency and accuracy preset mid-session. See the `mode` connection parameter for the behavior of each preset. Universal-3.5 Pro Streaming only. enumValues: - max_accuracy - min_latency - balanced required: false - name: end_of_turn_confidence_threshold type: number description: >- The confidence threshold (0.0 to 1.0) to use when determining if the end of a turn has been reached. Universal Streaming (English and Multilingual) only. See [Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details. required: false - name: language_codes type: array description: >- Update the set of languages transcription is steered toward. Takes effect from the next turn. See the `language_codes` connection parameter for accepted codes and behavior. Pass an empty list to clear steering and return to native code-switching across all supported languages. Universal-3.5 Pro Streaming only. required: false properties: - name: item type: string required: false headers: [] jsonPayloadSchema: type: object properties: type: type: string const: UpdateConfiguration x-parser-schema-id: prompt: type: string description: >- A contextual prompt describing what the audio is about, such as its domain, scenario, or full conversation details. The model uses this context to better recognize the vocabulary it makes likely. It carries context about your audio, not transcription instructions, so formatting or behavioral commands are not supported. Maximum 1750 characters. See [Prompting and keyterms](https://www.assemblyai.com/docs/streaming/prompting-and-keyterms) for details. x-parser-schema-id: keyterms_prompt: type: array items: type: string x-parser-schema-id: description: >- A list of words and phrases to boost recognition for. See [Keyterms Prompting](https://www.assemblyai.com/docs/streaming/keyterms-prompting) for more details. x-parser-schema-id: min_turn_silence: type: integer description: >- Silence duration in milliseconds before a speculative end-of-turn check. See [Configuring Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details. x-parser-schema-id: max_turn_silence: type: integer description: >- Maximum silence in milliseconds before the turn is forced to end, regardless of punctuation. See [Configuring Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details. x-parser-schema-id: continuous_partials: type: boolean description: >- Whether to emit additional partial transcripts during long turns at a steady ~3 second cadence. When enabled (default), additional partials covering the full turn transcript are emitted approximately every 3 seconds while speech continues. When disabled, only one early partial is emitted near turn start. The first partial (timed by `interruption_delay`) is unaffected. x-parser-schema-id: vad_threshold: type: number description: >- The confidence threshold (0.0 to 1.0) for classifying audio frames as silence. Frames with VAD confidence below this value are considered silent. Increase for noisy environments to reduce false speech detection. x-parser-schema-id: interruption_delay: type: integer minimum: 0 maximum: 1000 description: >- How soon the first partial is emitted in milliseconds. Useful for tuning voice agent barge-in responsiveness or allowing earlier partials for early LLM inference. Larger values are more confident on interruptions, smaller values result in faster time to first partial. x-parser-schema-id: agent_context: type: string description: >- Your voice agent's most recent spoken reply (TTS text). The model uses this as context for the next user turn, which improves accuracy on short or ambiguous replies and on spelled-out entities like emails or IDs. Each `UpdateConfiguration` replaces the previously set value. Maximum 1750 characters. See [Context Carryover](https://www.assemblyai.com/docs/streaming/universal-3-pro/context-carryover#passing-your-agents-reply-as-context). x-parser-schema-id: mode: type: string enum: - max_accuracy - min_latency - balanced description: >- Switch the latency and accuracy preset mid-session. See the `mode` connection parameter for the behavior of each preset. Universal-3.5 Pro Streaming only. x-parser-schema-id: end_of_turn_confidence_threshold: type: number description: >- The confidence threshold (0.0 to 1.0) to use when determining if the end of a turn has been reached. Universal Streaming (English and Multilingual) only. See [Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details. x-parser-schema-id: language_codes: type: array items: type: string x-parser-schema-id: description: >- Update the set of languages transcription is steered toward. Takes effect from the next turn. See the `language_codes` connection parameter for accepted codes and behavior. Pass an empty list to clear steering and return to native code-switching across all supported languages. Universal-3.5 Pro Streaming only. x-parser-schema-id: required: - type x-parser-schema-id: UpdateConfigurationPayload title: Update Streaming Configuration description: >- Client message to update streaming configuration parameters during an active session. example: |- { "type": "UpdateConfiguration", "prompt": "Transcribe product names accurately.", "keyterms_prompt": [ "AssemblyAI", "Krabby Patty" ], "min_turn_silence": 700, "max_turn_silence": 1600, "agent_context": "Sure — what date would you like to book?" } bindings: [] extensions: - id: x-parser-unique-object-id value: updateConfiguration bindings: [] extensions: *ref_0 - &ref_3 id: forceEndpoint title: Force endpoint description: Manually force an endpoint in the transcription. type: receive messages: - &ref_14 id: forceEndpoint contentType: application/json payload: - name: Force Endpoint description: Client message to manually force an endpoint in the transcription. type: object properties: - name: type type: string description: ForceEndpoint required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: ForceEndpoint x-parser-schema-id: required: - type x-parser-schema-id: ForceEndpointPayload title: Force Endpoint description: Client message to manually force an endpoint in the transcription. example: |- { "type": "ForceEndpoint" } bindings: [] extensions: - id: x-parser-unique-object-id value: forceEndpoint bindings: [] extensions: *ref_0 - &ref_4 id: terminateSession title: Terminate session description: Gracefully terminate the streaming session. type: receive messages: - &ref_15 id: sessionTermination contentType: application/json payload: - name: Terminate Session (Client Initiated) description: Client message to gracefully terminate the streaming session. type: object properties: - name: type type: string description: Terminate required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: Terminate x-parser-schema-id: required: - type x-parser-schema-id: SessionTerminationPayload title: Terminate Session (Client Initiated) description: Client message to gracefully terminate the streaming session. example: |- { "type": "" } bindings: [] extensions: - id: x-parser-unique-object-id value: sessionTermination bindings: [] extensions: *ref_0 - &ref_5 id: keepAlive title: Keep alive description: >- Send a keep-alive message to reset the `inactivity_timeout` timer. This is not necessary by default — sessions remain open until explicitly terminated or until the 3-hour maximum session duration is reached. This message is only needed if you have set `inactivity_timeout` and want to keep the session open during periods where no audio is being sent. type: receive messages: - &ref_16 id: keepAlive contentType: application/json payload: - name: Keep Alive description: >- Client message to reset the inactivity timeout timer. This is not necessary by default — sessions remain open until explicitly terminated or until the 3-hour maximum session duration is reached. This message is only needed if you have set `inactivity_timeout` and want to keep the session open during periods where no audio is being sent. type: object properties: - name: type type: string description: KeepAlive required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: KeepAlive x-parser-schema-id: required: - type x-parser-schema-id: KeepAlivePayload title: Keep Alive description: >- Client message to reset the inactivity timeout timer. This is not necessary by default — sessions remain open until explicitly terminated or until the 3-hour maximum session duration is reached. This message is only needed if you have set `inactivity_timeout` and want to keep the session open during periods where no audio is being sent. example: |- { "type": "KeepAlive" } bindings: [] extensions: - id: x-parser-unique-object-id value: keepAlive bindings: [] extensions: *ref_0 - &ref_6 id: sessionBegins title: Session begins description: Receive confirmation that the streaming session has successfully started. type: send messages: - &ref_17 id: sessionBegins contentType: application/json payload: - name: Session Begins Confirmation description: >- Server message indicating the streaming session has successfully started. type: object properties: - name: type type: string description: Identifies the type of the message. required: true - name: id type: string description: Unique identifier for the streaming session. required: true - name: expires_at type: integer description: Unix timestamp indicating when the session will expire. required: true - name: configuration type: object description: >- Echo of the configuration the server applied to the session. Always check that `model` matches the `speech_model` you requested — unrecognized or misspelled query parameters are ignored rather than rejected. Optional settings that were not set are null. required: false properties: - name: model type: string description: The speech model applied to the session. required: false - name: mode type: string description: >- The latency/accuracy preset in effect. Universal-3.5 Pro Streaming only; null otherwise. required: false - name: api_version type: string description: >- The API version in effect for the session (see the `AssemblyAI-Version` header). required: false - name: speaker_labels type: boolean description: Whether speaker diarization is enabled. required: false - name: redact_pii type: boolean description: Whether PII redaction is enabled. required: false - name: filter_profanity type: boolean description: Whether profanity filtering is enabled. required: false - name: domain type: string description: >- The domain specialization in effect (for example `medical-v1`), or null. required: false - name: voice_focus type: string description: >- The Voice Focus model in effect (`near-field` or `far-field`), or null. required: false headers: [] jsonPayloadSchema: type: object properties: type: type: string const: Begin description: Identifies the type of the message. x-parser-schema-id: id: type: string format: uuid description: Unique identifier for the streaming session. x-parser-schema-id: expires_at: type: integer description: Unix timestamp indicating when the session will expire. x-parser-schema-id: configuration: type: object description: >- Echo of the configuration the server applied to the session. Always check that `model` matches the `speech_model` you requested — unrecognized or misspelled query parameters are ignored rather than rejected. Optional settings that were not set are null. properties: model: type: string description: The speech model applied to the session. x-parser-schema-id: mode: type: string nullable: true description: >- The latency/accuracy preset in effect. Universal-3.5 Pro Streaming only; null otherwise. x-parser-schema-id: api_version: type: string description: >- The API version in effect for the session (see the `AssemblyAI-Version` header). x-parser-schema-id: speaker_labels: type: boolean description: Whether speaker diarization is enabled. x-parser-schema-id: redact_pii: type: boolean description: Whether PII redaction is enabled. x-parser-schema-id: filter_profanity: type: boolean description: Whether profanity filtering is enabled. x-parser-schema-id: domain: type: string nullable: true description: >- The domain specialization in effect (for example `medical-v1`), or null. x-parser-schema-id: voice_focus: type: string nullable: true description: >- The Voice Focus model in effect (`near-field` or `far-field`), or null. x-parser-schema-id: x-parser-schema-id: required: - type - id - expires_at x-parser-schema-id: SessionBeginsPayload title: Session Begins Confirmation description: >- Server message indicating the streaming session has successfully started. example: |- { "type": "Begin", "id": "3207b601-2054-48df-ba77-8784dfcf9fb8", "expires_at": 1772570132, "configuration": { "model": "universal-3-5-pro", "mode": "balanced", "api_version": "2025-05-12", "speaker_labels": false, "redact_pii": false, "filter_profanity": false, "domain": null, "voice_focus": null } } bindings: [] extensions: - id: x-parser-unique-object-id value: sessionBegins bindings: [] extensions: *ref_0 - &ref_7 id: speechStarted title: Speech started description: >- Receive a notification that speech has been detected. This event is only emitted when the model produces a transcript. Every `SpeechStarted` is guaranteed to be followed by one or more Turn messages. type: send messages: - &ref_18 id: speechStarted contentType: application/json payload: - name: Speech Started description: Server message indicating that speech has been detected. type: object properties: - name: type type: string description: Identifies the type of the message. required: true - name: timestamp type: integer description: >- The timestamp in milliseconds when speech was detected, relative to the beginning of the audio stream. required: true - name: confidence type: number description: The confidence score that speech has started. required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: SpeechStarted description: Identifies the type of the message. x-parser-schema-id: timestamp: type: integer description: >- The timestamp in milliseconds when speech was detected, relative to the beginning of the audio stream. format: ms x-parser-schema-id: confidence: type: number format: float description: The confidence score that speech has started. x-parser-schema-id: required: - type - timestamp - confidence x-parser-schema-id: SpeechStartedPayload title: Speech Started description: Server message indicating that speech has been detected. example: |- { "type": "", "timestamp": 123, "confidence": 123 } bindings: [] extensions: - id: x-parser-unique-object-id value: speechStarted bindings: [] extensions: *ref_0 - &ref_8 id: turn title: Turn description: Receive a turn-based transcription result (partial or final). type: send messages: - &ref_19 id: turn contentType: application/json payload: - name: Turn Transcript description: Server message containing a turn-based transcription result. type: object properties: - name: type type: string description: Turn required: true - name: turn_order type: integer description: Order of this turn in the conversation. required: true - name: turn_is_formatted type: boolean description: >- Whether this turn has been formatted. For Universal-3.5 Pro Streaming, this always matches `end_of_turn`. required: true - name: end_of_turn type: boolean description: >- Whether this marks the end of a turn. See [Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for more information. required: true - name: transcript type: string description: Transcript of all finalized words in the turn. required: true - name: utterance type: string description: >- Finalized transcript of the turn, populated only on end_of_turn messages. Empty string on all other Turn messages. Equivalent to transcript when populated. required: false - name: language_code type: string description: >- The language of the turn. Only populated when language detection is enabled and an utterance is complete or turn is final. required: false - name: language_confidence type: number description: >- The confidence score for the detected language, between 0 (low confidence) and 1 (high confidence). Only populated when language detection is enabled and an utterance is complete or turn is final. required: false - name: speaker_label type: string description: >- The speaker label for this turn (e.g. `A`, `B`). Only present when `speaker_labels` is enabled. Short turns with less than approximately 1 second of audio will have the label `UNKNOWN`. See [Streaming Diarization](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels) for more details. required: false - name: end_of_turn_confidence type: number description: >- The confidence score that this is the end of a turn, between 0.0 (low confidence) and 1.0 (high confidence). For Universal-3.5 Pro Streaming, this is 1.0 when `end_of_turn` is true and 0.0 otherwise. required: true - name: words type: array description: Array of word-level details for this turn. required: true properties: - name: text type: string description: The transcribed word. required: true - name: start type: integer description: >- Start time in milliseconds relative to the beginning of the audio stream. required: true - name: end type: integer description: >- End time in milliseconds relative to the beginning of the audio stream. required: true - name: confidence type: number description: Confidence score for the word (0.0 to 1.0). required: true - name: word_is_final type: boolean description: Whether the word is final. required: true - name: speaker type: string description: >- The speaker label for this word (e.g. `A`, `B`, `UNKNOWN`). Only present on final words (`word_is_final` is `true`) when `speaker_labels` is enabled. May be absent on individual words even when diarization is enabled — treat absent as unattributed and fall back to the turn-level `speaker_label` if needed. See [Streaming Diarization](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels) for more details. required: false headers: [] jsonPayloadSchema: type: object properties: type: type: string const: Turn x-parser-schema-id: turn_order: type: integer description: Order of this turn in the conversation. x-parser-schema-id: turn_is_formatted: type: boolean description: >- Whether this turn has been formatted. For Universal-3.5 Pro Streaming, this always matches `end_of_turn`. x-parser-schema-id: end_of_turn: type: boolean description: >- Whether this marks the end of a turn. See [Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for more information. x-parser-schema-id: transcript: type: string description: Transcript of all finalized words in the turn. x-parser-schema-id: utterance: type: string description: >- Finalized transcript of the turn, populated only on end_of_turn messages. Empty string on all other Turn messages. Equivalent to transcript when populated. x-parser-schema-id: language_code: type: string description: >- The language of the turn. Only populated when language detection is enabled and an utterance is complete or turn is final. x-parser-schema-id: language_confidence: type: number format: float minimum: 0 maximum: 1 description: >- The confidence score for the detected language, between 0 (low confidence) and 1 (high confidence). Only populated when language detection is enabled and an utterance is complete or turn is final. x-parser-schema-id: speaker_label: type: string description: >- The speaker label for this turn (e.g. `A`, `B`). Only present when `speaker_labels` is enabled. Short turns with less than approximately 1 second of audio will have the label `UNKNOWN`. See [Streaming Diarization](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels) for more details. x-parser-schema-id: end_of_turn_confidence: type: number format: float minimum: 0 maximum: 1 description: >- The confidence score that this is the end of a turn, between 0.0 (low confidence) and 1.0 (high confidence). For Universal-3.5 Pro Streaming, this is 1.0 when `end_of_turn` is true and 0.0 otherwise. x-parser-schema-id: words: type: array items: type: object properties: text: type: string description: The transcribed word. x-parser-schema-id: start: type: integer description: >- Start time in milliseconds relative to the beginning of the audio stream. format: ms x-parser-schema-id: end: type: integer description: >- End time in milliseconds relative to the beginning of the audio stream. format: ms x-parser-schema-id: confidence: type: number format: float minimum: 0 maximum: 1 description: Confidence score for the word (0.0 to 1.0). x-parser-schema-id: word_is_final: type: boolean description: Whether the word is final. x-parser-schema-id: speaker: type: string description: >- The speaker label for this word (e.g. `A`, `B`, `UNKNOWN`). Only present on final words (`word_is_final` is `true`) when `speaker_labels` is enabled. May be absent on individual words even when diarization is enabled — treat absent as unattributed and fall back to the turn-level `speaker_label` if needed. See [Streaming Diarization](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels) for more details. x-parser-schema-id: required: - text - start - end - confidence - word_is_final x-parser-schema-id: Word description: Array of word-level details for this turn. x-parser-schema-id: required: - type - turn_order - turn_is_formatted - end_of_turn - transcript - end_of_turn_confidence - words x-parser-schema-id: TurnPayload title: Turn Transcript description: Server message containing a turn-based transcription result. example: |- { "type": "Turn", "turn_order": 0, "turn_is_formatted": true, "end_of_turn": true, "transcript": "Hello world.", "end_of_turn_confidence": 1, "words": [ { "text": "Hello", "start": 0, "end": 500, "confidence": 0.99 }, { "text": "world.", "start": 500, "end": 1000, "confidence": 0.98 } ] } bindings: [] extensions: - id: x-parser-unique-object-id value: turn bindings: [] extensions: *ref_0 - &ref_9 id: speakerRevision title: Speaker revision description: >- Receive revised speaker labels at the end of a session. Emitted as a single message immediately before `Termination` when `speaker_labels` is enabled. Contains a `revisions` array with only the turns whose labels changed — unchanged turns are omitted. Text and word timestamps are never changed — only speaker assignments. Adds approximately 400ms of latency at session close. See [Revised speaker labels](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels#revised-speaker-labels) for details. type: send messages: - &ref_20 id: speakerRevision contentType: application/json payload: - name: Revised Speaker Labels description: >- Server message containing corrected speaker labels for any turns that changed. Emitted as a single message after the client sends `Terminate`, when `speaker_labels` is enabled. A session may produce zero or one `SpeakerRevision` message; if sent, only changed turns are included in the `revisions` array. See [Revised speaker labels](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels#revised-speaker-labels) for details. type: object properties: - name: type type: string description: Identifies the type of the message. required: true - name: revisions type: array description: >- Array of turn corrections. Only turns whose speaker labels changed are included. required: true properties: - name: turn_order type: integer description: >- Matches the `turn_order` of the original Turn message being corrected. required: true - name: speaker_label type: string description: >- The corrected turn-level speaker label (e.g. `A`, `B`, `UNKNOWN`). Replaces the `speaker_label` originally delivered on the matching Turn message. required: true - name: words type: array description: >- Words with corrected per-word `speaker` assignments. Text and timestamps match the original Turn — only the `speaker` field may change. required: true properties: - name: text type: string description: The word text (unchanged from the original Turn). required: true - name: speaker type: string description: >- The revised speaker label for this word (e.g. `A`, `B`, `UNKNOWN`). required: true - name: start type: integer description: >- Start time of the word in milliseconds from the start of the session. required: true - name: end type: integer description: >- End time of the word in milliseconds from the start of the session. required: true headers: [] jsonPayloadSchema: type: object description: >- Sent at the end of a streaming session when `speaker_labels` is enabled. Contains corrections for any turns whose speaker labels changed. Text content and word timestamps are never changed — only speaker assignments. properties: type: type: string const: SpeakerRevision description: Identifies the type of the message. x-parser-schema-id: revisions: type: array description: >- Array of turn corrections. Only turns whose speaker labels changed are included. items: type: object properties: turn_order: type: integer description: >- Matches the `turn_order` of the original Turn message being corrected. x-parser-schema-id: speaker_label: type: string nullable: true description: >- The corrected turn-level speaker label (e.g. `A`, `B`, `UNKNOWN`). Replaces the `speaker_label` originally delivered on the matching Turn message. x-parser-schema-id: words: type: array description: >- Words with corrected per-word `speaker` assignments. Text and timestamps match the original Turn — only the `speaker` field may change. items: type: object properties: text: type: string description: The word text (unchanged from the original Turn). x-parser-schema-id: speaker: type: string description: >- The revised speaker label for this word (e.g. `A`, `B`, `UNKNOWN`). x-parser-schema-id: start: type: integer description: >- Start time of the word in milliseconds from the start of the session. x-parser-schema-id: end: type: integer description: >- End time of the word in milliseconds from the start of the session. x-parser-schema-id: required: - text - speaker - start - end x-parser-schema-id: x-parser-schema-id: required: - turn_order - speaker_label - words x-parser-schema-id: x-parser-schema-id: required: - type - revisions x-parser-schema-id: SpeakerRevisionPayload title: Revised Speaker Labels description: >- Server message containing corrected speaker labels for any turns that changed. Emitted as a single message after the client sends `Terminate`, when `speaker_labels` is enabled. A session may produce zero or one `SpeakerRevision` message; if sent, only changed turns are included in the `revisions` array. See [Revised speaker labels](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels#revised-speaker-labels) for details. example: |- { "type": "SpeakerRevision", "revisions": [ { "turn_order": 3, "speaker_label": "B", "words": [ { "text": "Hello", "speaker": "B", "start": 1200, "end": 1450 }, { "text": "there.", "speaker": "B", "start": 1450, "end": 1780 } ] }, { "turn_order": 7, "speaker_label": "A", "words": [ { "text": "Got it.", "speaker": "A", "start": 4100, "end": 4520 } ] } ] } bindings: [] extensions: - id: x-parser-unique-object-id value: speakerRevision bindings: [] extensions: *ref_0 - &ref_10 id: sessionTermination title: Session termination description: Receive confirmation that the session has been terminated by the server. type: send messages: - &ref_21 id: termination contentType: application/json payload: - name: Session Terminated (Server Confirmation) description: >- Server message confirming session termination with session statistics. type: object properties: - name: type type: string description: Indicates the session has been terminated. required: true - name: audio_duration_seconds type: integer description: Duration of the audio in seconds. required: true - name: session_duration_seconds type: integer description: Duration of the session in seconds. required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: Termination description: Indicates the session has been terminated. x-parser-schema-id: audio_duration_seconds: type: integer description: Duration of the audio in seconds. x-parser-schema-id: session_duration_seconds: type: integer description: Duration of the session in seconds. x-parser-schema-id: required: - type - audio_duration_seconds - session_duration_seconds x-parser-schema-id: TerminationPayload title: Session Terminated (Server Confirmation) description: Server message confirming session termination with session statistics. example: |- { "type": "", "audio_duration_seconds": 123, "session_duration_seconds": 123 } bindings: [] extensions: - id: x-parser-unique-object-id value: termination bindings: [] extensions: *ref_0 - &ref_11 id: llmGatewayResponse title: Llm gateway response description: >- Receive an LLM Gateway response for a finalized turn. Emitted once per turn when `llm_gateway` is configured on the connection. type: send messages: - &ref_22 id: llmGatewayResponse contentType: application/json payload: - name: LLM Gateway Response description: >- Server message containing an LLM Gateway response for a finalized turn. type: object properties: - name: type type: string description: Identifies the type of the message. required: true - name: turn_order type: integer description: >- The order of the finalized turn that triggered the LLM Gateway call. required: true - name: transcript type: string description: >- The finalized turn transcript that triggered the LLM Gateway call. required: true - name: data type: object description: The chat completions response from the LLM Gateway. required: true properties: - name: request_id type: string description: Unique identifier for the LLM Gateway request. required: false - name: choices type: array description: Array of completion choices returned by the model. required: false properties: - name: index type: integer description: The index of the choice in the response. required: false - name: message type: object required: false properties: - name: role type: string required: false - name: content type: string description: >- The text content of the model's response. Null when only tool_calls are present. required: false - name: tool_calls type: array description: >- Tool calls requested by the model. Present when the model invokes tools. required: false properties: - name: id type: string description: Unique identifier for the tool call. required: true - name: type type: string enumValues: - function required: true - name: function type: object required: true properties: - name: name type: string description: The name of the function to call. required: true - name: arguments type: string description: >- The arguments to call the function with, as a JSON-formatted string. required: true - name: finish_reason type: string description: The reason the model stopped generating tokens. required: false - name: usage type: object description: Token usage statistics for the request. required: false properties: - name: input_tokens type: integer description: Number of tokens in the prompt. required: false - name: output_tokens type: integer description: Number of tokens in the completion. required: false - name: total_tokens type: integer description: Total tokens used (prompt + completion). required: false - name: prompt_tokens_details type: object description: Detailed breakdown of prompt token usage. required: false properties: [] - name: completion_tokens_details type: object description: Detailed breakdown of completion token usage. required: false properties: [] - name: request type: object description: >- A copy of the original request, excluding `prompt` and `messages`. required: false properties: [] - name: response_time type: integer description: The response time in nanoseconds. required: false - name: llm_status_code type: integer description: The status code from the LLM provider. required: false headers: [] jsonPayloadSchema: type: object properties: type: type: string const: LLMGatewayResponse description: Identifies the type of the message. x-parser-schema-id: turn_order: type: integer description: >- The order of the finalized turn that triggered the LLM Gateway call. x-parser-schema-id: transcript: type: string description: >- The finalized turn transcript that triggered the LLM Gateway call. x-parser-schema-id: data: type: object description: The chat completions response from the LLM Gateway. required: [] properties: request_id: type: string format: uuid description: Unique identifier for the LLM Gateway request. x-parser-schema-id: choices: type: array items: type: object required: [] properties: index: type: integer description: The index of the choice in the response. x-parser-schema-id: message: type: object required: [] properties: role: type: string x-parser-schema-id: content: type: string description: >- The text content of the model's response. Null when only tool_calls are present. x-parser-schema-id: tool_calls: type: array items: type: object properties: id: type: string description: Unique identifier for the tool call. x-parser-schema-id: type: type: string enum: - function x-parser-schema-id: function: type: object properties: name: type: string description: The name of the function to call. x-parser-schema-id: arguments: type: string description: >- The arguments to call the function with, as a JSON-formatted string. x-parser-schema-id: required: - name - arguments x-parser-schema-id: required: - id - type - function x-parser-schema-id: description: >- Tool calls requested by the model. Present when the model invokes tools. x-parser-schema-id: x-parser-schema-id: finish_reason: type: string description: The reason the model stopped generating tokens. x-parser-schema-id: x-parser-schema-id: description: Array of completion choices returned by the model. x-parser-schema-id: usage: type: object description: Token usage statistics for the request. required: [] properties: input_tokens: type: integer description: Number of tokens in the prompt. x-parser-schema-id: output_tokens: type: integer description: Number of tokens in the completion. x-parser-schema-id: total_tokens: type: integer description: Total tokens used (prompt + completion). x-parser-schema-id: prompt_tokens_details: type: object additionalProperties: true required: [] properties: {} description: Detailed breakdown of prompt token usage. x-parser-schema-id: completion_tokens_details: type: object additionalProperties: true required: [] properties: {} description: Detailed breakdown of completion token usage. x-parser-schema-id: x-parser-schema-id: request: type: object additionalProperties: true required: [] properties: {} description: >- A copy of the original request, excluding `prompt` and `messages`. x-parser-schema-id: response_time: type: integer description: The response time in nanoseconds. x-parser-schema-id: llm_status_code: type: integer description: The status code from the LLM provider. x-parser-schema-id: x-parser-schema-id: required: - type - turn_order - transcript - data x-parser-schema-id: LLMGatewayResponsePayload title: LLM Gateway Response description: >- Server message containing an LLM Gateway response for a finalized turn. example: |- { "type": "LLMGatewayResponse", "turn_order": 0, "transcript": "Hello world.", "data": { "request_id": "3c90c3cc-0d44-4b50-8888-8dd25736052a", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Hello! How can I help?" }, "finish_reason": "stop" } ], "usage": { "input_tokens": 12, "output_tokens": 8, "total_tokens": 20, "prompt_tokens_details": {}, "completion_tokens_details": {} }, "request": {}, "response_time": 123456789 } } bindings: [] extensions: - id: x-parser-unique-object-id value: llmGatewayResponse bindings: [] extensions: *ref_0 sendOperations: - *ref_1 - *ref_2 - *ref_3 - *ref_4 - *ref_5 receiveOperations: - *ref_6 - *ref_7 - *ref_8 - *ref_9 - *ref_10 - *ref_11 sendMessages: - *ref_12 - *ref_13 - *ref_14 - *ref_15 - *ref_16 receiveMessages: - *ref_17 - *ref_18 - *ref_19 - *ref_20 - *ref_21 - *ref_22 extensions: - id: x-parser-unique-object-id value: streaming securitySchemes: [] ````