> ## Documentation Index > Fetch the complete documentation index at: https://assemblyai.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Message frames > Every frame in the Streaming TTS protocol, in both directions, with the order they arrive in. Every message in both directions is a JSON text frame with a `type` field naming the frame. ## Frame rules These rules apply to everything you send: * Send text frames only, each one a JSON object with a `type` field. A binary frame, text that is not a JSON object, a missing or unknown `type`, or a required field that is missing or of the wrong type (for example a `Generate` without `text`) ends the session: the server sends an `Error` frame with code `3006` and closes. * Extra fields you add to a known frame are ignored. * A WebSocket message over 1 MiB is refused by the WebSocket layer itself, with close code `1009` and no `Error` frame. The size counts the whole message, so a message sent in several fragments is measured once they are joined. * The server does not negotiate WebSocket compression (`permessage-deflate`), so frames in both directions travel uncompressed. And one rule for everything you receive: ignore frame types you do not recognize. The protocol may add frame types. There is no warning frame: everything the server tells you about the session is in `Begin`, in the frames below, or in the `Error` frame that ends the session. ## Client to server | Frame | Fields | Description | | - | - | - | | `Generate` | `type`, `text` | Appends text to the session buffer. Short text is silent until you `Flush`; once enough complete sentences are buffered the server starts speaking them before any `Flush` (see [Generate](#generate)) | | `Flush` | `type` | Ends the current request: synthesizes whatever is still unspoken and concludes it with `FlushDone` | | `Cancel` | `type` | Barge-in. Stops the current synthesis and discards any text you have sent but not yet heard, including pending flushes | | `KeepAlive` | `type` | Restarts the two-minute idle timer | | `Terminate` | `type` | Ends the session cleanly | ### Generate ```json theme={null} { "type": "Generate", "text": "Hi, thanks for calling." } ``` `Generate` appends text to the session buffer. The server does not always wait for `Flush`: once the buffer holds enough complete sentences, it starts speaking them under the current `flush_id`. * **Text that arrives in one `Generate`** starts once its complete sentences add up to a long enough stretch of text. * **Text streamed over several `Generate` frames**, as an LLM produces it, starts sooner: once its first sentences are complete, or at a clause end (`,`, `;` or `:`) when a long sentence has none yet. The rest of the request follows in parts that start early enough that the audio already sent does not run out while text is still arriving. These rules apply from the start of the session and again after every `Flush` and `Cancel`. A sentence ends once whitespace follows its `.`, `!` or `?`, so the end of your text so far does not count until the next text, or a trailing space, arrives; a period after an abbreviation or an initial, as in `Dr.` or `J.`, does not end a sentence. A sentence of one word, such as `Sure.` or a list number such as `4.`, is never spoken on its own before `Flush`; it is read with the sentence after it. How much text the server gathers before it starts speaking can change, so do not rely on it: `Flush` is what concludes a request. Short text is buffered, not spoken. A session that sends a short `Generate` and never flushes therefore stays silent, looking perfectly healthy, until the [idle timeout](https://assemblyai.com/docs/tts/connect-parameters.md#inactivity-timeout) closes it with `3006` two minutes later, and the leftover text is never spoken. Only `Flush` (or `Terminate`) guarantees that everything you sent is spoken and gives you the `FlushDone` boundary. Text from consecutive `Generate` frames is joined exactly as sent, with no separator added. `Good` followed by `morning.` is spoken, and reported in `WordBoundaries`, as the single word `Goodmorning.`. Include the space or newline yourself when you split text across frames. A single `Generate` accepts up to 2000 characters, counted as Unicode code points. Over that, the session closes with `3006`. Empty text is accepted and does nothing, but it counts toward the control-frame limit of 20 per second, like `Flush`, `KeepAlive` and `Cancel`. ### Flush ```json theme={null} { "type": "Flush" } ``` `Flush` concludes the current request: whatever is still unspoken is synthesized, then a `FlushDone` arrives. For short text this is when audio starts. For longer text the server may already be streaming, and `Flush` synthesizes the remainder. `FlushDone` means the request's audio has been fully delivered to your socket, not that it has finished playing. Synthesis runs several times faster than real time, so `FlushDone` usually lands well before playback ends. A `Flush` with nothing left to speak still returns exactly one `FlushDone`, with `audio_duration_ms` of `0` and no `Audio` frames. The `flush_id` still advances and the session stays usable. You do not have to wait for one `FlushDone` before sending the next `Flush`. Queued requests are answered in `flush_id` order. The server works through them one at a time, so each request you queue adds to the wait before its own audio starts. When you are streaming text out of an LLM, flush on sentence or clause boundaries. Flushing per token produces choppy prosody, because each fragment is synthesized on its own and complete sentences give the model the context it needs. It also costs you the session: `Flush` is a control frame, capped at 20 per second sustained with a burst allowance of about 60, and a per-token loop quickly exceeds that and is closed with `3010`. See [Limits](https://assemblyai.com/docs/tts/limits.md). ### Cancel ```json theme={null} { "type": "Cancel" } ``` `Cancel` is for barge-in, when the listener starts talking over the speech. It stops the in-flight synthesis and discards everything you have sent but not yet heard: text sent with `Generate` but not yet flushed, and every `Flush` still waiting its turn. None of them gets a `FlushDone`. The server answers with a `Cancelled` frame: ```json theme={null} { "type": "Cancelled", "audio_duration_seconds": 0.4 } ``` `audio_duration_seconds` is the audio that was streamed for the cancelled request only, in seconds rounded to 3 decimal places, and `0.0` when nothing was in flight. A `Cancel` with nothing in flight is still answered, and still advances `flush_id`. Once the session has reached `expires_at` the server reads no more frames, so a `Cancel` sent then gets no `Cancelled`: the request in flight streams to its `FlushDone`, and the session closes with `3008`. Stop playback yourself in that case. Three things to design around: * **A short tail of the old audio may still arrive.** `Audio` frames already on the way when you sent `Cancel` can land before `Cancelled` does. Every `Audio` frame between your `Cancel` and its `Cancelled` belongs to a request you sent before the `Cancel`: drop it, and stop the buffers you have already scheduled for playback, or they keep playing. Each `Cancel` is answered by its own `Cancelled`, in the order sent. * **The next request takes a new `flush_id`.** It is higher than every id before the `Cancel`, but it can skip numbers, so read it from the first `Audio` or `FlushDone` after `Cancelled` rather than computing it. The next `Generate` and `Flush` continue under that id. The server handles your frames in the order you send them, so a `Generate` sent after `Cancel` is not caught by the cancel; you do not have to wait for `Cancelled` before sending the next reply. * **Re-send anything you still want spoken.** Unflushed text is gone; the next `Flush` speaks only what you send after the cancel. The cancelled request gets no `WordBoundaries`. The session totals in `Termination` still count the cancelled text that had been passed to synthesis and the audio that was delivered, and the output timeline used by `WordBoundaries` counts that audio too, the tail delivered between your `Cancel` and the server's `Cancelled` included. ### KeepAlive ```json theme={null} { "type": "KeepAlive" } ``` Restarts the [idle timer](https://assemblyai.com/docs/tts/connect-parameters.md#inactivity-timeout) and does nothing else. A session closes after two minutes with no frame in either direction, or after the shorter `inactivity_timeout` you set, and any frame restarts that timer, so you only need `KeepAlive` while a session is idle: send one at least once per timeout to hold the socket open. It is a control frame and counts against the same 20 per second cap as `Flush` and `Cancel`. ### Terminate ```json theme={null} { "type": "Terminate" } ``` Ends the session cleanly, in this order: 1. Text you sent with `Generate` but never flushed is flushed for you and gets its own `FlushDone` with the next `flush_id`. 2. Audio still in flight is delivered, and any pending `WordBoundaries` arrive. 3. A `Termination` frame carries the session summary, then the socket closes with code `1000` and reason `Session ended`. Anything you send after `Terminate` is never read: no error, no audio, and it is not counted in the summary. If the server produces nothing for 30 seconds while a `FlushDone` is still owed, the session closes with `3005` instead of `Termination`. That window restarts whenever the server produces output, so a long final request is never cut short. You can also just close the socket. The server then discards any synthesis still in flight and sends no `Termination`; the audio it already delivered to you still counts. ## Server to client | Frame | Fields | Description | | - | - | - | | `Begin` | `type`, `id`, `expires_at`, `configuration` | Session opened. The authoritative configuration | | `Audio` | `type`, `audio`, `flush_id` | A chunk of base64 audio | | `FlushDone` | `type`, `flush_id`, `audio_duration_ms` | That request's audio has all been sent | | `WordBoundaries` | `type`, `flush_id`, `audio_start_ms`, `words` | Per-word timings for that request, best-effort. `words` is a list of `{word, start, end, confidence}` | | `Cancelled` | `type`, `audio_duration_seconds` | Answers `Cancel` | | `Termination` | `type`, `session_duration_seconds`, `total_input_char_length`, `audio_duration_seconds`, `voice_expires_at` | Session summary, on a clean end only | | `Error` | `type`, `error_code`, `error`, `error_type`, `retryable`, `param`, `retry_after_seconds` | Sent before every error close. `error_code` equals the close code that follows | ### Begin ```json theme={null} { "type": "Begin", "id": "10885144-0d4b-4928-9eff-ef292a352937", "expires_at": 1789556741, "configuration": { "language": "english", "sample_rate": 24000, "encoding": "pcm_s16le", "inactivity_timeout": 120, "word_boundaries": true, "voice_id": "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", "extend_biometric_consent": false, "word_boundaries_requested": true, "model": "assemblyai-tts-1", "session_ttl_sec": 3600, "limits": { "max_generate_text_length": 2000, "max_control_frames_per_sec": 20.0, "max_egress_audio_buffer_ms": 300000, "text_throttle_enabled": true, "text_throttle_max_chars_per_sec": 100.0, "text_throttle_initial_grace_chars": 1500, "text_max_buffer_chars": 8000, "session_ttl_grace_sec": 60, "max_message_bytes": 1048576, "max_queued_messages": 10000, "write_stall_timeout_sec": 30.0, "inactivity_timeout_min_sec": 5, "inactivity_timeout_max_sec": 120 }, "unknown_params": [] } } ``` `Begin` is always the first frame, sent before the server reads anything from you. It arrives only after the server confirms it has room for the connection, your credentials are checked, your parameters are validated, your account's limit of new sessions per minute is checked, and the session's language and voice are available on its model; if any of those fail you get an `Error` frame and a close instead. Read `configuration` as the session you actually got: every connection parameter except `token` and `model_id` with its effective value, whether you sent it or not, then what you requested where it differs (`word_boundaries_requested`), the model the session runs on (`model`), the session's lifetime and limits, and the names of any query parameters the server ignored (`unknown_params`). [Read the configuration back](https://assemblyai.com/docs/tts/connect-parameters.md#read-the-configuration-back) describes these keys. The example above is abridged to them; a session with a cloned voice also shows the keys listed below. `configuration` also carries keys for options these pages do not cover, and it may gain keys over time, so read the keys you need, ignore the others, and do not validate it against a closed set. Quote `id` in support requests. `expires_at` is a Unix timestamp in seconds; when it passes, the server stops accepting input, flushes and drains what it has, and closes with `3008`. `configuration` carries two more keys about the voice. With a [cloned voice](https://assemblyai.com/docs/tts/cloned-voices.md), `language` is the voice's own: | Key | Meaning | | - | - | | `voice_expires_at` | When the voice's consent period ends, as a Unix timestamp in seconds, as the session opened; `null` for an AssemblyAI voice, for another account's voice and for a voice whose consent period has no end. It is not the session's own `expires_at`. See [Consent period](https://assemblyai.com/docs/tts/cloned-voices.md#consent-period) | | `resolved_voice` | `voice_id`, the voice's UUID, also when you sent its slug, and `language`, the voice's language | ### Audio ```json theme={null} { "type": "Audio", "audio": "IwB6AOMABQHHAIMAJADw/wwA...", "flush_id": 0 } ``` Base64-encoded raw codec bytes with no header. Every `Audio` frame carries audio; the server never sends an empty one. Frame sizes are not fixed, so do not assume a chunk length: at 24 kHz they are about 80 ms each, the last frame of a request included; at other sample rates they vary, and the last frame of a request is often shorter. Audio for one request never interleaves with another's, and all of a request's `Audio` arrives before its `FlushDone`, but it can begin before you send `Flush`. See [Audio formats](https://assemblyai.com/docs/tts/connect-parameters.md#audio-formats) on WebSocket API basics. ### FlushDone ```json theme={null} { "type": "FlushDone", "flush_id": 0, "audio_duration_ms": 2320 } ``` Exactly one per `Flush`, including the implicit flush that `Terminate` performs, in `flush_id` order. It means every byte of that request's audio has been sent to you, not that it has been played. `audio_duration_ms` is `0` when the request had nothing left to synthesize. To check that you received as much as the server sent, compare `audio_duration_ms` with `bytes_received / (sample_rate * bytes_per_sample) * 1000`, where `bytes_per_sample` is 2 for `pcm_s16le` and 1 for `pcm_mulaw` and `pcm_alaw`. ### WordBoundaries ```json theme={null} { "type": "WordBoundaries", "flush_id": 0, "audio_start_ms": 0, "words": [ { "word": "Hello", "start": 265, "end": 530, "confidence": 0.4791773557662964 }, { "word": "there,", "start": 546, "end": 712, "confidence": 0.9253882169723511 }, { "word": "friend,", "start": 845, "end": 1126, "confidence": 0.8478114008903503 } ] } ``` One frame per request, sent after that request's `FlushDone` and never after `Termination`. If you have already flushed again it may arrive after the next request's first `Audio` frames, so match it to its request by `flush_id`, not by position. Each entry has `word`, `start`, `end` and `confidence`. `words` has one entry for every whitespace-delimited token of the request's text, in order, with its punctuation attached; a token with no sound of its own, such as a dash standing alone between spaces, gets a zero-length entry (`start` equal to `end`, `confidence` `0`) at the edge of its neighboring word. `start` and `end` are milliseconds on the session's continuous audio timeline, counted from the first audio the session delivered, and `audio_start_ms` is where this request's audio begins on that same timeline. To place a word within this request's own audio, subtract `audio_start_ms` from `start`; never add the two. A long request gets one frame like any other. The frame is best-effort: a cancelled request gets none, a request whose audio leaves part of its text unspoken gets none, and it can be missing when timing data is unavailable. See [Word boundaries](https://assemblyai.com/docs/tts/word-boundaries.md). ### Termination ```json theme={null} { "type": "Termination", "session_duration_seconds": 12, "total_input_char_length": 96, "audio_duration_seconds": 3.12, "voice_expires_at": null } ``` `session_duration_seconds` is a whole number of seconds, truncated. `total_input_char_length` is every character of your `Generate` frames that was passed to synthesis, whether or not it was spoken, so text you cancelled or never flushed counts too. Text a `Cancel` discarded while the server was still holding it back under [text intake pacing](https://assemblyai.com/docs/tts/limits.md#pacing) was never passed to synthesis, and does not count. `audio_duration_seconds` is the audio actually delivered to you, including the delivered part of a cancelled request, in seconds rounded to 3 decimal places. `voice_expires_at` is when the consent period of the session's [cloned voice](https://assemblyai.com/docs/tts/cloned-voices.md#consent-period) ends, as a Unix timestamp in seconds, after the session's [renewal of the voice's consent](https://assemblyai.com/docs/tts/cloned-voices.md#renew-consent), if any, and `null` otherwise. `Termination` may gain keys over time; ignore keys you do not use. `Termination` is sent only on a clean finish. If the session fails you receive an `Error` frame and a close code instead, and if you close the socket without `Terminate` no summary is sent. ### Error ```json theme={null} { "type": "Error", "error_code": 3009, "error": "Too many sessions opened for this account in the last minute (allowance 100 per minute); retry in 17 s.", "error_type": "rate_limited", "retryable": true, "param": null, "retry_after_seconds": 17 } ``` | Field | Meaning | | - | - | | `error_code` | The WebSocket close code that follows | | `error` | What went wrong and what to do, written for people. It may be reworded, so log it rather than matching on it | | `error_type` | A stable, machine-readable cause, finer than the close code: for example `invalid_parameter`, `inactivity_timeout` or `rate_limited`. See [Error types](https://assemblyai.com/docs/tts/error-handling.md#error-types) | | `retryable` | Whether reconnecting with the same parameters can succeed | | `param` | The connection parameter or message field at fault, such as `sample_rate`, `voice_id` or `language` at connect, or `Generate.text` or `type` for a frame you sent; `null` when there is none | | `retry_after_seconds` | The earliest useful retry, in seconds; `null` when there is no hint | Sent immediately before every failure close the server initiates. `error_code` equals the WebSocket close code that follows, and the close reason is normally the fixed string `See Error message for details`, so the frame is where the detail lives. If the `Error` frame could not be delivered, for example to a client that stopped reading, the close reason carries a shortened copy of its text instead. There is no `Error` frame for a clean `1000` close, nor for closes raised by the WebSocket layer itself: `1009` for an oversized message, `1011` when pings go unanswered, `1002` or `1007` for a frame that breaks the WebSocket protocol or is not valid UTF-8, or `1006` when the connection drops. See [Error handling](https://assemblyai.com/docs/tts/error-handling.md). ## Frame order A single short request through a session looks like this. Client frames are on the left, server frames on the right: ```text theme={null} <- Begin {configuration} -> Generate {text} -> Flush <- Audio {audio, flush_id: 0} <- Audio {audio, flush_id: 0} <- FlushDone {flush_id: 0, audio_duration_ms} <- WordBoundaries {flush_id: 0} (best-effort) -> Terminate <- Termination {summary} close 1000 ``` `Begin` arrives once, as the very first frame. `flush_id` starts at `0` and each request you conclude with `Flush` takes the next id, so a session that handles several requests can use it to keep their audio apart. After a [`Cancel`](#cancel) the next id can skip numbers. Two things the single-request picture hides: * **`WordBoundaries` trails its `FlushDone`.** With several requests in flight it often lands after the next request's first `Audio` frames. `Audio` frames themselves never interleave across requests. * **Audio can start before `Flush`.** Once enough complete sentences are buffered (see [Generate](#generate)), the server begins streaming them; `Flush` then completes the request and produces the `FlushDone`. You can send several `Generate` and `Flush` groups before the first `FlushDone` arrives. This is the real order for three requests flushed back to back, with runs of `Audio` frames collapsed: ```text theme={null} Begin | Audio(0) x29 | FlushDone(0) | Audio(1) x2 | WordBoundaries(0) | Audio(1) x26 | FlushDone(1) | Audio(2) | WordBoundaries(1) | Audio(2) x20 | FlushDone(2) | WordBoundaries(2) | Termination ``` After `Termination` the socket closes with `1000`. The `Audio` counts and the exact position of each `WordBoundaries` differ between runs. The program on [Word boundaries](https://assemblyai.com/docs/tts/word-boundaries.md#when-it-arrives) prints this sequence for a session of your own. ## Frames that do less than they look like **`KeepAlive` is not a warm-up.** It restarts the idle timer and nothing else. It does not reduce latency, does not reserve anything, and sending it in a tight loop trips the control-frame rate limit and closes the session with `3010`. If you want a request to be fast, hold the socket open, with a `KeepAlive` whenever two minutes might otherwise pass without a frame; that is what removes the setup cost. **There is no warm or prewarm endpoint.** The only client-side warm-up that exists is connecting early and holding the socket. See [Optimizing accuracy and latency](https://assemblyai.com/docs/tts/getting-started/optimizing-accuracy-and-latency.md) for what that saves. ## Related * [WebSocket API basics](https://assemblyai.com/docs/tts/connect-parameters.md): configuring the session before any of these frames, and decoding and playing `Audio` frames ([Audio formats](https://assemblyai.com/docs/tts/connect-parameters.md#audio-formats)). * [Word boundaries](https://assemblyai.com/docs/tts/word-boundaries.md): using `WordBoundaries` timings for highlighting. * [Error handling](https://assemblyai.com/docs/tts/error-handling.md): `Error` frames and close codes. * [Limits](https://assemblyai.com/docs/tts/limits.md): the caps these frames are subject to.