Message frames

Every message in both directions is a JSON text frame with a type field naming the frame.

Frame rules

These rules apply to everything you send:

Send text frames only, each one a JSON object with a type field. A binary frame, text that is not a JSON object, a missing or unknown type, or a required field that is missing or of the wrong type (for example a Generate without text) ends the session: the server sends an Error frame with code 3006 and closes.
Extra fields you add to a known frame are ignored.
A WebSocket message over 1 MiB is refused by the WebSocket layer itself, with close code 1009 and no Error frame. The size counts the whole message, so a message sent in several fragments is measured once they are joined.
The server does not negotiate WebSocket compression (permessage-deflate), so frames in both directions travel uncompressed.

And one rule for everything you receive: ignore frame types you do not recognize. The protocol may add frame types. There is no warning frame: everything the server tells you about the session is in Begin, in the frames below, or in the Error frame that ends the session.

Client to server

Frame	Fields	Description
`Generate`	`type`, `text`	Appends text to the session buffer. Short text is silent until you `Flush`; once enough complete sentences are buffered the server starts speaking them before any `Flush` (see Generate)
`Flush`	`type`	Ends the current request: synthesizes whatever is still unspoken and concludes it with `FlushDone`
`Cancel`	`type`	Barge-in. Stops the current synthesis and discards any text you have sent but not yet heard, including pending flushes
`KeepAlive`	`type`	Restarts the two-minute idle timer
`Terminate`	`type`	Ends the session cleanly

Generate

{ "type": "Generate", "text": "Hi, thanks for calling." }

Generate appends text to the session buffer. The server does not always wait for Flush: once the buffer holds enough complete sentences, it starts speaking them under the current flush_id.

Text that arrives in one Generate starts once its complete sentences add up to a long enough stretch of text.
Text streamed over several Generate frames, as an LLM produces it, starts sooner: once its first sentences are complete, or at a clause end (,, ; or :) when a long sentence has none yet. The rest of the request follows in parts that start early enough that the audio already sent does not run out while text is still arriving.

These rules apply from the start of the session and again after every Flush and Cancel. A sentence ends once whitespace follows its ., ! or ?, so the end of your text so far does not count until the next text, or a trailing space, arrives; a period after an abbreviation or an initial, as in Dr. or J., does not end a sentence. A sentence of one word, such as Sure. or a list number such as 4., is never spoken on its own before Flush; it is read with the sentence after it. How much text the server gathers before it starts speaking can change, so do not rely on it: Flush is what concludes a request. Short text is buffered, not spoken. A session that sends a short Generate and never flushes therefore stays silent, looking perfectly healthy, until the idle timeout closes it with 3006 two minutes later, and the leftover text is never spoken. Only Flush (or Terminate) guarantees that everything you sent is spoken and gives you the FlushDone boundary. Text from consecutive Generate frames is joined exactly as sent, with no separator added. Good followed by morning. is spoken, and reported in WordBoundaries, as the single word Goodmorning.. Include the space or newline yourself when you split text across frames. A single Generate accepts up to 2000 characters, counted as Unicode code points. Over that, the session closes with 3006. Empty text is accepted and does nothing, but it counts toward the control-frame limit of 20 per second, like Flush, KeepAlive and Cancel.

Flush

{ "type": "Flush" }

Flush concludes the current request: whatever is still unspoken is synthesized, then a FlushDone arrives. For short text this is when audio starts. For longer text the server may already be streaming, and Flush synthesizes the remainder. FlushDone means the request’s audio has been fully delivered to your socket, not that it has finished playing. Synthesis runs several times faster than real time, so FlushDone usually lands well before playback ends. A Flush with nothing left to speak still returns exactly one FlushDone, with audio_duration_ms of 0 and no Audio frames. The flush_id still advances and the session stays usable. You do not have to wait for one FlushDone before sending the next Flush. Queued requests are answered in flush_id order. The server works through them one at a time, so each request you queue adds to the wait before its own audio starts. When you are streaming text out of an LLM, flush on sentence or clause boundaries. Flushing per token produces choppy prosody, because each fragment is synthesized on its own and complete sentences give the model the context it needs. It also costs you the session: Flush is a control frame, capped at 20 per second sustained with a burst allowance of about 60, and a per-token loop quickly exceeds that and is closed with 3010. See Limits.

Cancel

{ "type": "Cancel" }

Cancel is for barge-in, when the listener starts talking over the speech. It stops the in-flight synthesis and discards everything you have sent but not yet heard: text sent with Generate but not yet flushed, and every Flush still waiting its turn. None of them gets a FlushDone. The server answers with a Cancelled frame:

{ "type": "Cancelled", "audio_duration_seconds": 0.4 }

audio_duration_seconds is the audio that was streamed for the cancelled request only, in seconds rounded to 3 decimal places, and 0.0 when nothing was in flight. A Cancel with nothing in flight is still answered, and still advances flush_id. Once the session has reached expires_at the server reads no more frames, so a Cancel sent then gets no Cancelled: the request in flight streams to its FlushDone, and the session closes with 3008. Stop playback yourself in that case. Three things to design around:

A short tail of the old audio may still arrive. Audio frames already on the way when you sent Cancel can land before Cancelled does. Every Audio frame between your Cancel and its Cancelled belongs to a request you sent before the Cancel: drop it, and stop the buffers you have already scheduled for playback, or they keep playing. Each Cancel is answered by its own Cancelled, in the order sent.
The next request takes a new flush_id. It is higher than every id before the Cancel, but it can skip numbers, so read it from the first Audio or FlushDone after Cancelled rather than computing it. The next Generate and Flush continue under that id. The server handles your frames in the order you send them, so a Generate sent after Cancel is not caught by the cancel; you do not have to wait for Cancelled before sending the next reply.
Re-send anything you still want spoken. Unflushed text is gone; the next Flush speaks only what you send after the cancel.

The cancelled request gets no WordBoundaries. The session totals in Termination still count the cancelled text that had been passed to synthesis and the audio that was delivered, and the output timeline used by WordBoundaries counts that audio too, the tail delivered between your Cancel and the server’s Cancelled included.

KeepAlive

{ "type": "KeepAlive" }

Restarts the idle timer and does nothing else. A session closes after two minutes with no frame in either direction, or after the shorter inactivity_timeout you set, and any frame restarts that timer, so you only need KeepAlive while a session is idle: send one at least once per timeout to hold the socket open. It is a control frame and counts against the same 20 per second cap as Flush and Cancel.

Terminate

{ "type": "Terminate" }

Ends the session cleanly, in this order:

Text you sent with Generate but never flushed is flushed for you and gets its own FlushDone with the next flush_id.
Audio still in flight is delivered, and any pending WordBoundaries arrive.
A Termination frame carries the session summary, then the socket closes with code 1000 and reason Session ended.

Anything you send after Terminate is never read: no error, no audio, and it is not counted in the summary. If the server produces nothing for 30 seconds while a FlushDone is still owed, the session closes with 3005 instead of Termination. That window restarts whenever the server produces output, so a long final request is never cut short. You can also just close the socket. The server then discards any synthesis still in flight and sends no Termination; the audio it already delivered to you still counts.

Server to client

Frame	Fields	Description
`Begin`	`type`, `id`, `expires_at`, `configuration`	Session opened. The authoritative configuration
`Audio`	`type`, `audio`, `flush_id`	A chunk of base64 audio
`FlushDone`	`type`, `flush_id`, `audio_duration_ms`	That request’s audio has all been sent
`WordBoundaries`	`type`, `flush_id`, `audio_start_ms`, `words`	Per-word timings for that request, best-effort. `words` is a list of `{word, start, end, confidence}`
`Cancelled`	`type`, `audio_duration_seconds`	Answers `Cancel`
`Termination`	`type`, `session_duration_seconds`, `total_input_char_length`, `audio_duration_seconds`, `voice_expires_at`	Session summary, on a clean end only
`Error`	`type`, `error_code`, `error`, `error_type`, `retryable`, `param`, `retry_after_seconds`	Sent before every error close. `error_code` equals the close code that follows

Begin

{
  "type": "Begin",
  "id": "10885144-0d4b-4928-9eff-ef292a352937",
  "expires_at": 1789556741,
  "configuration": {
    "language": "english",
    "sample_rate": 24000,
    "encoding": "pcm_s16le",
    "inactivity_timeout": 120,
    "word_boundaries": true,
    "voice_id": "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3",
    "extend_biometric_consent": false,
    "word_boundaries_requested": true,
    "model": "assemblyai-tts-1",
    "session_ttl_sec": 3600,
    "limits": {
      "max_generate_text_length": 2000,
      "max_control_frames_per_sec": 20.0,
      "max_egress_audio_buffer_ms": 300000,
      "text_throttle_enabled": true,
      "text_throttle_max_chars_per_sec": 100.0,
      "text_throttle_initial_grace_chars": 1500,
      "text_max_buffer_chars": 8000,
      "session_ttl_grace_sec": 60,
      "max_message_bytes": 1048576,
      "max_queued_messages": 10000,
      "write_stall_timeout_sec": 30.0,
      "inactivity_timeout_min_sec": 5,
      "inactivity_timeout_max_sec": 120
    },
    "unknown_params": []
  }
}

Begin is always the first frame, sent before the server reads anything from you. It arrives only after the server confirms it has room for the connection, your credentials are checked, your parameters are validated, your account’s limit of new sessions per minute is checked, and the session’s language and voice are available on its model; if any of those fail you get an Error frame and a close instead. Read configuration as the session you actually got: every connection parameter except token and model_id with its effective value, whether you sent it or not, then what you requested where it differs (word_boundaries_requested), the model the session runs on (model), the session’s lifetime and limits, and the names of any query parameters the server ignored (unknown_params). Read the configuration back describes these keys. The example above is abridged to them; a session with a cloned voice also shows the keys listed below. configuration also carries keys for options these pages do not cover, and it may gain keys over time, so read the keys you need, ignore the others, and do not validate it against a closed set. Quote id in support requests. expires_at is a Unix timestamp in seconds; when it passes, the server stops accepting input, flushes and drains what it has, and closes with 3008. configuration carries two more keys about the voice. With a cloned voice, language is the voice’s own:

Key	Meaning
`voice_expires_at`	When the voice’s consent period ends, as a Unix timestamp in seconds, as the session opened; `null` for an AssemblyAI voice, for another account’s voice and for a voice whose consent period has no end. It is not the session’s own `expires_at`. See Consent period
`resolved_voice`	`voice_id`, the voice’s UUID, also when you sent its slug, and `language`, the voice’s language

Audio

{ "type": "Audio", "audio": "IwB6AOMABQHHAIMAJADw/wwA...", "flush_id": 0 }

Base64-encoded raw codec bytes with no header. Every Audio frame carries audio; the server never sends an empty one. Frame sizes are not fixed, so do not assume a chunk length: at 24 kHz they are about 80 ms each, the last frame of a request included; at other sample rates they vary, and the last frame of a request is often shorter. Audio for one request never interleaves with another’s, and all of a request’s Audio arrives before its FlushDone, but it can begin before you send Flush. See Audio formats on WebSocket API basics.

FlushDone

{ "type": "FlushDone", "flush_id": 0, "audio_duration_ms": 2320 }

Exactly one per Flush, including the implicit flush that Terminate performs, in flush_id order. It means every byte of that request’s audio has been sent to you, not that it has been played. audio_duration_ms is 0 when the request had nothing left to synthesize. To check that you received as much as the server sent, compare audio_duration_ms with bytes_received / (sample_rate * bytes_per_sample) * 1000, where bytes_per_sample is 2 for pcm_s16le and 1 for pcm_mulaw and pcm_alaw.

WordBoundaries

{
  "type": "WordBoundaries",
  "flush_id": 0,
  "audio_start_ms": 0,
  "words": [
    { "word": "Hello", "start": 265, "end": 530, "confidence": 0.4791773557662964 },
    { "word": "there,", "start": 546, "end": 712, "confidence": 0.9253882169723511 },
    { "word": "friend,", "start": 845, "end": 1126, "confidence": 0.8478114008903503 }
  ]
}

One frame per request, sent after that request’s FlushDone and never after Termination. If you have already flushed again it may arrive after the next request’s first Audio frames, so match it to its request by flush_id, not by position. Each entry has word, start, end and confidence. words has one entry for every whitespace-delimited token of the request’s text, in order, with its punctuation attached; a token with no sound of its own, such as a dash standing alone between spaces, gets a zero-length entry (start equal to end, confidence 0) at the edge of its neighboring word. start and end are milliseconds on the session’s continuous audio timeline, counted from the first audio the session delivered, and audio_start_ms is where this request’s audio begins on that same timeline. To place a word within this request’s own audio, subtract audio_start_ms from start; never add the two. A long request gets one frame like any other. The frame is best-effort: a cancelled request gets none, a request whose audio leaves part of its text unspoken gets none, and it can be missing when timing data is unavailable. See Word boundaries.

Termination

{
  "type": "Termination",
  "session_duration_seconds": 12,
  "total_input_char_length": 96,
  "audio_duration_seconds": 3.12,
  "voice_expires_at": null
}

session_duration_seconds is a whole number of seconds, truncated. total_input_char_length is every character of your Generate frames that was passed to synthesis, whether or not it was spoken, so text you cancelled or never flushed counts too. Text a Cancel discarded while the server was still holding it back under text intake pacing was never passed to synthesis, and does not count. audio_duration_seconds is the audio actually delivered to you, including the delivered part of a cancelled request, in seconds rounded to 3 decimal places. voice_expires_at is when the consent period of the session’s cloned voice ends, as a Unix timestamp in seconds, after the session’s renewal of the voice’s consent, if any, and null otherwise. Termination may gain keys over time; ignore keys you do not use. Termination is sent only on a clean finish. If the session fails you receive an Error frame and a close code instead, and if you close the socket without Terminate no summary is sent.

Error

{
  "type": "Error",
  "error_code": 3009,
  "error": "Too many sessions opened for this account in the last minute (allowance 100 per minute); retry in 17 s.",
  "error_type": "rate_limited",
  "retryable": true,
  "param": null,
  "retry_after_seconds": 17
}

Field	Meaning
`error_code`	The WebSocket close code that follows
`error`	What went wrong and what to do, written for people. It may be reworded, so log it rather than matching on it
`error_type`	A stable, machine-readable cause, finer than the close code: for example `invalid_parameter`, `inactivity_timeout` or `rate_limited`. See Error types
`retryable`	Whether reconnecting with the same parameters can succeed
`param`	The connection parameter or message field at fault, such as `sample_rate`, `voice_id` or `language` at connect, or `Generate.text` or `type` for a frame you sent; `null` when there is none
`retry_after_seconds`	The earliest useful retry, in seconds; `null` when there is no hint

Sent immediately before every failure close the server initiates. error_code equals the WebSocket close code that follows, and the close reason is normally the fixed string See Error message for details, so the frame is where the detail lives. If the Error frame could not be delivered, for example to a client that stopped reading, the close reason carries a shortened copy of its text instead. There is no Error frame for a clean 1000 close, nor for closes raised by the WebSocket layer itself: 1009 for an oversized message, 1011 when pings go unanswered, 1002 or 1007 for a frame that breaks the WebSocket protocol or is not valid UTF-8, or 1006 when the connection drops. See Error handling.

Frame order

A single short request through a session looks like this. Client frames are on the left, server frames on the right:

                                  <-  Begin {configuration}
->  Generate {text}
->  Flush
                                  <-  Audio {audio, flush_id: 0}
                                  <-  Audio {audio, flush_id: 0}
                                  <-  FlushDone {flush_id: 0, audio_duration_ms}
                                  <-  WordBoundaries {flush_id: 0}   (best-effort)
->  Terminate
                                  <-  Termination {summary}
                                      close 1000

Begin arrives once, as the very first frame. flush_id starts at 0 and each request you conclude with Flush takes the next id, so a session that handles several requests can use it to keep their audio apart. After a Cancel the next id can skip numbers. Two things the single-request picture hides:

WordBoundaries trails its FlushDone. With several requests in flight it often lands after the next request’s first Audio frames. Audio frames themselves never interleave across requests.
Audio can start before Flush. Once enough complete sentences are buffered (see Generate), the server begins streaming them; Flush then completes the request and produces the FlushDone.

You can send several Generate and Flush groups before the first FlushDone arrives. This is the real order for three requests flushed back to back, with runs of Audio frames collapsed:

Begin | Audio(0) x29 | FlushDone(0) | Audio(1) x2 | WordBoundaries(0) | Audio(1) x26 | FlushDone(1)
| Audio(2) | WordBoundaries(1) | Audio(2) x20 | FlushDone(2) | WordBoundaries(2) | Termination

After Termination the socket closes with 1000. The Audio counts and the exact position of each WordBoundaries differ between runs. The program on Word boundaries prints this sequence for a session of your own.

Frames that do less than they look like

KeepAlive is not a warm-up. It restarts the idle timer and nothing else. It does not reduce latency, does not reserve anything, and sending it in a tight loop trips the control-frame rate limit and closes the session with 3010. If you want a request to be fast, hold the socket open, with a KeepAlive whenever two minutes might otherwise pass without a frame; that is what removes the setup cost. There is no warm or prewarm endpoint. The only client-side warm-up that exists is connecting early and holding the socket. See Optimizing accuracy and latency for what that saves.

WebSocket API basics: configuring the session before any of these frames, and decoding and playing Audio frames (Audio formats).
Word boundaries: using WordBoundaries timings for highlighting.
Error handling: Error frames and close codes.
Limits: the caps these frames are subject to.

Getting started

Features

Integrations

API reference

Advanced

Frame rules

Client to server

Generate

Flush

Cancel

KeepAlive

Terminate

Server to client

Begin

Audio

FlushDone

WordBoundaries

Termination

Error

Frame order

Frames that do less than they look like

​Frame rules

​Client to server

​Generate

​Flush

​Cancel

​KeepAlive

​Terminate

​Server to client

​Begin

​Audio

​FlushDone

​WordBoundaries

​Termination

​Error

​Frame order

​Frames that do less than they look like

​Related

Frame rules

Client to server

Generate

Flush

Cancel

KeepAlive

Terminate

Server to client

Begin

Audio

FlushDone

WordBoundaries

Termination

Error

Frame order

Frames that do less than they look like

Related