> ## Documentation Index > Fetch the complete documentation index at: https://assemblyai.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Events reference > Every client-to-server and server-to-client event for the Voice Agent API. Every message exchanged over the Voice Agent API WebSocket, grouped by direction. You'll send [`session.update`](#sessionupdate) to configure, [`input.audio`](#inputaudio) to stream mic audio, [`tool.result`](#toolresult) to respond to tool calls, and [`session.end`](#sessionend) to cleanly end the call. The server streams everything else back, including [`session.ended`](#sessionended) on every clean teardown. ## Event flow A typical voice agent session moves through the events in this order: ``` Client Server │ │ │── WebSocket connect ─────────────►│ │── session.update ────────────────►│ (system prompt + tools + greeting) │ │ │◄─── session.ready ────────────────│ (save session_id) │ │ │── input.audio (stream) ──────────►│ (only after session.ready) │── input.audio (stream) ──────────►│ │ │ │◄─── input.speech.started ─────────│ │◄─── transcript.user.delta ────────│ │◄─── input.speech.stopped ─────────│ │◄─── transcript.user ──────────────│ │ │ │◄─── reply.started ────────────────│ │◄─── reply.audio ──────────────────│ │◄─── transcript.agent ─────────────│ │◄─── reply.done ───────────────────│ │ │ │ [tool call flow] │ │◄─── tool.call ────────────────────│ (arguments is a dict) │◄─── reply.done ───────────────────│ ← send tool.result here │── tool.result ───────────────────►│ │◄─── reply.started ────────────────│ │◄─── reply.audio ──────────────────│ │◄─── reply.done ───────────────────│ │ │ │ [clean teardown] │ │── session.end ───────────────────►│ │◄─── session.ended ────────────────│ │◄── WebSocket close ───────────────│ ``` ## Client → Server ### `input.audio` Stream PCM16 audio to the agent. ```json theme={null} { "type": "input.audio", "audio": "" } ``` | Field | Type | Description | | ------- | ------ | ------------------------------------- | | `audio` | string | Base64-encoded PCM16 mono 24kHz audio | See [Audio format](/voice-agents/voice-agent-api/audio-format) for the full format specification. *** ### `session.update` Configure the session. Send immediately on WebSocket connect (before `session.ready`). Can also be sent mid-conversation to update most fields. See [Mutability after `session.ready`](/voice-agents/voice-agent-api/session-configuration#mutability-after-sessionready) for which fields can change once the session is established. ```json expandable theme={null} { "type": "session.update", "session": { "system_prompt": "You are a concise assistant.", "greeting": "Hi! How can I help?", "input": { "format": { "encoding": "audio/pcm" }, "turn_detection": { "vad_threshold": 0.5 } }, "output": { "voice": "ivy", "format": { "encoding": "audio/pcm" }, "volume": 100 }, "tools": [ { "type": "function", "name": "get_weather", "description": "Get weather for a city", "parameters": { "type": "object", "properties": { "city": { "type": "string" } }, "required": ["city"] } } ] } } ``` All fields are optional. Include only what you want to set or change. After `session.ready`, only a subset of fields can be changed; changing `greeting`, `session.output.voice`, or `session.output.format` raises `immutable_field`. `session.output.volume` is mutable mid-session. | Field | Type | Description | | ------------------------------ | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `session.system_prompt` | string | Sets the agent's personality and context | | `session.greeting` | string | Spoken aloud at the start of the conversation | | `session.input.format` | object | Input audio format (`encoding`). See [Audio format](/voice-agents/voice-agent-api/audio-format) | | `session.input.keyterms` | array | List of strings to boost in transcription. See [Key terms](/voice-agents/voice-agent-api/session-configuration#key-terms) | | `session.input.turn_detection` | object | Turn detection configuration. See [Session configuration](/voice-agents/voice-agent-api/session-configuration#turn-detection) | | `session.output.voice` | string | The voice used for the agent's speech. See [Voices](/voice-agents/voice-agent-api/voices) | | `session.output.format` | object | Output audio format (`encoding`). See [Audio format](/voice-agents/voice-agent-api/audio-format) | | `session.output.volume` | number | Playback volume for the agent's speech, `0` (silent) to `100` (loudest). Mutable mid-session. See [Output volume](/voice-agents/voice-agent-api/session-configuration#output-volume) | | `session.tools` | array | Tool definitions. See [Tool calling](/voice-agents/voice-agent-api/tools/overview) | *** ### `session.resume` Reconnect to an existing session using the `session_id` from a previous `session.ready`. Preserves conversation context across dropped connections. ```json theme={null} { "type": "session.resume", "session_id": "sess_abc123" } ``` Sessions are preserved for 30 seconds after every disconnection before expiring. If the session has expired, the server returns a `session.error` with code `session_not_found` or `session_forbidden`. Start a fresh connection without `session.resume`. **Example.** Capture `session_id` from `session.ready` on the first connection, then send `session.resume` as the first message when reconnecting: ```python expandable theme={null} import json import websockets session_id: str | None = None async def connect(): global session_id async with websockets.connect(URL, additional_headers={"Authorization": f"Bearer {API_KEY}"}) as ws: # If we already have a session_id from a previous connection, resume it. if session_id: await ws.send(json.dumps({"type": "session.resume", "session_id": session_id})) else: await ws.send(json.dumps({"type": "session.update", "session": {...}})) async for raw in ws: event = json.loads(raw) if event["type"] == "session.ready": session_id = event["session_id"] # save for next reconnect elif event["type"] == "session.error" and event["code"] in ("session_not_found", "session_forbidden"): session_id = None # session expired - start fresh next time # ... handle other events # On disconnect, call connect() again within 30 seconds to resume. ``` *** ### `session.end` End the session cleanly. Send this when the call is over and you don't intend to reconnect. The server emits a final [`session.ended`](#sessionended) and closes the WebSocket; the `session_id` is dead immediately and cannot be resumed. ```json theme={null} { "type": "session.end" } ``` No other fields. **What happens next:** 1. The server emits [`session.ended`](#sessionended). 2. The server closes the WebSocket. 3. The `session_id` is dead. Sending [`session.resume`](#sessionresume) with it returns `session_not_found`. **When to send it vs. just closing the socket.** If you just close the WebSocket, the server holds the session for 30 seconds so you can [`session.resume`](#sessionresume) from a new connection, and that 30-second window is billable. `session.end` short-circuits the grace window and stops billing immediately. | What you do | Server behavior | Resumable? | Billing | | ------------------------------ | ---------------------------------------------- | --------------- | ------------------------------- | | Send `session.end`, then close | Closes immediately, emits `session.ended` | No | Stops immediately | | Just close the WebSocket | Holds the session for 30s for `session.resume` | Yes, within 30s | Continues during the 30s window | *** ### `tool.result` Send a tool result back to the agent. Send this when `reply.done` is the latest event you've received (and nothing has happened since). The simplest pattern is to accumulate on `tool.call` and drain inside the `reply.done` handler. See [Client-side tools](/voice-agents/voice-agent-api/tools/client-side-tools#returning-tool-results). ```json theme={null} { "type": "tool.result", "call_id": "call_abc123", "result": "{\"temp_c\": 22, \"description\": \"Sunny\"}" } ``` | Field | Type | Description | | --------- | ------ | ---------------------------------------- | | `call_id` | string | The `call_id` from the `tool.call` event | | `result` | string | JSON string containing the tool result | *** ### `reply.create` Ask the agent to generate a reply right now, optionally with custom `instructions`. Useful for delivering status updates during long-running [`hold`-mode tool calls](/voice-agents/voice-agent-api/tools/overview#execution-modes), or any time you want the agent to speak without a user utterance triggering it. ```json theme={null} { "type": "reply.create", "instructions": "Let the customer know we're still processing the transfer." } ``` | Field | Type | Description | | -------------- | ------ | ----------------------------------------------------------------------------------------------------- | | `instructions` | string | Optional. One-shot instruction the agent uses to compose this reply. Does not modify `system_prompt`. | The agent generates a normal reply (`reply.started` → `reply.audio` → `transcript.agent` → `reply.done`) using the provided instructions on top of the existing system prompt and conversation history. *** ## Server → Client ### `session.ready` Session is established and ready to receive audio. Save `session_id` for reconnection. Start sending `input.audio` only after this event. ```json theme={null} { "type": "session.ready", "session_id": "sess_abc123" } ``` | Field | Type | Description | | ------------ | ------ | ------------------------------------------------------------------- | | `session_id` | string | Always present. Save this value to reconnect with `session.resume`. | *** ### `session.updated` Sent after `session.update` is applied successfully. ```json theme={null} { "type": "session.updated" } ``` *** ### `session.ended` Final event emitted on every clean teardown, right before the server closes the WebSocket. Always handle it. ```json theme={null} { "type": "session.ended", "session_duration_seconds": 42.7, "audio_duration_seconds": 38.2, "timestamp": 1717180000.123 } ``` | Field | Type | Description | | -------------------------- | -------------- | --------------------------------------------------------- | | `session_duration_seconds` | number | Total wall-clock duration of the session. | | `audio_duration_seconds` | number \| null | Total audio you streamed in. `null` if you streamed none. | | `timestamp` | number | Unix epoch seconds when the server emitted the event. | **When you'll see it:** * You sent [`session.end`](#sessionend). * The session hit `max_session_duration_seconds`. * The server hit an unrecoverable error. * You disconnected and the 30-second grace window expired. **How to use it:** * **Got `session.ended` before the WebSocket closed**: clean server-initiated close. Done. Don't try to resume. * **WebSocket closed without `session.ended`**: probably a network drop. If you want to keep going, reconnect and call [`session.resume`](#sessionresume) with your saved `session_id` within 30 seconds. *** ### `input.speech.started` Turn detection determined the user has started speaking. ```json theme={null} { "type": "input.speech.started" } ``` *** ### `input.speech.stopped` Turn detection determined the user has stopped speaking. ```json theme={null} { "type": "input.speech.stopped" } ``` *** ### `transcript.user.delta` Partial transcript of what the user is saying, updating in real-time. ```json theme={null} { "type": "transcript.user.delta", "text": "What's the weather in" } ``` Live user transcripts pause while a [`hold`-mode tool](/voice-agents/voice-agent-api/tools/overview#execution-modes) is in flight and resume once the hold ends. Anything the user said during the hold is preserved in the conversation context. *** ### `transcript.user` Final transcript of the user's utterance. ```json theme={null} { "type": "transcript.user", "text": "What's the weather in Tokyo?", "item_id": "item_abc123" } ``` *** ### `reply.started` Agent has begun generating a response. ```json theme={null} { "type": "reply.started", "reply_id": "reply_abc123" } ``` *** ### `reply.audio` A chunk of the agent's spoken response as base64 PCM16. Decode and play immediately. ```json theme={null} { "type": "reply.audio", "data": "" } ``` See [Audio format](/voice-agents/voice-agent-api/audio-format#playing-output-audio) for playback guidance. *** ### `transcript.agent` Full text of the agent's response, sent after all audio for the response has been delivered. If the agent was interrupted, `interrupted` is `true` and `text` contains only what was actually spoken before the interruption. ```json theme={null} { "type": "transcript.agent", "text": "It's currently 22°C and sunny in Tokyo.", "reply_id": "reply_abc123", "item_id": "item_abc123", "interrupted": false } ``` | Field | Type | Description | | ------------- | ------- | ------------------------------------------------------------------ | | `text` | string | What the agent said (trimmed to interruption point if interrupted) | | `reply_id` | string | ID of the reply | | `item_id` | string | Conversation item ID | | `interrupted` | boolean | `true` if the user interrupted mid-response | *** ### `reply.done` Agent has finished speaking. The optional `status` field indicates why the reply ended. ```json theme={null} { "type": "reply.done" } ``` ```json theme={null} { "type": "reply.done", "status": "interrupted" } ``` | Field | Type | Description | | -------- | ------ | ------------------------------------------------------------------- | | `status` | string | `"interrupted"` if the user barged in, absent for normal completion | *** ### `tool.call` Agent wants to call a registered tool. `arguments` is a dict, ready to use directly as-is. ```json theme={null} { "type": "tool.call", "call_id": "call_abc123", "name": "get_weather", "arguments": { "location": "Tokyo" } } ``` | Field | Type | Description | | ----------- | ------ | ---------------------------------- | | `call_id` | string | Include this in `tool.result` | | `name` | string | Tool name to call | | `arguments` | object | Arguments as a dict (use directly) | See [Tool calling](/voice-agents/voice-agent-api/tools/overview) for the full pattern. *** ### `session.error` Session or protocol error. The payload always includes `type`, `timestamp`, `code`, and `message`. Some errors (like `session.update` validation failures) also include a `param` field naming the offending field. ```json theme={null} { "type": "session.error", "code": "invalid_format", "message": "Invalid message format", "timestamp": "2025-01-01T00:00:00Z" } ``` **Connection and handshake errors** Sent before or instead of `session.ready`. The WebSocket closes after these with the indicated close code. | Code | Close code | Description | | ---------------- | ---------- | -------------------------------------------- | | `UNAUTHORIZED` | 1008 | Missing or invalid `Authorization` token | | `FORBIDDEN` | 1008 | Valid token, but insufficient permissions | | `server_error` | 1008 | Service at capacity (try again later) | | `INTERNAL_ERROR` | 1011 | Unexpected exception during connection setup | **Session resume errors** Sent when `session.resume` fails. The WebSocket closes after these. | Code | Close code | Description | | ------------------- | ---------- | ----------------------------------------------------------------- | | `session_not_found` | 1008 | The `session_id` is unknown or the 30-second grace window expired | | `session_forbidden` | 1008 | The `session_id` belongs to a different account | | `session_expired` | 1008 | Session TTL elapsed during the grace window | **Agent startup errors** Sent after the WebSocket is accepted but before `session.ready`. | Code | Description | | ------------------- | -------------------------------------------------- | | `agent_init_failed` | Voice agent worker reported initialization failure | | `agent_timeout` | Agent did not signal ready within 10 seconds | **Client message errors** Sent on the open socket when an inbound message is invalid. The session stays alive (except `session_expired`). | Code | Description | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `invalid_format` | Bad JSON, missing or unknown `type`, validation failure, or missing `audio` field on `input.audio` | | `invalid_audio` | `input.audio` payload failed base64 decode or PCM conversion | | `invalid_value` | `session.update` with an invalid voice or field type | | `immutable_field` | `session.update` tried to change `greeting`, `output.voice`, or `output.format` after the first update was applied. `output.volume` is mutable and does not raise this error. | | `invalid_config` | `session.update` raised a validation error | | `server_error` | Unexpected exception while applying `session.update` | **Live session errors** | Code | Close code | Description | | ----------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `session_expired` | 1008 | Session duration TTL reached. There is no separate "closing soon" warning event before this, so run a client-side timer if you need to wrap up gracefully. | If the server cancels the session due to an internal error, the WebSocket closes with code `1011` without any `session.error` payload. In browsers, pre-handshake failures (like `UNAUTHORIZED`) surface as a `close` event with code `1006`. You won't receive a `session.error`. Always fetch a fresh token immediately before each connection attempt. *** ## Interruptions When the user speaks mid-response (barge-in), the server stops the agent and emits [`reply.done`](#replydone) with `status: "interrupted"` and [`transcript.agent`](#transcriptagent) with `interrupted: true`. The decision is semantic. Back-channels like "uh-huh" don't trigger an interruption. **On `reply.done` with `status: "interrupted"`:** 1. Flush your local audio playback buffer. 2. Discard any pending `tool.result` accumulators from the just-ended reply. 3. Restart the playback stream so it's ready for the next response. See [Turn detection and interruptions](/voice-agents/voice-agent-api/turn-detection-and-interruptions) for how the model decides what counts as an interruption, and [Handling interruptions](/voice-agents/voice-agent-api/audio-format#handling-interruptions) for the platform-specific flush pattern.