> ## Documentation Index > Fetch the complete documentation index at: https://assemblyai.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Voice Agent WebSocket API > Connect to the Voice Agent API to run a real-time voice conversation. The client streams PCM16 audio to the server and receives the agent's spoken response (also PCM16), along with transcripts, tool calls, and lifecycle events. After the WebSocket opens, send a [`session.update`](#sendSessionUpdate) as your first message. You have two ways to configure the agent: - **Stored agent.** Send `{ "agent_id": "" }` as the only field in `session` to bind to a reusable agent created via the [Agents REST API](/api-reference/voice-agent-api/create-agent). The stored `system_prompt`, `greeting`, `tools`, `input`, and `output` are applied server-side. - **Inline configuration.** Omit `agent_id` and send `system_prompt`, `greeting`, `tools`, `input`, and `output` directly. Useful for one-off or fully dynamic agents. The two modes are mutually exclusive. See [Deploy your agent](/voice-agents/voice-agent-api/deploy) and [Inline session configuration](/voice-agents/voice-agent-api/session-configuration) for details, or jump to the [Voice Agent API overview](/voice-agents/voice-agent-api) for the full event flow and a runnable quickstart. ## AsyncAPI ````yaml api-reference/specs/voice-agent-api.yaml voiceAgent id: voiceAgent title: Voice agent description: > Connect to the Voice Agent API to run a real-time voice conversation. The client streams PCM16 audio to the server and receives the agent's spoken response (also PCM16), along with transcripts, tool calls, and lifecycle events. After the WebSocket opens, send a [`session.update`](#sendSessionUpdate) as your first message. You have two ways to configure the agent: - **Stored agent.** Send `{ "agent_id": "" }` as the only field in `session` to bind to a reusable agent created via the [Agents REST API](/api-reference/voice-agent-api/create-agent). The stored `system_prompt`, `greeting`, `tools`, `input`, and `output` are applied server-side. - **Inline configuration.** Omit `agent_id` and send `system_prompt`, `greeting`, `tools`, `input`, and `output` directly. Useful for one-off or fully dynamic agents. The two modes are mutually exclusive. See [Deploy your agent](/voice-agents/voice-agent-api/deploy) and [Inline session configuration](/voice-agents/voice-agent-api/session-configuration) for details, or jump to the [Voice Agent API overview](/voice-agents/voice-agent-api) for the full event flow and a runnable quickstart. servers: - id: production protocol: wss host: agents.assemblyai.com bindings: [] variables: [] address: /v1/ws parameters: - id: ApiKey jsonSchema: type: string description: >- Pass your API key as a Bearer token in the `Authorization` header on the WebSocket upgrade request. For browser apps (which can't set custom headers on WebSockets), generate a [temporary token](/api-reference/voice-agent-api/generate-voice-agent-token) and pass it via the `token` query parameter instead. See [Browser integration](/voice-agents/voice-agent-api/browser-integration). examples: - Bearer YOUR_ASSEMBLYAI_API_KEY description: >- Pass your API key as a Bearer token in the `Authorization` header on the WebSocket upgrade request. For browser apps (which can't set custom headers on WebSockets), generate a [temporary token](/api-reference/voice-agent-api/generate-voice-agent-token) and pass it via the `token` query parameter instead. See [Browser integration](/voice-agents/voice-agent-api/browser-integration). type: string required: true deprecated: false - id: token jsonSchema: type: string description: >- Temporary authentication token for client-side connections. Generate one with [`GET /v1/token`](/api-reference/voice-agent-api/generate-voice-agent-token) on your server and pass it here so you don't expose your permanent API key in the browser. Each token is one-time use. description: >- Temporary authentication token for client-side connections. Generate one with [`GET /v1/token`](/api-reference/voice-agent-api/generate-voice-agent-token) on your server and pass it here so you don't expose your permanent API key in the browser. Each token is one-time use. type: string required: true deprecated: false bindings: [] operations: - &ref_18 id: sendSessionUpdate title: Send session update description: > Configure the session. Send immediately on connect — before `session.ready` — to either: - **Bind to a stored agent** by sending `{ "agent_id": "" }` as the only field in `session`. The agent's stored `system_prompt`, `greeting`, `tools`, `input`, and `output` are loaded server-side. Create stored agents with [`POST /v1/agents`](/api-reference/voice-agent-api/create-agent). - **Configure inline** by sending any combination of `system_prompt`, `greeting`, `tools`, `input`, and `output` (omit `agent_id`). `agent_id` is mutually exclusive with the inline fields; sending both raises a validation error. Can also be sent mid-conversation to update mutable fields (e.g. `system_prompt`, `input.turn_detection`). `greeting` and `output` are immutable after `session.ready` and changing them returns `immutable_field`. type: send messages: - &ref_37 id: sessionUpdate contentType: application/json payload: - name: Update Session description: > Client message to configure the session. Either bind to a stored agent by sending `agent_id`, or configure an inline agent with `system_prompt`, `greeting`, `input`, `output`, and `tools`. The two modes are mutually exclusive. type: object properties: - name: type type: string description: session.update required: true - name: session type: object description: > Session configuration fields. All fields are optional — only include the ones you want to change. There are two ways to configure a session: - **Bind to a stored agent.** Send `agent_id` (and only `agent_id`) in your first `session.update` to load a reusable agent created via the [Agents REST API](/api-reference/voice-agent-api/create-agent). The agent's stored `system_prompt`, `greeting`, `tools`, `input`, and `output` are applied server-side. - **Configure inline.** Omit `agent_id` and send any combination of `system_prompt`, `greeting`, `tools`, `input`, and `output` directly. `agent_id` is mutually exclusive with the inline fields — sending both in the same `session.update` is rejected. See [Deploy your agent](/voice-agents/voice-agent-api/deploy) for details. required: true properties: - name: agent_id type: string description: > ID of a stored agent (from [`POST /v1/agents`](/api-reference/voice-agent-api/create-agent)) to bind this session to. When set, the agent's stored configuration is applied and you must not send `system_prompt`, `greeting`, `tools`, `input`, or `output` in the same `session.update`. Can only be set in the first `session.update`, before `session.ready`. examples: &ref_0 - 7ad24396-b822-4dca-871a-be9cc4781cf9 required: false - name: system_prompt type: string description: >- The agent's personality and context. Can be updated mid-session. Mutually exclusive with `agent_id`. required: false - name: greeting type: string description: > What the agent says at the start of the conversation. Sent directly to the TTS engine and spoken verbatim. It is NOT run through the LLM, so write the exact words you want the user to hear. Immutable after `session.ready`. Omit to have the agent wait silently for the user to speak first. required: false - name: input type: object description: Configuration for the user audio input stream. required: false properties: - name: format type: object description: >- Audio format configuration. The encoding determines the sample rate. See [Audio format](/voice-agents/voice-agent-api/audio-format). required: false properties: - name: encoding type: string description: >- Audio encoding. `audio/pcm` (PCM16, 16-bit little-endian, 24 kHz), `audio/pcmu` (G.711 μ-law, 8 kHz), or `audio/pcma` (G.711 A-law, 8 kHz). enumValues: - audio/pcm - audio/pcmu - audio/pcma required: false - name: keyterms type: array description: >- List of rare or domain-specific terms to boost in transcription (e.g. names, brands, jargon). See [Key terms](/voice-agents/voice-agent-api/session-configuration#key-terms). required: false properties: - name: item type: string required: false - name: turn_detection type: object description: >- Configures turn detection sensitivity, end-of-turn detection, and barge-in behavior. Lives under `session.input.turn_detection`. required: false properties: - name: vad_threshold type: number description: >- Speech detection sensitivity (0.0–1.0). Lower = more sensitive to speech. required: false - name: min_silence type: integer description: >- Minimum silence to consider a confident end-of-turn, in milliseconds. Must be less than `max_silence`. required: false - name: max_silence type: integer description: >- Maximum silence before forcing end-of-turn, in milliseconds. Must be greater than `min_silence`. required: false - name: interrupt_response type: boolean description: >- Whether user speech interrupts the agent. Set `false` to disable barge-in. required: false - name: output type: object description: Configuration for the agent audio output stream. required: false properties: - name: voice type: string description: >- Voice used for the agent's speech. See [Voices](/voice-agents/voice-agent-api/voices). examples: &ref_1 - ivy required: false - name: format type: object description: >- Audio format configuration. The encoding determines the sample rate. See [Audio format](/voice-agents/voice-agent-api/audio-format). required: false properties: - name: encoding type: string description: >- Audio encoding. `audio/pcm` (PCM16, 16-bit little-endian, 24 kHz), `audio/pcmu` (G.711 μ-law, 8 kHz), or `audio/pcma` (G.711 A-law, 8 kHz). enumValues: - audio/pcm - audio/pcmu - audio/pcma required: false - name: volume type: number description: >- Playback volume for the agent's speech. `0` is silent, `100` is loudest. If omitted, the voice plays at its native level. Mutable mid-session. examples: &ref_3 - 100 required: false - name: tools type: array description: >- Tool definitions. See [Tool calling](/voice-agents/voice-agent-api/tools/overview). required: false properties: - name: type type: string description: Always `function`. required: true - name: name type: string description: Name of the tool, referenced by `tool.call`. required: true - name: description type: string description: Human-readable description of what the tool does. required: true - name: parameters type: object description: > JSON Schema describing the tool's arguments. Must be a JSON Schema object (`{"type": "object", "properties": {...}, "required": [...]}`). Passed to the agent's LLM verbatim. The server does NOT validate the schema at `session.update` time, so a malformed schema is accepted but produces unpredictable tool calls. Validate your schema before sending. Every property should have a `description`. That string is how the model extracts argument values from user speech. required: true properties: [] - name: execution_mode type: string description: > How the agent waits for the result. `"interactive"` (default): the agent speaks a short transition phrase (e.g. "let me check that") while the tool runs, then delivers the result conversationally. Use for short tools (DB lookups, REST calls). `"hold"`: the agent stays silent while the tool runs and suppresses replies triggered by user speech. Send `reply.create` to deliver status updates during the hold. When `tool.result` arrives, the agent gives a brief, direct delivery of the result. Use for long-running tools (transfers, escalations, async jobs). See [Tool calling — execution modes](/voice-agents/voice-agent-api/tools/overview#execution-modes). enumValues: - interactive - hold required: false - name: timeout_seconds type: number description: >- How long to wait for the matching `tool.result` before timing out. required: false headers: [] jsonPayloadSchema: type: object properties: type: type: string const: session.update x-parser-schema-id: session: type: object description: > Session configuration fields. All fields are optional — only include the ones you want to change. There are two ways to configure a session: - **Bind to a stored agent.** Send `agent_id` (and only `agent_id`) in your first `session.update` to load a reusable agent created via the [Agents REST API](/api-reference/voice-agent-api/create-agent). The agent's stored `system_prompt`, `greeting`, `tools`, `input`, and `output` are applied server-side. - **Configure inline.** Omit `agent_id` and send any combination of `system_prompt`, `greeting`, `tools`, `input`, and `output` directly. `agent_id` is mutually exclusive with the inline fields — sending both in the same `session.update` is rejected. See [Deploy your agent](/voice-agents/voice-agent-api/deploy) for details. required: [] properties: agent_id: type: string description: > ID of a stored agent (from [`POST /v1/agents`](/api-reference/voice-agent-api/create-agent)) to bind this session to. When set, the agent's stored configuration is applied and you must not send `system_prompt`, `greeting`, `tools`, `input`, or `output` in the same `session.update`. Can only be set in the first `session.update`, before `session.ready`. examples: *ref_0 x-parser-schema-id: system_prompt: type: string description: >- The agent's personality and context. Can be updated mid-session. Mutually exclusive with `agent_id`. x-parser-schema-id: greeting: type: string description: > What the agent says at the start of the conversation. Sent directly to the TTS engine and spoken verbatim. It is NOT run through the LLM, so write the exact words you want the user to hear. Immutable after `session.ready`. Omit to have the agent wait silently for the user to speak first. x-parser-schema-id: input: type: object description: Configuration for the user audio input stream. required: [] properties: format: &ref_2 type: object description: >- Audio format configuration. The encoding determines the sample rate. See [Audio format](/voice-agents/voice-agent-api/audio-format). required: [] properties: encoding: type: string enum: - audio/pcm - audio/pcmu - audio/pcma default: audio/pcm description: >- Audio encoding. `audio/pcm` (PCM16, 16-bit little-endian, 24 kHz), `audio/pcmu` (G.711 μ-law, 8 kHz), or `audio/pcma` (G.711 A-law, 8 kHz). x-parser-schema-id: x-parser-schema-id: AudioFormat keyterms: type: array items: type: string x-parser-schema-id: description: >- List of rare or domain-specific terms to boost in transcription (e.g. names, brands, jargon). See [Key terms](/voice-agents/voice-agent-api/session-configuration#key-terms). x-parser-schema-id: turn_detection: type: object description: >- Configures turn detection sensitivity, end-of-turn detection, and barge-in behavior. Lives under `session.input.turn_detection`. required: [] properties: vad_threshold: type: number format: float minimum: 0 maximum: 1 default: 0.5 description: >- Speech detection sensitivity (0.0–1.0). Lower = more sensitive to speech. x-parser-schema-id: min_silence: type: integer minimum: 50 maximum: 10000 default: 1000 description: >- Minimum silence to consider a confident end-of-turn, in milliseconds. Must be less than `max_silence`. x-parser-schema-id: max_silence: type: integer minimum: 50 maximum: 10000 default: 3000 description: >- Maximum silence before forcing end-of-turn, in milliseconds. Must be greater than `min_silence`. x-parser-schema-id: interrupt_response: type: boolean default: true description: >- Whether user speech interrupts the agent. Set `false` to disable barge-in. x-parser-schema-id: x-parser-schema-id: TurnDetection x-parser-schema-id: SessionInputConfig output: type: object description: Configuration for the agent audio output stream. required: [] properties: voice: type: string description: >- Voice used for the agent's speech. See [Voices](/voice-agents/voice-agent-api/voices). default: ivy examples: *ref_1 x-parser-schema-id: format: *ref_2 volume: type: number minimum: 0 maximum: 100 description: >- Playback volume for the agent's speech. `0` is silent, `100` is loudest. If omitted, the voice plays at its native level. Mutable mid-session. examples: *ref_3 x-parser-schema-id: x-parser-schema-id: SessionOutputConfig tools: type: array description: >- Tool definitions. See [Tool calling](/voice-agents/voice-agent-api/tools/overview). items: type: object description: A single tool the agent can call. properties: type: type: string const: function description: Always `function`. x-parser-schema-id: name: type: string description: Name of the tool, referenced by `tool.call`. x-parser-schema-id: description: type: string description: Human-readable description of what the tool does. x-parser-schema-id: parameters: type: object description: > JSON Schema describing the tool's arguments. Must be a JSON Schema object (`{"type": "object", "properties": {...}, "required": [...]}`). Passed to the agent's LLM verbatim. The server does NOT validate the schema at `session.update` time, so a malformed schema is accepted but produces unpredictable tool calls. Validate your schema before sending. Every property should have a `description`. That string is how the model extracts argument values from user speech. additionalProperties: true required: [] properties: {} x-parser-schema-id: execution_mode: type: string enum: - interactive - hold default: interactive description: > How the agent waits for the result. `"interactive"` (default): the agent speaks a short transition phrase (e.g. "let me check that") while the tool runs, then delivers the result conversationally. Use for short tools (DB lookups, REST calls). `"hold"`: the agent stays silent while the tool runs and suppresses replies triggered by user speech. Send `reply.create` to deliver status updates during the hold. When `tool.result` arrives, the agent gives a brief, direct delivery of the result. Use for long-running tools (transfers, escalations, async jobs). See [Tool calling — execution modes](/voice-agents/voice-agent-api/tools/overview#execution-modes). x-parser-schema-id: timeout_seconds: type: number format: float minimum: 1 maximum: 300 default: 120 description: >- How long to wait for the matching `tool.result` before timing out. x-parser-schema-id: required: - type - name - description - parameters x-parser-schema-id: ToolDefinition x-parser-schema-id: x-parser-schema-id: SessionConfig required: - type - session x-parser-schema-id: SessionUpdatePayload title: Update Session description: > Client message to configure the session. Either bind to a stored agent by sending `agent_id`, or configure an inline agent with `system_prompt`, `greeting`, `input`, `output`, and `tools`. The two modes are mutually exclusive. example: |- { "type": "session.update", "session": { "agent_id": "7ad24396-b822-4dca-871a-be9cc4781cf9" } } bindings: [] extensions: - id: x-parser-unique-object-id value: sessionUpdate bindings: [] extensions: &ref_4 - id: x-parser-unique-object-id value: voiceAgent - &ref_19 id: sendSessionResume title: Send session resume description: > Resume a previous session using the `session_id` from a prior `session.ready`. Preserves conversation context across dropped connections. Sessions are held for 30 seconds after every disconnection. type: send messages: - &ref_38 id: sessionResume contentType: application/json payload: - name: Resume Session description: Client message to resume a previous session by `session_id`. type: object properties: - name: type type: string description: session.resume required: true - name: session_id type: string description: The `session_id` from a previous `session.ready` event. required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: session.resume x-parser-schema-id: session_id: type: string description: The `session_id` from a previous `session.ready` event. x-parser-schema-id: required: - type - session_id x-parser-schema-id: SessionResumePayload title: Resume Session description: Client message to resume a previous session by `session_id`. example: |- { "type": "session.resume", "session_id": "sess_abc123" } bindings: [] extensions: - id: x-parser-unique-object-id value: sessionResume bindings: [] extensions: *ref_4 - &ref_20 id: sendSessionEnd title: Send session end description: > Cleanly end the session. The server emits a final `session.ended` and closes the WebSocket; the `session_id` is dead immediately and cannot be resumed. Use this instead of just closing the socket when the call is over — closing the socket without sending `session.end` leaves the session resumable (and billable) for 30 seconds. type: send messages: - &ref_39 id: sessionEnd contentType: application/json payload: - name: End Session description: > Client message to cleanly end the session. The server emits a final `session.ended` and closes the WebSocket; the `session_id` is dead immediately and cannot be resumed. Use this instead of just closing the socket to stop billing right away — closing the socket without `session.end` leaves the session resumable (and billable) for 30 seconds. type: object properties: - name: type type: string description: session.end required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: session.end x-parser-schema-id: required: - type x-parser-schema-id: SessionEndPayload title: End Session description: > Client message to cleanly end the session. The server emits a final `session.ended` and closes the WebSocket; the `session_id` is dead immediately and cannot be resumed. Use this instead of just closing the socket to stop billing right away — closing the socket without `session.end` leaves the session resumable (and billable) for 30 seconds. example: |- { "type": "session.end" } bindings: [] extensions: - id: x-parser-unique-object-id value: sessionEnd bindings: [] extensions: *ref_4 - &ref_21 id: sendInputAudio title: Send input audio description: > Stream a chunk of user audio to the agent. Only send `input.audio` after `session.ready`. See [Audio format](/voice-agents/voice-agent-api/audio-format) for the expected encoding (base64-encoded, mono). Supports `audio/pcm` (24 kHz), `audio/pcmu` (8 kHz), and `audio/pcma` (8 kHz). type: send messages: - &ref_40 id: inputAudio contentType: application/json payload: - name: Input Audio Chunk description: Client streams a chunk of PCM16 audio as base64. type: object properties: - name: type type: string description: input.audio required: true - name: audio type: string description: Base64-encoded audio chunk in the configured input encoding. required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: input.audio x-parser-schema-id: audio: type: string description: Base64-encoded audio chunk in the configured input encoding. x-parser-schema-id: required: - type - audio x-parser-schema-id: InputAudioPayload title: Input Audio Chunk description: Client streams a chunk of PCM16 audio as base64. example: |- { "type": "input.audio", "audio": "EAAgADAAQAAwACAAEAAAAPD/4P/Q/8D/" } bindings: [] extensions: - id: x-parser-unique-object-id value: inputAudio bindings: [] extensions: *ref_4 - &ref_22 id: sendToolResult title: Send tool result description: > Return a tool result to the agent. Send this inside your `reply.done` handler — not immediately on `tool.call`. See [Tool calling](/voice-agents/voice-agent-api/tools/overview). type: send messages: - &ref_41 id: toolResult contentType: application/json payload: - name: Tool Result description: Client returns the result of a tool invocation to the agent. type: object properties: - name: type type: string description: tool.result required: true - name: call_id type: string description: >- The `call_id` from the `tool.call` event you are responding to. required: true - name: result type: string description: > JSON-encoded string containing the tool result. Always a string, not a nested object. Use `json.dumps(...)` (Python) or `JSON.stringify(...)` (JS) on the payload before sending. required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: tool.result x-parser-schema-id: call_id: type: string description: The `call_id` from the `tool.call` event you are responding to. x-parser-schema-id: result: type: string description: > JSON-encoded string containing the tool result. Always a string, not a nested object. Use `json.dumps(...)` (Python) or `JSON.stringify(...)` (JS) on the payload before sending. x-parser-schema-id: required: - type - call_id - result x-parser-schema-id: ToolResultPayload title: Tool Result description: Client returns the result of a tool invocation to the agent. example: |- { "type": "tool.result", "call_id": "call_abc123", "result": "{\"temp_c\": 22, \"description\": \"Sunny\"}" } bindings: [] extensions: - id: x-parser-unique-object-id value: toolResult bindings: [] extensions: *ref_4 - &ref_23 id: sendReplyCreate title: Send reply create description: > Ask the agent to generate a reply right now, optionally with one-shot `instructions`. Primary use case: deliver status updates while a `hold`-mode tool call is in flight. See [Tool calling — hold mode](/voice-agents/voice-agent-api/tools/overview#hold). type: send messages: - &ref_42 id: replyCreate contentType: application/json payload: - name: Reply Create description: >- Client asks the agent to generate a reply now, optionally with one-shot instructions. type: object properties: - name: type type: string description: reply.create required: true - name: instructions type: string description: > Optional one-shot instructions the agent uses to compose this reply. Does not modify `system_prompt`. Useful for status updates during a `hold`-mode tool call. See [Tool calling — execution modes](/voice-agents/voice-agent-api/tools/overview#execution-modes). required: false headers: [] jsonPayloadSchema: type: object properties: type: type: string const: reply.create x-parser-schema-id: instructions: type: string description: > Optional one-shot instructions the agent uses to compose this reply. Does not modify `system_prompt`. Useful for status updates during a `hold`-mode tool call. See [Tool calling — execution modes](/voice-agents/voice-agent-api/tools/overview#execution-modes). x-parser-schema-id: required: - type x-parser-schema-id: ReplyCreatePayload title: Reply Create description: >- Client asks the agent to generate a reply now, optionally with one-shot instructions. example: |- { "type": "reply.create", "instructions": "Let the customer know we're still processing the transfer." } bindings: [] extensions: - id: x-parser-unique-object-id value: replyCreate bindings: [] extensions: *ref_4 - &ref_5 id: receiveSessionReady title: Receive session ready description: >- Session is established. Save `session_id` for reconnection and start streaming audio. type: receive messages: - &ref_24 id: sessionReady contentType: application/json payload: - name: Session Ready description: Server confirms the session is established and ready for audio. type: object properties: - name: type type: string description: session.ready required: true - name: session_id type: string description: >- Unique identifier for this session. Save this to reconnect with `session.resume`. required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: session.ready x-parser-schema-id: session_id: type: string description: >- Unique identifier for this session. Save this to reconnect with `session.resume`. x-parser-schema-id: required: - type - session_id x-parser-schema-id: SessionReadyPayload title: Session Ready description: Server confirms the session is established and ready for audio. example: |- { "type": "session.ready", "session_id": "sess_abc123" } bindings: [] extensions: - id: x-parser-unique-object-id value: sessionReady bindings: [] extensions: *ref_4 - &ref_6 id: receiveSessionUpdated title: Receive session updated description: Sent after a `session.update` is applied successfully. type: receive messages: - &ref_25 id: sessionUpdated contentType: application/json payload: - name: Session Updated description: >- Server acknowledges that a `session.update` was applied successfully. type: object properties: - name: type type: string description: session.updated required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: session.updated x-parser-schema-id: required: - type x-parser-schema-id: SessionUpdatedPayload title: Session Updated description: Server acknowledges that a `session.update` was applied successfully. example: |- { "type": "session.updated" } bindings: [] extensions: - id: x-parser-unique-object-id value: sessionUpdated bindings: [] extensions: *ref_4 - &ref_7 id: receiveSessionEnded title: Receive session ended description: > Final event emitted on every clean teardown, right before the WebSocket closes. Sent when the client sends `session.end`, when the session hits `max_session_duration_seconds`, when the server hits an unrecoverable error, or when the 30-second grace window after a disconnect expires. type: receive messages: - &ref_26 id: sessionEnded contentType: application/json payload: - name: Session Ended description: > Final event emitted on every clean teardown, right before the WebSocket closes. Sent when the client sends `session.end`, the session hits `max_session_duration_seconds`, the server hits an unrecoverable error, or the 30-second grace window after a disconnect expires. type: object properties: - name: type type: string description: session.ended required: true - name: session_duration_seconds type: number description: Total wall-clock duration of the session, in seconds. required: true - name: audio_duration_seconds type: number description: >- Total audio streamed in by the client, in seconds. `null` if no audio was streamed. required: false - name: timestamp type: number description: Unix epoch seconds when the server emitted the event. required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: session.ended x-parser-schema-id: session_duration_seconds: type: number description: Total wall-clock duration of the session, in seconds. x-parser-schema-id: audio_duration_seconds: type: number nullable: true description: >- Total audio streamed in by the client, in seconds. `null` if no audio was streamed. x-parser-schema-id: timestamp: type: number description: Unix epoch seconds when the server emitted the event. x-parser-schema-id: required: - type - session_duration_seconds - timestamp x-parser-schema-id: SessionEndedPayload title: Session Ended description: > Final event emitted on every clean teardown, right before the WebSocket closes. Sent when the client sends `session.end`, the session hits `max_session_duration_seconds`, the server hits an unrecoverable error, or the 30-second grace window after a disconnect expires. example: |- { "type": "session.ended", "session_duration_seconds": 42.7, "audio_duration_seconds": 38.2, "timestamp": 1717180000.123 } bindings: [] extensions: - id: x-parser-unique-object-id value: sessionEnded bindings: [] extensions: *ref_4 - &ref_8 id: receiveSessionError title: Receive session error description: A session- or protocol-level error occurred. type: receive messages: - &ref_27 id: sessionError contentType: application/json payload: - name: Session Error description: Server reports a session- or protocol-level error. type: object properties: - name: type type: string description: > `session.error` for session/protocol errors and `error` for connection-level errors. enumValues: - session.error - error required: true - name: code type: string description: > Machine-readable error code. See [error codes](/voice-agents/voice-agent-api/events-reference#sessionerror) for the full table grouped by lifecycle stage. enumValues: - UNAUTHORIZED - FORBIDDEN - INTERNAL_ERROR - server_error - session_not_found - session_forbidden - session_expired - agent_init_failed - agent_timeout - invalid_format - invalid_audio - invalid_value - immutable_field - invalid_config required: true - name: message type: string description: Human-readable error description. required: true - name: timestamp type: string description: ISO-8601 timestamp of the error. required: false - name: param type: string description: >- Name of the offending field when applicable (e.g. on `session.update` validation failures). required: false headers: [] jsonPayloadSchema: type: object properties: type: type: string enum: - session.error - error description: > `session.error` for session/protocol errors and `error` for connection-level errors. x-parser-schema-id: code: type: string enum: - UNAUTHORIZED - FORBIDDEN - INTERNAL_ERROR - server_error - session_not_found - session_forbidden - session_expired - agent_init_failed - agent_timeout - invalid_format - invalid_audio - invalid_value - immutable_field - invalid_config description: > Machine-readable error code. See [error codes](/voice-agents/voice-agent-api/events-reference#sessionerror) for the full table grouped by lifecycle stage. x-parser-schema-id: message: type: string description: Human-readable error description. x-parser-schema-id: timestamp: type: string format: date-time description: ISO-8601 timestamp of the error. x-parser-schema-id: param: type: string description: >- Name of the offending field when applicable (e.g. on `session.update` validation failures). x-parser-schema-id: required: - type - code - message x-parser-schema-id: SessionErrorPayload title: Session Error description: Server reports a session- or protocol-level error. example: |- { "type": "session.error", "code": "invalid_format", "message": "Invalid message format" } bindings: [] extensions: - id: x-parser-unique-object-id value: sessionError bindings: [] extensions: *ref_4 - &ref_9 id: receiveInputSpeechStarted title: Receive input speech started description: Turn detection determined the user has started speaking. type: receive messages: - &ref_28 id: inputSpeechStarted contentType: application/json payload: - name: User Started Speaking description: >- Server signals that turn detection determined the user started speaking. type: object properties: - name: type type: string description: input.speech.started required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: input.speech.started x-parser-schema-id: required: - type x-parser-schema-id: InputSpeechStartedPayload title: User Started Speaking description: >- Server signals that turn detection determined the user started speaking. example: |- { "type": "input.speech.started" } bindings: [] extensions: - id: x-parser-unique-object-id value: inputSpeechStarted bindings: [] extensions: *ref_4 - &ref_10 id: receiveInputSpeechStopped title: Receive input speech stopped description: Turn detection determined the user has stopped speaking. type: receive messages: - &ref_29 id: inputSpeechStopped contentType: application/json payload: - name: User Stopped Speaking description: >- Server signals that turn detection determined the user stopped speaking. type: object properties: - name: type type: string description: input.speech.stopped required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: input.speech.stopped x-parser-schema-id: required: - type x-parser-schema-id: InputSpeechStoppedPayload title: User Stopped Speaking description: >- Server signals that turn detection determined the user stopped speaking. example: |- { "type": "input.speech.stopped" } bindings: [] extensions: - id: x-parser-unique-object-id value: inputSpeechStopped bindings: [] extensions: *ref_4 - &ref_11 id: receiveTranscriptUserDelta title: Receive transcript user delta description: Partial transcript of the user's utterance, updating in real-time. type: receive messages: - &ref_30 id: transcriptUserDelta contentType: application/json payload: - name: User Transcript Delta description: Partial transcript of the user's current utterance. type: object properties: - name: type type: string description: transcript.user.delta required: true - name: text type: string description: Partial transcript of what the user is saying. required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: transcript.user.delta x-parser-schema-id: text: type: string description: Partial transcript of what the user is saying. x-parser-schema-id: required: - type - text x-parser-schema-id: TranscriptUserDeltaPayload title: User Transcript Delta description: Partial transcript of the user's current utterance. example: |- { "type": "transcript.user.delta", "text": "What's the weather in" } bindings: [] extensions: - id: x-parser-unique-object-id value: transcriptUserDelta bindings: [] extensions: *ref_4 - &ref_12 id: receiveTranscriptUser title: Receive transcript user description: Final transcript of the user's utterance. type: receive messages: - &ref_31 id: transcriptUser contentType: application/json payload: - name: User Transcript description: Final transcript of the user's utterance. type: object properties: - name: type type: string description: transcript.user required: true - name: text type: string description: Final transcript of the user's utterance. required: true - name: item_id type: string description: Conversation item ID. required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: transcript.user x-parser-schema-id: text: type: string description: Final transcript of the user's utterance. x-parser-schema-id: item_id: type: string description: Conversation item ID. x-parser-schema-id: required: - type - text - item_id x-parser-schema-id: TranscriptUserPayload title: User Transcript description: Final transcript of the user's utterance. example: |- { "type": "transcript.user", "text": "What's the weather in Tokyo?", "item_id": "item_abc123" } bindings: [] extensions: - id: x-parser-unique-object-id value: transcriptUser bindings: [] extensions: *ref_4 - &ref_13 id: receiveReplyStarted title: Receive reply started description: Agent has begun generating a response. type: receive messages: - &ref_32 id: replyStarted contentType: application/json payload: - name: Reply Started description: Agent has begun generating a reply. type: object properties: - name: type type: string description: reply.started required: true - name: reply_id type: string description: ID of this reply. required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: reply.started x-parser-schema-id: reply_id: type: string description: ID of this reply. x-parser-schema-id: required: - type - reply_id x-parser-schema-id: ReplyStartedPayload title: Reply Started description: Agent has begun generating a reply. example: |- { "type": "reply.started", "reply_id": "reply_abc123" } bindings: [] extensions: - id: x-parser-unique-object-id value: replyStarted bindings: [] extensions: *ref_4 - &ref_14 id: receiveReplyAudio title: Receive reply audio description: > A chunk of the agent's spoken response (base64 PCM16). Decode and play immediately. See [Audio format](/voice-agents/voice-agent-api/audio-format#playing-output-audio) for playback guidance. type: receive messages: - &ref_33 id: replyAudio contentType: application/json payload: - name: Reply Audio Chunk description: A chunk of the agent's spoken response as base64 PCM16. type: object properties: - name: type type: string description: reply.audio required: true - name: data type: string description: Base64-encoded audio chunk in the configured output encoding. required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: reply.audio x-parser-schema-id: data: type: string description: Base64-encoded audio chunk in the configured output encoding. x-parser-schema-id: required: - type - data x-parser-schema-id: ReplyAudioPayload title: Reply Audio Chunk description: A chunk of the agent's spoken response as base64 PCM16. example: |- { "type": "reply.audio", "data": "EAAgADAAQAAwACAAEAAAAPD/4P/Q/8D/" } bindings: [] extensions: - id: x-parser-unique-object-id value: replyAudio bindings: [] extensions: *ref_4 - &ref_15 id: receiveTranscriptAgent title: Receive transcript agent description: >- Full text of the agent's response, delivered after all audio for the reply has been sent. type: receive messages: - &ref_34 id: transcriptAgent contentType: application/json payload: - name: Agent Transcript description: >- Text of the agent's response, sent after all reply audio has been delivered. type: object properties: - name: type type: string description: transcript.agent required: true - name: text type: string description: >- What the agent said. If interrupted, trimmed to the point of interruption. required: true - name: reply_id type: string description: ID of the reply this transcript belongs to. required: true - name: item_id type: string description: Conversation item ID. required: true - name: interrupted type: boolean description: Whether the user interrupted the agent mid-response. required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: transcript.agent x-parser-schema-id: text: type: string description: >- What the agent said. If interrupted, trimmed to the point of interruption. x-parser-schema-id: reply_id: type: string description: ID of the reply this transcript belongs to. x-parser-schema-id: item_id: type: string description: Conversation item ID. x-parser-schema-id: interrupted: type: boolean description: Whether the user interrupted the agent mid-response. x-parser-schema-id: required: - type - text - reply_id - item_id - interrupted x-parser-schema-id: TranscriptAgentPayload title: Agent Transcript description: >- Text of the agent's response, sent after all reply audio has been delivered. example: |- { "type": "transcript.agent", "text": "It's currently 22°C and sunny in Tokyo.", "reply_id": "reply_abc123", "item_id": "item_abc123", "interrupted": false } bindings: [] extensions: - id: x-parser-unique-object-id value: transcriptAgent bindings: [] extensions: *ref_4 - &ref_16 id: receiveReplyDone title: Receive reply done description: >- Agent has finished speaking. Send any accumulated `tool.result` events here. type: receive messages: - &ref_35 id: replyDone contentType: application/json payload: - name: Reply Done description: > Agent has finished speaking. If the user barged in, `status` is `"interrupted"`. Send accumulated `tool.result` events on this event. type: object properties: - name: type type: string description: reply.done required: true - name: status type: string description: > `"completed"` for normal completion, `"interrupted"` if the user barged in. enumValues: - completed - interrupted required: true headers: [] jsonPayloadSchema: type: object properties: type: type: string const: reply.done x-parser-schema-id: status: type: string enum: - completed - interrupted description: > `"completed"` for normal completion, `"interrupted"` if the user barged in. x-parser-schema-id: required: - type - status x-parser-schema-id: ReplyDonePayload title: Reply Done description: > Agent has finished speaking. If the user barged in, `status` is `"interrupted"`. Send accumulated `tool.result` events on this event. example: |- { "type": "reply.done" } bindings: [] extensions: - id: x-parser-unique-object-id value: replyDone bindings: [] extensions: *ref_4 - &ref_17 id: receiveToolCall title: Receive tool call description: > Agent wants to invoke a registered tool. Execute the tool, then send the result with `tool.result` after `reply.done` fires. type: receive messages: - &ref_36 id: toolCall contentType: application/json payload: - name: Tool Call description: Agent wants to invoke a registered tool. type: object properties: - name: type type: string description: tool.call required: true - name: call_id type: string description: Include this value in the corresponding `tool.result`. required: true - name: name type: string description: Name of the tool the agent is invoking. required: true - name: arguments type: object description: Arguments to pass to the tool, as a dictionary. required: true properties: [] headers: [] jsonPayloadSchema: type: object properties: type: type: string const: tool.call x-parser-schema-id: call_id: type: string description: Include this value in the corresponding `tool.result`. x-parser-schema-id: name: type: string description: Name of the tool the agent is invoking. x-parser-schema-id: arguments: type: object description: Arguments to pass to the tool, as a dictionary. additionalProperties: true required: [] properties: {} x-parser-schema-id: required: - type - call_id - name - arguments x-parser-schema-id: ToolCallPayload title: Tool Call description: Agent wants to invoke a registered tool. example: |- { "type": "tool.call", "call_id": "call_abc123", "name": "get_weather", "arguments": { "location": "Tokyo" } } bindings: [] extensions: - id: x-parser-unique-object-id value: toolCall bindings: [] extensions: *ref_4 sendOperations: - *ref_5 - *ref_6 - *ref_7 - *ref_8 - *ref_9 - *ref_10 - *ref_11 - *ref_12 - *ref_13 - *ref_14 - *ref_15 - *ref_16 - *ref_17 receiveOperations: - *ref_18 - *ref_19 - *ref_20 - *ref_21 - *ref_22 - *ref_23 sendMessages: - *ref_24 - *ref_25 - *ref_26 - *ref_27 - *ref_28 - *ref_29 - *ref_30 - *ref_31 - *ref_32 - *ref_33 - *ref_34 - *ref_35 - *ref_36 receiveMessages: - *ref_37 - *ref_38 - *ref_39 - *ref_40 - *ref_41 - *ref_42 extensions: - id: x-parser-unique-object-id value: voiceAgent securitySchemes: [] ````