Message Sequence

This page is the canonical reference for the Voice Agent WebSocket protocol: every event the client sends, every event the server emits, and the order they appear in. For the field-level schema of each event, see the Voice Agent WebSocket API reference and the Events reference.

Sequence diagram

Events at a glance

Client → server

Event	Purpose
`session.update`	Configure the agent. First message after connect; can also be sent mid-session to update mutable fields.
`session.resume`	Resume a previous session by `session_id`.
`input.audio`	Stream a chunk of PCM16 audio to the agent (base64-encoded).
`tool.result`	Return the result of a tool invocation the agent requested.
`reply.create`	Ask the agent to generate a reply now, optionally with one-shot instructions.
`session.end`	Cleanly end the session.

Server → client

Event	When	Purpose
`session.ready`	Once, after `session.update` is applied	Session established. Includes the `session_id`. Start streaming audio.
`session.updated`	After a mid-session `session.update`	Acknowledges the update was applied.
`input.speech.started`	Per user turn	Turn detection determined the user started speaking.
`transcript.user.delta`	Repeatedly during a user turn	Partial transcript of the current user utterance.
`input.speech.stopped`	Per user turn	Turn detection determined the user stopped speaking.
`transcript.user`	Once per user turn	Final transcript of the user’s utterance.
`reply.started`	Per agent reply	Agent has begun generating a reply.
`reply.audio`	Repeatedly during a reply	A chunk of the agent’s spoken response as base64 PCM16.
`transcript.agent`	Per agent reply	Text of the agent’s response, sent after all reply audio has been delivered.
`tool.call`	When the agent invokes a tool	The agent wants to run a registered tool. Respond with `tool.result`.
`reply.done`	End of each reply	Marks the end of a reply. If a `tool.call` preceded it, send `tool.result` next.
`session.ended`	Once, last message	Session totals and clean shutdown. Nothing follows it.
`session.error`	On failure	Session- or protocol-level error.

Session initialization

Immediately after the WebSocket opens, send a session.update to configure the agent. You have two ways to configure:

Stored agent. Send { "agent_id": "<id>" } to bind to an agent created via the Agents REST API. The stored system_prompt, greeting, tools, input, and output are applied server-side.
Inline configuration. Omit agent_id and send system_prompt, greeting, tools, input, and output directly.

The two modes are mutually exclusive. See Deploy your agent and Session configuration for details.

{
  "type": "session.update",
  "session": {
    "system_prompt": "You are a concise assistant.",
    "greeting": "Hi! How can I help?",
    "input":  { "format": { "encoding": "audio/pcm" } },
    "output": { "voice": "ivy", "format": { "encoding": "audio/pcm" }, "volume": 100 }
  }
}

The server responds with session.ready once the session is established:

{
  "type": "session.ready",
  "session_id": "3207b601-2054-48df-ba77-8784dfcf9fb8"
}

Save session_id — you’ll need it to reconnect with session.resume if the WebSocket drops mid-conversation.

Sending audio

After session.ready, stream audio to the agent as input.audio events. Each event carries a base64-encoded chunk of PCM16 audio:

{
  "type": "input.audio",
  "audio": "<base64-encoded PCM16>"
}

Send audio in chunks of roughly 50 ms. The default input format is mono 24 kHz PCM16; you can change encoding and sample rate through session.input.format. See Audio format for the full format specification and Encoding for supported encodings.

Do not send input.audio before session.ready. Audio sent before the session is established is discarded.

User turn

When turn detection decides the user has started speaking, the server emits input.speech.started. As speech continues, partial transcripts stream back as transcript.user.delta. When the user stops speaking, the server emits input.speech.stopped followed by a final transcript.user:

{
  "type": "input.speech.started"
}

{
  "type": "transcript.user.delta",
  "delta": "hey there"
}

{
  "type": "input.speech.stopped"
}

{
  "type": "transcript.user",
  "transcript": "Hey there, what's the weather in Paris?"
}

See Turn detection and interruptions for tuning turn detection.

Agent reply

Once the user turn ends, the agent generates a reply. Every reply is bracketed by reply.started and reply.done. In between, audio streams back as reply.audio events (base64 PCM16), and the full text arrives as transcript.agent after all audio has been delivered:

{ "type": "reply.started" }

{ "type": "reply.audio", "audio": "<base64 PCM16 chunk>" }

{
  "type": "transcript.agent",
  "transcript": "It's 18 degrees and sunny in Paris right now."
}

{ "type": "reply.done" }

transcript.agent is emitted after the final reply.audio chunk, so it can be used to display the agent’s reply once playback completes.

Tool calls

If the agent decides to invoke a registered tool, the server emits a tool.call event. The current reply ends with reply.done, and the client is expected to run the tool and return the result via tool.result. A fresh reply.started … reply.done cycle follows once the tool result is received:

{
  "type": "tool.call",
  "tool_call_id": "call_01H...",
  "name": "get_weather",
  "arguments": { "city": "Paris" }
}

{
  "type": "tool.result",
  "tool_call_id": "call_01H...",
  "result": { "temperature_c": 18, "condition": "sunny" }
}

See Tool calling for the full tool-calling flow, including HTTP tools and client-side tools.

Updating configuration mid-session

You can send another session.update at any time to change mutable fields such as system_prompt, input.turn_detection, or output.volume. The server acknowledges with session.updated. Changing immutable fields (for example greeting, output.voice, or output.format) raises immutable_field. See Mutability after session.ready.

Session termination

To end a session cleanly, send session.end. The server flushes any in-flight events, emits session.ended, then closes the WebSocket with code 1000:

{ "type": "session.end" }

{
  "type": "session.ended",
  "reason": "client_ended"
}

Always terminate sessions explicitly. Sessions that are not ended remain open and continue to accrue charges until the server auto-closes them.

Errors

If the session fails at any point, the server sends a session.error event and closes the connection:

{
  "type": "session.error",
  "error_code": "immutable_field",
  "message": "Field 'session.output.voice' cannot be changed after session.ready."
}

See Troubleshooting for the full list of error codes and recovery guidance.

Getting started

Create & manage agents

Agent behavior

Conversational experience

Deploy

After a session

Reference

API reference

Sequence diagram

Events at a glance

Session initialization

Sending audio

User turn

Agent reply

Tool calls

Updating configuration mid-session

Session termination

Errors

​Sequence diagram

​Events at a glance

​Session initialization

​Sending audio

​User turn

​Agent reply

​Tool calls

​Updating configuration mid-session

​Session termination

​Errors

Sequence diagram

Events at a glance

Session initialization

Sending audio

User turn

Agent reply

Tool calls

Updating configuration mid-session

Session termination

Errors