If you want a voice agent and you would rather describe it than assemble it, the shortest path today is to open a coding agent — Claude Code, Cursor, GitHub Copilot or Windsurf — connect it to AssemblyAI’s live documentation, and describe the agent you want. There is one WebSocket, wss://agents.assemblyai.com/v1/ws. Speech-to-text, the LLM and text-to-speech are bundled behind it at a flat $4.50/hr. Most of your agent’s configuration lives server-side as a stored agent, so the code your coding agent writes is mostly a connection and an event loop.
Below: what a coding agent actually is if you have not used one, why this API is an unusually good target for one, the real event flow, a working Python example, and the benchmark data behind the speech layer.
What is a coding agent, and do I need to know how to code?
A coding agent is an LLM that has been given a terminal, a filesystem and your repository. You describe what you want in plain English; it reads the existing code, writes new files, runs them, reads the errors and fixes them. The loop is the point. A chat assistant hands you a snippet and wishes you luck. A coding agent runs the snippet, sees the stack trace and iterates until the thing works.
You do not need to be able to write the code yourself. You do need to be able to read enough of it to tell when the agent has gone somewhere strange, and you need to be able to say precisely what you want. In practice the skill that matters is specification, not syntax: “when the caller gives an order number, look it up and read back the ship date” is a better instruction than anything you would get out of a dropdown.
The four coding agents developers are actually using
- Claude Code — terminal-native. You run it inside your project directory and talk to it there. Strong at multi-file changes and at running what it writes, which matters for a WebSocket integration where the only real test is connecting.
- Cursor — a full editor built around the agent. Good when you want to watch diffs land in a familiar IDE and intervene by hand mid-task.
- GitHub Copilot — the most widely installed of the four, and increasingly agentic rather than just autocomplete. Usually the path of least resistance if your team already lives in GitHub.
- Windsurf — editor-based like Cursor, with a stronger emphasis on the agent keeping its own map of the codebase across a long task.
They differ in ergonomics more than capability. For the integration described here, any of the four will do the job. What determines whether the result works on the first run is not which agent you picked — it is whether the agent is reading current documentation or guessing from training data.
Visual builder or coding agent?
The honest version of this comparison is that visual builders are genuinely faster for the first hour and genuinely worse for the next six months.
A drag-and-drop voice agent builder gets you to a working demo quickly because it has already made every decision for you: which speech model, which LLM, which voice, how turns are detected, what happens on a barge-in. That is exactly why it stops being useful. The moment you need something the builder did not anticipate — a custom tool call against your internal API, different behaviour for a drive-thru than for a headset, a retry policy your ops team will actually sign off on — you are either filing a feature request or rebuilding the thing in code.
There is also a portability problem. A flow built in a vendor’s canvas lives in that vendor’s canvas. A flow built as code lives in your repository, gets code review, gets tested in CI, and gets deployed by the same pipeline as everything else you run.
A coding agent collapses the gap. You get the speed of describing what you want in English and the durability of owning actual source. The tradeoff you are accepting is that you have to be able to read the output — which is a real cost, and a smaller one than being stuck inside someone else’s flowchart.
Why this API is a good target for a coding agent
Coding agents fail in a specific way. They do not usually fail by writing nonsense. They fail by writing plausible code against an API surface they have half-memorized — inventing a field that sounds right, or reaching for a parameter that existed two versions ago. Two properties of the Voice Agent API reduce that risk more than anything else.
The minimum viable event loop is small. The full event surface is not tiny — there are seven client-to-server events and fourteen server-to-client events, and you will eventually care about most of them. But you do not need most of them to ship. A working agent is: send session.update, wait for session.ready, stream input.audio, and handle transcript.user, reply.audio and reply.done. That is the whole first version. Barge-in handling, session resume, mid-call reconfiguration and injected context are all opt-in additions on top of a loop that already works.
That matters for a coding agent because the failure mode is a half-finished state machine. When the path to a working call is six events long, the agent gets to “it runs” quickly, and everything after that is an incremental change against something you can already hear.
Most of the configuration does not live in your code at all. The documented recommendation is to create a stored agent — your system prompt, greeting, voice, tools and transcription settings held server-side — and then connect to it by ID. Your client code shrinks to a connection, an audio pump and an event loop. That is less code for an agent to get wrong, and it means changing your agent’s behavior does not require redeploying your application.
Start with a stored agent
From the session-configuration docs, verbatim: “Most agents should create a stored agent instead.” Storing the configuration server-side keeps secrets off the client, lets AssemblyAI run HTTP tools for you rather than round-tripping through your process, and lets you deploy the same agent across the API, a browser and a Twilio phone number by agent_id.
A stored agent is a JSON body posted to https://agents.assemblyai.com/v1/agents. The minimal one is four fields:
{
"name": "Order status agent",
"system_prompt": "You are a support agent for an online retailer. Keep every reply to one or two short sentences. Lead with the answer. Never invent an order status - call lookup_order and read back what it returns. No exclamation marks.",
"voice": { "voice_id": "anna" },
"greeting": "Hey, what can I do for you?"
}Then you bind it on your first session.update and send nothing else:
{
"type": "session.update",
"session": { "agent_id": "7ad24396-b822-4dca-871a-be9cc4781cf9" }
}agent_id is only valid on the first session.update, and it is mutually exclusive with the inline fields — sending both in one message returns a session.error with code agent_id_not_first.
Inline configuration is the alternative, and it is the right choice when the agent is genuinely dynamic or one-off. The field reference is identical either way, which is why the rest of this post shows the inline shape: it is the same set of fields you would put in a stored agent, just travelling over the socket.
Run your own audio through the model that powers the Voice Agent API before you write a line of integration code. No setup, no signup flow to get through.
The real event flow
Before any code, this is the sequence you are coding against:
connect
→ session.update (config, or just agent_id)
← session.ready (save session_id; only now send audio)
→ input.audio (stream) (base64 PCM16 mono 24 kHz)
← input.speech.started
← transcript.user.delta (full text so far - replace, do not concatenate)
← input.speech.stopped
← transcript.user (final)
← reply.started
← reply.audio (base64 chunk in `data`)
← transcript.agent.delta
← transcript.agent
← reply.done (status: completed | interrupted)
tool flow:
← tool.call (arguments already a dict)
← reply.done ← send tool.result here
→ tool.result (result must be a JSON *string*)
teardown:
→ session.end
← session.endedThree things in that diagram cause most first-run bugs. Do not send input.audio before session.ready. transcript.user.delta carries the full transcript so far rather than an increment, so render the latest and never concatenate. And tool.result goes out when reply.done is the most recent event you have seen — not the instant your tool returns, which may be mid-turn.
A working Python example
Connecting and configuring
import asyncio
import base64
import json
import os
import websockets
API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
AGENTS_WS_URL = "wss://agents.assemblyai.com/v1/ws"
LOOKUP_ORDER = {
"type": "function", # required, always "function"
"name": "lookup_order",
"description": (
"Get the status and estimated ship date for a customer order. "
"Use whenever the caller gives or asks about an order number. "
"Returns status (string) and ships_on (ISO-8601 date)."
),
"parameters": {
"type": "object",
"properties": {
"order_number": {
"type": "string",
"description": "Digits only, no dashes, e.g. 41820933.",
}
},
"required": ["order_number"],
},
"execution_mode": "interactive",
"timeout_seconds": 120,
}
SESSION_UPDATE = {
"type": "session.update",
"session": {
"system_prompt": (
"You are a support agent for an online retailer. Keep every "
"reply to one or two short sentences. Lead with the answer. "
"Never invent an order status - call lookup_order and read "
"back what it returns. No exclamation marks."
),
"greeting": "Hey, what can I do for you?",
"tools": [LOOKUP_ORDER],
"input": {
"format": {"encoding": "audio/pcm"}, # 24 kHz PCM16 mono
"transcription_mode": "balanced", # or min_latency / max_accuracy
"transcription_prompt": (
"A customer service call about order status. Expect order "
"numbers read as digit strings and UK place names."
),
"keyterms": ["Assembly", "expedited", "backorder"],
"voice_focus": "near-field", # hyphen, not underscore
"voice_focus_threshold": 0.85, # 0.0-1.0, default 0.85
# Note: min_silence / max_silence are deliberately NOT set here.
# Left unset they are adaptive, and the agent slows down on its
# own to capture entities like phone numbers. See below.
"turn_detection": {
"interrupt_response": True,
},
},
"output": {
"voice": "alba",
"format": {"encoding": "audio/pcm"},
"volume": 100,
},
},
}Details worth internalizing, because they are where generated code tends to go wrong.
Auth is Authorization: Bearer <API_KEY>. Note the contrast with the Dictation API, which takes the raw key with no Bearer prefix. If your coding agent has seen both, it will mix them up.
voice_focus takes near-field or far-field with a hyphen. near-field is the default and suits headsets and phone handsets; far-field is for rooms, kiosks and drive-thrus. voice_focus_threshold sets the strength from 0.0 to 1.0 and defaults to 0.85 on this API — the streaming speech-to-text API uses the same parameter with a 0.7 default, so do not carry a value across. Both are set at connect; a mid-session change is accepted but only takes effect on the next speech-to-text reconnect.
Some fields are immutable after session.ready. greeting, output.voice and output.format are locked once the first update is applied; changing them raises immutable_field. system_prompt, turn_detection, keyterms, transcription_mode, transcription_prompt, output.volume and tools can all be changed mid-call.
Leave the silence windows alone. This is the single most common way a generated config makes an agent worse. min_silence and max_silence have no fixed default — left unset, the agent paces turn-taking adaptively and automatically slows down when it is collecting a value your tools need, like a phone number or an email address. The docs are explicit that setting either one turns that adaptive, entity-aware behaviour off for the rest of the session. If you are tempted to widen the window so the caller can finish reading out an order number, the default already does that for you, and setting it manually is what breaks it. The documented way to improve turn-taking is better tool descriptions, not VAD knobs. Treat the raw thresholds as a last resort, and if you do reach for them, the docs’ own override example uses 1000 / 3000.
The event loop
async def send_audio(ws, mic_queue):
"""Stream base64 PCM16 mono 24 kHz. Real time, not faster."""
while True:
chunk = await mic_queue.get() # ~50 ms of int16 PCM
await ws.send(json.dumps({
"type": "input.audio",
"audio": base64.b64encode(chunk).decode(),
}))
async def run(ws, speaker):
last_event = None
pending_tools = []
async def flush_if_idle():
# tool.result goes out only when reply.done is the latest event.
if last_event != "reply.done" or not pending_tools:
return
for tool in pending_tools:
await ws.send(json.dumps({
"type": "tool.result",
"call_id": tool["call_id"],
"result": json.dumps(tool["result"]), # a JSON *string*
}))
pending_tools.clear()
async for raw in ws:
event = json.loads(raw)
etype = event.get("type")
if etype == "session.ready":
session_id = event["session_id"] # save it for session.resume
asyncio.create_task(send_audio(ws, mic_queue))
elif etype == "transcript.user":
print("user:", event["text"])
elif etype == "reply.audio":
speaker.write(base64.b64decode(event["data"]))
elif etype == "transcript.agent":
print("agent:", event["text"])
elif etype == "tool.call":
result = await TOOLS[event["name"]](**event["arguments"])
pending_tools.append({"call_id": event["call_id"], "result": result})
await flush_if_idle() # may already be idle
elif etype in ("reply.started", "input.speech.started"):
last_event = etype
if etype == "input.speech.started":
speaker.abort(); speaker.start() # flush on barge-in
elif etype == "reply.done":
last_event = etype
if event.get("status") == "interrupted":
pending_tools.clear() # agent moved on
speaker.abort(); speaker.start()
else:
await flush_if_idle()
elif etype == "session.error":
print("error:", event["code"], event["message"])
async def main():
headers = {"Authorization": f"Bearer {API_KEY}"}
async with websockets.connect(AGENTS_WS_URL, additional_headers=headers) as ws:
await ws.send(json.dumps(SESSION_UPDATE))
try:
await run(ws, speaker)
finally:
# Just closing the socket keeps the session - and the billing -
# alive for 30 seconds. session.end stops both immediately.
await ws.send(json.dumps({"type": "session.end"}))
asyncio.run(main())That is a complete agent. The full Voice Agent API documentation covers what to add next; the pieces most people reach for first are session resume and barge-in tuning.
Two behaviors worth designing on purpose
Tool failures. In a text application an exception becomes a retry. In a voice application it becomes dead air, which the caller reads as the call dropping. Return a structured error, and make it specific — the model reads the error text verbatim. “Lookup failed” makes the agent re-ask for everything. “Could not resolve order 41820933; the account lookup succeeded. Ask the caller to re-read just the order number” gets a clean recovery.
Disconnects. If the socket drops unintentionally, the session is held for 30 seconds — reconnect and send session.resume with the saved session_id and the conversation context survives. If the call is genuinely over, send session.end instead. That distinction is also a billing decision: the 30-second grace window is billable, and session.end stops the meter immediately.
Test the whole loop before you decide anything: the free tier is 185 hours of pre-recorded transcription plus 333 hours of streaming. No card, no sales call.
Pointing your coding agent at current documentation
The single highest-leverage thing you can do is stop the agent guessing. AssemblyAI publishes a docs MCP server; for Claude Code:
claude mcp add --transport http --scope user assemblyai-docs https://mcp.assemblyai.com/docsCursor, Windsurf, Codex and any other MCP client take the same server. For prompt-based builders like Lovable or v0, paste the docs URL into the prompt instead. Then describe what you want:
Using the AssemblyAI Voice Agent API, create a stored agent that handles
order-status calls, then give me a single-page browser app that connects
to it by agent ID and lets me talk to it, with echo cancellation on.
Read the events reference and session configuration pages before writing
code and use the exact event and field names documented there. Do not
infer any field. If something is not in the docs, ask me.
The agent should call a lookup_order tool against our internal API.
Handle barge-in: flush the playback buffer on input.speech.started.
Send session.end on intentional disconnect so billing stops.
Leave min_silence and max_silence unset - the defaults are adaptive.We maintain a full system prompt for exactly this — encoding the non-obvious gotchas, the browser audio constraints and a docs map — at Build with AI coding tools. There is also an llms-full.txt firehose dump of the entire Voice Agent section if your tool prefers one large file, and two starter repos (Python and JavaScript) that both ship an AGENTS.md your coding tool will read for repo conventions. Neither has dependencies to install.
If you take one thing from that system prompt, take the browser audio constraints, because they cause the most confusing class of bug:
navigator.mediaDevices.getUserMedia({
audio: {
echoCancellation: true, // ON. Stops the agent interrupting itself.
noiseSuppression: false, // OFF. Voice Focus runs server-side.
autoGainControl: true,
},
});noiseSuppression: false is the counterintuitive one. Browser noise suppression and server-side Voice Focus are independent passes, and running both eats real speech and degrades recognition in noisy rooms.
Where the accuracy levers are
Most of the tuning surface on a voice agent is not in the system prompt. It is in what the speech model knows about the moment it is transcribing.
On this API the two levers are session.input.transcription_prompt — up to 1750 characters describing the situation, mutable mid-call — and session.input.keyterms, up to 100 exact strings to bias toward. Pair keyterms with any lookup tool you ship: if you have a lookup_company tool, push the candidate company names into keyterms so the transcript does not mangle a name before your tool ever sees it. Keyterms are mutable, so narrow the list as the conversation narrows.
Contextual prompting of this kind is well measured on the underlying streaming model. Across a benchmark of 10,000+ voice agent audio files, supplying the agent’s own spoken reply as context to Universal-3.6 Pro Realtime cut word error rate by 8.9%, rising to 16.4% when a context prompt is added on top. The published per-class breakdown is measured on that combined configuration: place-name entities down 30.7%, fabrications down 27.0%, medical entities down 26.2%, name entities down 21.8%, short-utterance errors down 20.5%, and entity errors overall down 11.1%. On a separate public benchmark of 20,000 real voice-agent calls, detailed context cut medical-term entity errors by 43% and scenario-level context by 24%. The prompting and keyterms documentation has the methodology.
The third lever is transcription_mode: min_latency, balanced (the default) or max_accuracy. Stay on balanced until you have measured that you are not using the accuracy.
What the speech layer actually scores
The Voice Agent API’s speech recognition is Universal-3.6 Pro Realtime. Here is how it performs on AssemblyAI’s English voice-agent benchmark, run against 12,460 scripted voice-agent scenarios rather than read audiobooks. Lower is better throughout.
| Metric | Universal-3.6 Pro Realtime | Deepgram Flux EN | ElevenLabs Scribe v2 | Deepgram Nova-3 |
|---|---|---|---|---|
| Word error rate | 5.19% | 13.50% | 7.78% | 8.64% |
| Entity error rate | 14.4% | 30.1% | 18.5% | 26.1% |
| Names | 10.9% | 29.0% | 14.8% | 24.3% |
| Codes / IDs | 10.0% | 46.1% | 12.0% | 27.6% |
| Phone numbers | 2.4% | 11.5% | 3.4% | 4.5% |
The entity rows are the ones to read if you are building anything transactional. Aggregate word error rate tells you how the transcript reads; entity error rate tells you whether the agent got the order number, the street, the name and the callback number right. A 14.4% entity error rate versus 30.1% is the difference between an agent that can take an address and one that cannot. Methodology is in the Universal-3.6 Pro Realtime research post.
Independent benchmarks point the same way. On Pipecat’s open STT benchmark, Universal-3.6 Pro Realtime posts a 0.96% pooled semantic word error rate, with the final transcript landing a median 307 ms after the user stops speaking, and Coval’s STT benchmark shows it with the lowest word error rate, at 2.2%.
End to end, a full turn — caller stops speaking, agent starts speaking — lands around one second. That budget covers transcription, the language model and speech synthesis together, so it is not comparable to a per-layer number from a transcription-only vendor. When you compare, measure the same span on both.
What the flat rate covers
The Voice Agent API is $4.50/hr, or $0.075/min, billed per second. From our pricing page, verbatim: “Every feature is included in the $4.50/hr rate. There are no per-layer add-ons, concurrency fees, or per-agent subscriptions.”
Concretely: the speech-to-text, the LLM and the text-to-speech are in the number. There are no minimums and no commitment. You are not modeling token usage against three vendors’ price sheets to find out what a call costs — you multiply minutes by $0.075.
This also removes a class of coding-agent error that is easy to miss until the invoice arrives. When billing is token-based across several providers, an agent optimizing for “make it work” will happily ship something that streams the full conversation history to a model on every turn. On a flat per-second rate, that decision costs nothing extra. The one billing detail worth putting in your code review checklist is the opposite case: closing the socket without session.end leaves a 30-second billable grace window on every call.
Where to go from here
If you want to hear behavior before writing code, the playground will run your own audio through the speech layer. For the long-form build, the Voice Agent API build guide walks through a complete agent, and the launch post covers the design decisions. For where voice agents are being deployed and what breaks in production, AI voice agents is the overview. The two reference pages you will keep open while building are the events reference and session configuration.
Self-hosted deployment, EU data residency, or a Business Associate Addendum for PHI — talk it through before you build rather than after.
Frequently asked questions
Do I need to know how to code to build a voice agent with a coding agent?
No, but you need to be able to read code well enough to notice when the agent has gone somewhere strange. The skill that actually matters is specification — describing precisely what the agent should do, what it must never do, and which tools it can call. If you can write that clearly, Claude Code, Cursor, GitHub Copilot or Windsurf can write the integration.
Why use a coding agent instead of a visual voice agent builder?
Because you own the output. A visual builder is faster for the first hour and worse for the next six months: the moment you need a custom tool call, different turn-taking behaviour, or a retry policy your ops team will sign off on, you are filing a feature request. Code built by a coding agent lives in your repository, goes through code review, and deploys on your existing pipeline.
How do I stop my coding agent inventing API fields?
Connect it to the live documentation instead of letting it work from training data. Add the AssemblyAI docs MCP server at https://mcp.assemblyai.com/docs, then instruct the agent to read the events reference and session configuration pages before writing any code and to ask rather than infer. The Build with AI coding tools page has a full system prompt that encodes the non-obvious gotchas.
How many events do I actually have to handle to get a working agent?
Six, to get a first call working: send session.update, wait for session.ready, stream input.audio, then handle transcript.user, reply.audio and reply.done. The full surface is larger — seven client-to-server events and fourteen server-to-client — but barge-in handling, session resume, mid-call reconfiguration and injected context are all additions on top of a loop that already works.
Should I tune min_silence and max_silence for entity capture?
No — leave them unset. With no turn_detection config the agent paces turn-taking adaptively and slows down on its own when it is collecting a value your tools need, such as a phone number or an email address. Setting either field switches that adaptive, entity-aware behaviour off for the rest of the session, which is the opposite of what most people are reaching for it to do. The documented way to improve turn-taking is to write better tool descriptions. Treat the raw thresholds as a last resort.
What does the Voice Agent API cost?
$4.50/hr, or $0.075/min, billed per second. From our pricing page: “Every feature is included in the $4.50/hr rate. There are no per-layer add-ons, concurrency fees, or per-agent subscriptions.” There are no minimums and no commitment. One thing to watch: closing the WebSocket without sending session.end holds the session for a billable 30-second resume window.
How accurate is the speech recognition in a real conversation?
Universal-3.6 Pro Realtime posts a 5.19% word error rate and a 14.4% entity error rate on AssemblyAI’s English voice-agent benchmark of 12,460 scripted voice-agent scenarios. The closest comparison in that run is ElevenLabs Scribe v2 at 7.78% WER and 18.5% entity error rate. For transactional agents the entity numbers matter more than the aggregate — they determine whether the order number and the address came through.
