Why AssemblyAI's Voice Agent API is designed for coding agents

Most voice agent tools were built around a specific assumption: that the right way to make something accessible is to put a UI in front of it.

Drag a node. Fill in a field. Pick a voice from a dropdown. Connect the boxes. It's a reasonable assumption, and for about a decade it was straightforwardly correct — a visual builder was how you got a non-specialist from zero to a working thing.

We built the Voice Agent API on a different one. The assumption is that the person building your voice agent is increasingly not a person clicking through a form. It's a coding agent, reading your docs and writing the integration.

That changes what "accessible" means. And once it changes, a lot of design decisions that used to look obvious stop looking obvious.

What a voice agent is

A voice agent is software you talk to and it talks back, in real time, in a conversation that goes somewhere.

That last clause is doing the work. A chatbot with a microphone bolted on isn't a voice agent. Neither is an IVR tree, which is a phone menu wearing a nicer voice, and neither is a transcription service, which converts speech to text and stops there. A voice agent listens, decides, acts against your systems, and speaks — and it holds enough of the conversation in mind to do that across many turns.

Where they're actually being used: appointment scheduling, lead qualification, clinical intake, customer support triage, sales training and roleplay, order taking, and outbound follow-up. What are AI voice agents goes deeper on the category if you want the wider tour.

The pipeline—what happens during every turn

Every turn of a voice conversation runs the same four steps.

Listen. Audio streams in and gets transcribed as it arrives, not after the person stops talking. This is streaming speech-to-text, and it's returning partial results within a few hundred milliseconds.

Detect the end of the turn. The system has to decide the person is done. This is much harder than it sounds. Naïve implementations count silence — wait 700ms, assume they're finished — which is why so many voice agents interrupt people mid-thought. Someone pausing to think and someone finishing a sentence produce the same silence. What separates them is the punctuation it predicts: a question ends differently than a trailing-off thought, and the transcript shows it. Universal-3.5 Pro Realtime reads those signals rather than running a stopwatch, at roughly 300ms.

Reason. The transcript plus the conversation history goes to an LLM. It decides what to say, and whether to call one of your tools first.

Speak. Text becomes audio and streams back, ideally starting before the full reply has been generated.

Four steps, roughly a second end to end, repeated fifty times in a five-minute call. And the genuinely hard part isn't any single step — it's what happens when the customer talks over the agent halfway through step four. Interruption handling is where voice systems reveal what they're actually made of.

How most of the industry built the pipeline—and why it creates problems

The standard approach assembles it. Speech-to-text from one vendor, the LLM from a second, text-to-speech from a third, turn detection from a fourth or from your own code.

Each of those is a good product. The problem is what happens between them.

Reliability compounds downward. Give every vendor a genuinely excellent 99.9% availability. Four of them in series is 99.6% before a single line of your own code has run. You didn't degrade anything. You just multiplied.

Latency compounds too, and unlike reliability it compounds visibly. Every hop adds network time, and the person on the phone is the one who experiences the sum.

Interruption becomes a distributed systems problem. When someone talks over the agent, you have to stop text-to-speech mid-utterance, discard the audio already buffered, tell the LLM what part of its reply actually got heard, and resume listening — across four services that have no idea the others exist. This is where most voice agent projects lose their weeks. The production ceiling covers what this looks like when a stack meets real traffic.

And then there's the operational tax. Four log streams to correlate when something goes wrong. Four billing relationships. Four sets of API credentials to rotate. Four status pages to check at 2am. None of that is technically hard. All of it is a permanent cost.

What AssemblyAI did differently, and why

We built the pipeline as one system instead of four.

One WebSocket connection, not four vendor integrations. One billing relationship at a flat rate per hour. One place to look when something goes wrong. Interruption handling that works because the component doing the speaking and the component doing the listening are in the same system and already know about each other.

Powered by Universal-3.5 Pro Realtime (universal-3-5-pro) as the speech recognition foundation — our streaming flagship, and the reason the accuracy claim holds up. On Pipecat's open STT benchmark, which runs on real agent conversations rather than clean read speech, it posts 6.99% pooled word error rate and a 15.31% entity error rate. The entity number is the one to watch if your agent handles account numbers, addresses, or names, because that's where transcription failures turn into wrong actions rather than awkward sentences.

And the API surface stays small on purpose. The Voice Agent API client handles about six event types. The OpenAI Realtime API exposes more than thirty.

That's not a knock on OpenAI's design — a larger surface gives you finer control, and there are applications that need it. But it's a real fork in the road. Thirty-plus event types is a system you have to study. Six is a system you can hold in your head, or hand to a coding agent that will hold it for you.

Pricing follows the same logic. Flat $4.50/hr, covering speech-to-text, the LLM, voice generation, turn detection, and tool calling. Not metered per audio token across three separate invoices you reconcile at the end of the month. You can forecast it before you build, which turns out to matter a lot when you're deciding whether a voice feature is worth shipping.

Context is a parameter, not a workaround

Here's the part of this that took us longest to get right, and it's the part that best explains what we mean by "built for coding agents."

Speech recognition models normally get one input: audio. Everything the system knows about the conversation — what was just asked, what the agent is expecting, what the domain is — lives somewhere else entirely, in the LLM layer, well downstream of the point where the words get decided.

Which is backwards, because that's exactly the information that would resolve the hardest audio.

Think about what a voice agent actually asks people. "What's the email on the account?" "Can you read me the confirmation number?" "Should I go ahead and place that?" The replies are short, often mumbled, frequently alphanumeric, and acoustically thin. A one-word "yeah" carries almost no signal. A spelled-out account ID carries the wrong kind.

Universal-3.5 Pro Realtime takes agent_context — you pass the agent's own question into the transcription request, and the model uses it to constrain what it's likely hearing. Across a benchmark of 20,000 voice agent audio files, that cut word error rate by 10.2%. The breakdown is more interesting than the headline: fabrications down 18.3%, hallucinations down 17.2%, place-name entities down 15.5%, short-utterance errors down 13.7%.

It also keeps a short rolling memory of the conversation, Context Carryover, on by default. You don't build a context buffer. You don't decide how much history to pass. It's there.

Now connect that back to the thesis.

When a coding agent builds on this API, it writes a system prompt. That prompt is plain English describing what the agent asks and what it's listening for. With agent_context, that same text is also what steers recognition — the description of the conversation and the mechanism that improves the conversation are the same artifact.

In a four-vendor stack, they can't be. Your speech-to-text provider has no access to your prompt; it's a different company. The best you can do is push a keyword list at it and hope. Which is fine, and we support keyterm prompting too — it's included, no add-on. But it's a workaround for a system that was assembled rather than designed.

agent_context and Context Carryover are both included in the flat rate. There's no line item for either.

Why the build interface is a coding agent, not a form

Here's the shift that reframes everything above.

For years, "make it accessible" meant "make it visual," because writing the integration by hand was the expensive part. Drag-and-drop builders existed to remove that cost, and they removed it well.

Coding agents removed it differently. Claude Code, Cursor, and their peers will read an API's docs and write a working integration in one pass — and they're better at a small, well-documented, JSON-over-WebSocket API than they are at almost anything else. The cost that visual builders were designed to eliminate has largely been eliminated somewhere else, by something that doesn't need a UI at all.

So the design target changes. We optimized for the coding agent as the reader: six event types, plain JSON, no SDK required, one endpoint, and docs written to be parsed as much as browsed. Paste your system prompt into a coding agent along with the docs and you can have a working voice agent the same day — the thing people mean when they say they vibe coded it.

This is the model Vapi built around, and it's worth saying they built it well. A polished visual builder is a real bet on a real user: someone who wants a working agent without opening an editor, and who is well served by not having to. That user still exists in large numbers, and for a lot of teams a platform that handles orchestration and telephony is straightforwardly the faster path.

Our bet is just about a different person — or increasingly, a different kind of collaborator. When the builder is a coding agent, an interface designed for human eyes is friction rather than help, and a small API surface beats a good UI every time.

The two bets also aren't mutually exclusive, which is easy to miss in a post framed as a contrast. Vapi has a native AssemblyAI integration: you can run Universal-3.5 Pro Realtime as the transcriber inside a Vapi agent and get the same speech accuracy underneath their orchestration. Here's how that's configured. Plenty of teams should do exactly that.

What this means for you as a builder

Practically, a few things follow.

The system prompt is your main control surface. Not a config screen, not a node graph. Behavior, tone, escalation rules, what to ask and when — it's all prose. Prose is reviewable in a pull request, diffable, and something a coding agent can edit competently. With agent_context, it's also load-bearing for accuracy, which is a nice property for something you were going to write anyway.

Noisy conditions behave better. A unified pipeline has more to work with than four services passing messages. voice_focus isolates the primary speaker and suppresses background speech — near-field for headsets and phones, far-field for rooms, kiosks, and drive-thrus. And you pick a mode rather than tuning thresholds: min_latency, balanced (the default), or max_accuracy.

Multi-speaker audio is handled. Live diarization labels speakers as they talk, then sends a single re-clustering correction within about half a second of the stream ending, for up to 10 speakers. Streaming diarization has the details.

The six terms worth knowing:

• Session — one connection, one conversation, from open to close.
• Turn — one exchange. Person speaks, agent responds.
• System prompt — the plain-English instructions that define behavior.
• Tool calling — the agent invoking your functions to look something up or do something.
• agent_context — the agent's own question, passed to the speech model so short replies resolve correctly.
• WebSocket — the persistent connection carrying audio and events both ways.

That's the vocabulary. There isn't more.

Where to go from here

If you want to build something, how to build with the Voice Agent API is the practical walkthrough, and the API reference is what you'd hand to a coding agent. If you're evaluating rather than building, the 2026 insights report on what actually makes a good voice agent is drawn from teams running these in production, and our contact center guidance covers the highest-volume deployment shape.

But the thing worth sitting with is this.

Every API is a bet about who's on the other end of it. Thirty event types is a bet on a specialist who will study the system. A drag-and-drop canvas is a bet on someone who won't open an editor. Six event types and plain JSON is a bet that the reader is a coding agent, and that the scarce resource is no longer implementation effort but the amount of system a builder has to understand before they can trust it.

What's new is that this bet is now checkable in an afternoon. Paste the docs into Claude Code and see how far it gets before it needs you. That's not a marketing exercise — it's a real evaluation criterion, and it's one that didn't exist eighteen months ago. Run it on us, and run it on everyone else too.

Frequently asked questions

What is AssemblyAI's Voice Agent API?

The Voice Agent API is a single WebSocket API for building voice agents, bundling streaming speech-to-text, an LLM, text-to-speech, turn detection, and tool calling into one connection at a flat $4.50/hr. It's built on Universal-3.5 Pro Realtime for speech accuracy and runs at roughly one second end to end. You connect to wss://agents.assemblyai.com/v1/ws, send a system prompt and a tool registry, and stream audio — no SDK required.

How is a unified pipeline different from wiring together separate vendors?

In an assembled stack, speech-to-text, the LLM, text-to-speech, and turn detection come from different providers that don't know about each other, so latency and downtime compound and interruption handling becomes a coordination problem across four services. A unified pipeline runs those stages as one system, which is why barge-in works without you writing the logic. It also collapses four log streams, four bills, and four credential sets into one of each.

What makes the Voice Agent API different from the OpenAI Realtime API?

The clearest difference is surface area: the Voice Agent API client handles about six event types, while the OpenAI Realtime API exposes more than thirty. A larger surface buys finer-grained control, so this is a tradeoff rather than a defect — but it changes who can build on it quickly, and how well a coding agent can generate a working integration from the docs. Pricing differs structurally too: the Voice Agent API is a flat hourly rate covering every stage, rather than metered per audio token across separate line items.

Do I need to use a coding agent to build with the Voice Agent API?

No. It's plain JSON over a WebSocket and you can write it by hand in any language, with no SDK involved. The API was designed on the assumption that a coding agent is often the one reading the docs, which is why the surface is small and the docs are written to be parsed — but that design makes it easier for people too. Small APIs are easier for everyone.

What use cases work well with voice agents?

Appointment scheduling, lead qualification, customer support triage, order taking, clinical intake, sales roleplay and training, and outbound follow-up are the most common. The pattern they share is a bounded task with a clear success condition and a handful of backend actions the agent needs to reach. If your use case involves account numbers, addresses, or names, entity accuracy is the metric to evaluate on — Universal-3.5 Pro Realtime posts a 15.31% entity error rate on Pipecat's open STT benchmark, against 50.50% for Deepgram Flux and 39.70% for ElevenLabs Scribe v2.

How does passing agent context improve accuracy?

agent_context lets you pass the agent's own question into the speech model, so short or mumbled replies resolve against what was actually asked. Across a benchmark of 20,000 voice agent audio files, it cut word error rate by 10.2%, with short-utterance errors down 13.7% and place-name entities down 15.5%. It's included at no extra cost, as is Context Carryover, the rolling conversation memory that's on by default.

How do I get started with the Voice Agent API?

Get a free API key from the dashboard, then open a WebSocket to wss://agents.assemblyai.com/v1/ws and send a session.update with your system prompt and tools. How to build with the Voice Agent API is the step-by-step walkthrough, and the fastest path is usually to hand the API reference to a coding agent and describe the agent you want. Billing is per second with no minimums and no commitment.

‍