Most writing about Voice AI for customer service is aimed at a buyer picking a vendor. This isn't that. This is for the engineer who has to make a support agent work on an actual phone line, where the caller is driving, reading out an order number, and expecting the thing on the other end to get it right the first time.
That job changed shape on April 14, 2026, when the Voice Agent API went GA.
What does a customer service voice agent actually need?
Three things. Speech recognition that survives phone audio and background noise. A model that can call your backend without inventing the arguments it passes. And infrastructure that behaves the same at 500 concurrent calls as it does at five.
Everything else — voice selection, conversation design, escalation rules, tone — is downstream of those three. If the agent hears "Order 8B-4471" as "Order AB-4471," none of the rest matters.
The old way to assemble this was three vendors: one for speech-to-text, one for the language model, one for text-to-speech. Three bills, three sets of logs, and three places for latency to hide. Debugging one bad turn meant correlating timestamps across three dashboards that didn't agree with each other. More on that failure mode in where voice agent stacks break down.
One WebSocket replaces three providers
The Voice Agent API handles speech-to-text, the LLM turn, and text-to-speech over a single WebSocket connection at a flat $4.50/hr, all in. One bill. One set of logs. One place to look when something goes wrong on a call.
It speaks standard JSON, with no SDK requirement — if your language can open a WebSocket, you can build on it. There are drop-in plugins for LiveKit and Pipecat, and it works natively with Claude Code, which turns out to be how a lot of teams now prototype their first agent.
You can reconfigure a session mid-conversation without reconnecting. If a call drops, the session resumes within 30 seconds with context preserved, so the customer doesn't re-explain their problem to a machine that just forgot it. End-to-end response lands around 1 to 1.3 seconds, inside the range where callers stop noticing they're talking to software.
Here's the part worth being explicit about: we're not the agent. We're the platform upon which you build. Conversation design, escalation logic, brand voice, when a human takes over — that's yours. We're the layer underneath that's supposed to be invisible.
Drop in a real support recording — accents, hold music, background noise and all — and see how the entities come through.
Can you trust it with an order number?
This is the objection every support engineering team raises, and it deserves a straight answer. The Voice Agent API applies guardrails to tool calling: the arguments the model is allowed to pass are constrained to what is actually inferable from the caller's turns and from prior tool results, rather than left to whatever the model finds plausible. That's the difference between an agent that occasionally issues a refund against the wrong order and one you can put in front of paying customers.
Underneath that sits the speech recognition, and the accuracy that matters for support isn't average word error rate — it's whether the entities survive. Here's the Pipecat open STT benchmark, run on real voice agent conversations. Lower is better, and every figure is on our public benchmarks page.
| Metric | Universal-3.5 Pro Realtime | Deepgram Flux | ElevenLabs Scribe v2 | Google Chirp3 |
|---|---|---|---|---|
| Word error rate | 6.99% | 15.58% | 9.76% | 9.04% |
| Entity error rate | 15.31% | 50.50% | 39.70% | 21.51% |
| Names | 16.92% | 39.21% | 38.03% | 22.10% |
| Places | 6.28% | 14.86% | 34.06% | 10.04% |
| Phone numbers | 3.55% | 10.41% | 4.78% | 4.95% |
Look at the entity rows rather than the headline WER. A support agent spends most of its life capturing exactly the things those rows measure: a name to look up an account, a city to check delivery, a phone number to send a confirmation text. A model that's respectable on average and poor on entities will feel broken to your customers even though its scorecard looks fine.
The per-session controls support teams end up needing
Real support traffic isn't uniform, so the speech layer is configurable per session — at connect, or mid-conversation through session.update, without dropping the call.
- agent_context passes the agent's own question into the recognizer, so when a caller answers "yeah, B as in boy" or spells out an email, the model resolves it against what was just asked. The published contextual-prompting benchmark, run on 20,000 real voice-agent calls, puts numbers on what that kind of context buys: a one-sentence scenario prompt cuts word error rate about 10% and name and place entity errors 16% and 21%, and a detailed prompt naming the people and identifiers in play takes name errors down 49%. Full figures in the streaming prompting documentation.
- Voice focus isolates the primary speaker. Use near-field for headsets and phones, far-field for rooms, kiosks, and drive-thrus. Anyone running support for a customer base that calls from cars and airports will want this on.
- A contextual transcription prompt primes the recognizer with your product names, SKU formats, and account-number patterns. There's no hard word budget you need to manage — write what the model needs to know about your domain.
- Transcription mode trades latency against accuracy per session, so a quick order-status flow and a complex troubleshooting call don't have to share one setting.
- language_codes steering narrows the model to the languages your queue actually receives, which raises accuracy compared with leaving detection wide open.
After a call ends, the sessions API returns the recording and the transcript for completed sessions, so QA and analytics don't need a parallel capture pipeline. And GET /v1/builtin-tools lists the tools available to an agent, so you can discover them programmatically instead of hardcoding a list that drifts. Full request and response shapes are in the API reference.
Drop in a real support recording — accents, hold music, background noise and all — and see how the entities come through.
How does this compare to the alternatives?
Against the OpenAI Realtime API: we run dedicated best-in-class models at each step of the pipeline rather than one multimodal model that also happens to do voice. That shows up in turn detection, which weighs whether what has been said reads as complete instead of waiting on silence, and it shows up on the invoice — we're roughly 4x cheaper. The developer experience is also simpler, since you're configuring a pipeline rather than negotiating with a single opaque model.
Against Retell and Vapi: those are platforms, and platforms have ceilings — their integrations, their conversation design opinions, and after a while agents that all sound like each other. That's a good trade if you want something running this week and your requirements sit inside what the platform anticipated. It's a bad trade if the support experience has to be yours. We're infrastructure: more of the design is your problem, and none of the ceiling is. Worth noting that Vapi is also a partner — plenty of teams run Vapi on top of our speech models.
If you're still choosing the speech layer specifically, we wrote a longer piece on choosing a speech-to-text API for voice agents.
Who's building customer experiences on this
Gorgias, the ecommerce CX helpdesk, and Nextiva, on the UCaaS and CX side, both build on AssemblyAI. So does Siro, which runs on our streaming speech-to-text.
What those three have in common isn't a use case — it's a shape. None of them wanted a support agent handed to them. They wanted the speech and reasoning layer to be reliable and boring so they could spend their engineering time on the part their customers actually see.
Where to start
Open a session, point it at recorded audio from your own queue, and check entity accuracy on the calls you care about — order numbers, account IDs, names. That takes an afternoon and tells you more than any vendor comparison will. For a walkthrough with code, start with building with the Voice Agent API.
If you're building broader capability around the agent — QA scoring, sentiment, compliance review — the contact center solutions page covers what sits alongside it, and pricing has the full per-hour table.
Here's the thing that's changed, and it's easy to miss. For years the hard part of a customer service agent was the agent — the dialogue tree, the fallback logic, the endless tuning of when to escalate. Modern language models absorbed most of that. What's left as the genuinely hard part is the plumbing underneath: does it hear the order number, does it call the right function with the right arguments, does it stay up. Which is a much better problem to have, because plumbing is something you can measure.
If you're moving a live support queue onto voice agents, our team can walk through concurrency, deployment, and data residency with you.
Frequently asked questions
How does AssemblyAI handle large-scale enterprise conversation processing?
Pricing is pay-as-you-go and billed per second with no minimums, so a queue that spikes from 50 to 5,000 simultaneous calls doesn't need a contract change. Enterprise deployments can run in our Voice AI Cloud, self-hosted inside your own AWS VPC, or on EU infrastructure where data stays in the EU at the same price. Completed sessions return recordings and transcripts through the sessions API for downstream analytics.
How do I build a voice agent with function calling?
You define your tools, and the agent calls them over the same WebSocket that carries the audio. Tool-parameter guardrails constrain what the model can pass, limiting arguments to what is inferable from the caller's turns and from prior tool results rather than letting the model fill a field with something plausible. You can also discover the available built-in tools programmatically with a GET request to the builtin-tools endpoint rather than maintaining a hardcoded list.
What insights can AssemblyAI extract from contact center conversations?
Beyond the transcript, the Speech Understanding API returns entity detection across 50+ types, sentiment analysis, topic detection, content safety flags, PII redaction in both text and audio, and audio event tagging. Those run on completed call recordings, which makes them a natural fit for QA scoring and compliance review. Speaker diarization separates agent from customer so per-speaker metrics are possible without a separate pass.
Can AssemblyAI integrate with existing contact center platforms?
Yes — the Voice Agent API speaks standard JSON over a WebSocket, so it drops into whatever telephony or CCaaS layer you already run without an SDK. There are ready-made plugins for LiveKit and Pipecat if you're using either as your orchestrator. Teams running Vapi can also use our speech models underneath it, since Vapi is a partner rather than a competing stack.
What happens when a customer says an account number and the agent mishears it?
The best fix is preventing it, which is what agent_context and contextual transcription prompts are for — passing the agent's question and your account-number format into the recognizer resolves most short, spelled-out, or alphanumeric replies correctly. On the published contextual-prompting benchmark of 20,000 real voice-agent calls, a scenario-level prompt cuts word error rate about 10% and name and place entity errors 16% and 21%, and a detailed prompt takes name errors down 49%. For the residual cases, design a confirmation step into the flow and offer DTMF entry as a fallback.
How much does it cost to run a voice agent for customer service?
The Voice Agent API is a flat $4.50 per hour covering speech-to-text, the LLM turn, and text-to-speech together, billed per second. There's no separate line item for the model provider or the voice, and no minimum commitment. If you'd rather assemble your own stack, the underlying realtime speech model is available on its own at $0.45 per hour.
