If you're choosing between the Deepgram Voice Agent API and AssemblyAI, you're choosing between two shapes of product. The Deepgram Voice Agent API is a managed agent: one vendor runs listening, reasoning and speaking behind a single connection. AssemblyAI is the speech layer you build on: Universal-3.6 Pro Realtime hears the caller, the Text-to-Speech API answers them, and you run the loop in LiveKit, Pipecat or your own code with the LLM you pick. If your agent has to get names, numbers and codes right, the accuracy of the input layer decides it, and that's where the two stacks differ most.
We've already published a model-against-model look at the speech layer in AssemblyAI vs. Deepgram for voice agents, and a transcription-first view in our general AssemblyAI vs. Deepgram comparison. This post is about architecture: what you give up and what you get when one vendor runs your whole voice agent, versus owning the loop and buying the best speech in and speech out you can find.
The short version
A managed agent like the Deepgram Voice Agent API gets you to a first demo with the fewest moving parts. The tradeoff is that every layer comes from the same catalog, so its listening accuracy is bounded by Deepgram's own speech models. On AssemblyAI's English voice-agent benchmark (12,460 scripted voice-agent scenarios), Universal-3.6 Pro Realtime posts a 5.19% word error rate against 13.50% for Deepgram Flux EN and 8.64% for Nova-3. On entities, the gap is wider: 14.4% entity error for AssemblyAI versus 30.1% for Flux EN and 26.1% for Nova-3.
Building on AssemblyAI means a few more decisions up front: pick an orchestrator, pick an LLM (or route through LLM Gateway), and wire the voice. In return you control every layer, swap any one without rewriting the agent, and pay for each on its own line.
Managed agent vs. speech layer at a glance
| Dimension | AssemblyAI (Universal-3.6 Pro Realtime + Text-to-Speech API, with your orchestrator) | Deepgram Voice Agent API |
|---|---|---|
| What you get | Speech-to-text and text-to-speech from one vendor; you own orchestration and the LLM | Managed agent loop from one vendor behind one connection |
| Speech recognition | Universal-3.6 Pro Realtime | Deepgram's own models (Flux, Nova-3) |
| Voice-agent WER (lower is better) | 5.19% | 13.50% (Flux EN) / 8.64% (Nova-3) |
| Entity error rate | 14.4% | 30.1% (Flux EN) / 26.1% (Nova-3) |
| Names / codes and IDs / phone numbers | 10.9% / 10.0% / 2.4% | 29.0% / 46.1% / 11.5% (Flux EN) |
| Turn detection | Semantic context plus voice activity, built into the model | Turn-detection-led “conversational speech recognition” (Flux) |
| LLM | Any model, via your orchestrator or LLM Gateway | Chosen from what the managed agent supports; check Deepgram's docs |
| Orchestration | LiveKit, Pipecat, Vapi or your own loop | Runs inside the vendor session |
| Languages (realtime STT) | 32, with automatic detection and mid-sentence code-switching | Check Deepgram's current docs |
| Pricing shape | $0.45/hr base streaming STT, billed per second; LLM and TTS on their own lines | Tiered, usage-based pricing with concurrency charges |
| Healthcare | Medical Mode + BAA available | Check Deepgram's current terms |
Two architectures: a bundled agent vs. a speech layer you build on
Deepgram's voice-agent story starts from its speech recognition. Flux is marketed as “conversational speech recognition,” a model that leans heavily on turn detection to decide when the caller has finished speaking, and Deepgram's public accuracy claims stay vague, in the “>90%” range. The Deepgram Voice Agent API wraps that listening layer together with reasoning and voice output so you can stand up an agent against one endpoint.
AssemblyAI takes the other position: we're not the agent; we're the platform you build on. We supply the speech layer at both ends. Universal-3.6 Pro Realtime listens; the Text-to-Speech API, built for conversation rather than narration, handles hesitations, restarts and interruptions and reads digits and names correctly. Speech in and speech out come from one vendor. The orchestration and the LLM are yours.
Both approaches are cascaded: dedicated speech models feeding a text LLM, not one multimodal model doing everything. That's the right design for production, because the transcript feeds the model that decides what to do next. The question is who owns the seams between the layers, and how good the listening is.
Entity accuracy: the layer you can't fix downstream
Voice agents don't transcribe for the record. They transcribe to act. If the speech layer drops a digit in a phone number, the agent texts the wrong person. If it garbles a surname during identity verification, the call stalls. No prompt engineering in the LLM recovers a word the speech model never heard.
That's why the input model matters more in a managed agent than it first appears: you can't swap it. On AssemblyAI's English voice-agent benchmark, Universal-3.6 Pro Realtime ranks first, and on the benchmarks page it ranks 1st of 22 systems on the voice-agent suite with the lowest entity error rate, 14.41%. Against Deepgram's two real-time models, entity by entity:
| Metric (lower is better) | Universal-3.6 Pro Realtime | Deepgram Flux EN | Deepgram Nova-3 |
|---|---|---|---|
| Word error rate | 5.19% | 13.50% | 8.64% |
| Entity error rate | 14.4% | 30.1% | 26.1% |
| Names | 10.9% | 29.0% | — |
| Codes / IDs | 10.0% | 46.1% | — |
| Phone numbers | 2.4% | 11.5% | — |
Flux gets roughly twice as many entities wrong, and more than four times as many codes and IDs. This shows up in production, not just on benchmarks. HotelPlanner switched to AssemblyAI from Deepgram over credit-card and number accuracy on live calls, and saw a 10% lift in checkout conversion afterward. Accurate readback of payment details is the difference between a completed booking and an abandoned one.
Owning the speech layer also gives you knobs a bundled agent may not expose. Pass agent_context, the agent's last question, and the model interprets the reply through that lens: across 20,000 voice-agent files that cut WER by 10.2%. Add keyterms prompting (included on streaming) for product names and SKUs, and Voice Focus (+$0.10/hr) when callers are in cars, kitchens or call centers. It hears the caller, not the room.
Run a recording with names, phone numbers and confirmation codes through Universal-3.6 Pro Realtime, then compare the transcript to what your current agent heard.
Control: where your prompts, tools and state live
In a managed agent, the system prompt, tool definitions and conversation state live inside the vendor's session. That's convenient until you need something the session doesn't offer: a different LLM for one skill, a retrieval step before the model answers, a human handoff with full context, or a custom barge-in rule for a noisy line.
When you build on AssemblyAI, those live in your orchestrator. The speech layer does two jobs and does them well: turn audio into accurate text with reliable turn boundaries, and turn the reply back into natural speech. Everything between is code you can read. Universal-3.6 Pro Realtime drops into LiveKit, Pipecat and Vapi, or you can open a WebSocket to wss://streaming.assemblyai.com/v3/ws and run your own loop. Turn detection combines semantic context with voice activity, so a caller pausing mid-way through a card number isn't cut off.
The cost of control is assembly. Someone on your team picks the orchestrator, writes the tool-calling logic and owns the deploy. Frameworks like LiveKit and Pipecat do most of that work, and coding agents are good at the rest: we've written up how to scaffold one quickly.
Lock-in: what it costs to change one layer
Every voice agent eventually wants to change something. A better LLM ships. A customer needs Spanish and English in the same call. A healthcare deployment needs Medical Mode and a BAA.
With a bundled agent, changing one layer means checking whether the vendor supports it. If it doesn't, you're migrating the whole agent, prompts, tools and state included. With a speech layer plus an orchestrator, each layer is its own component. Swap the LLM in a config line. Turn on code-switching across 32 languages in the speech model without touching the rest. Keep your existing TTS and add AssemblyAI's later, or the other way around.
That modularity also helps with evaluation. You can run Universal-3.6 Pro Realtime next to your current speech-to-text on the same calls and compare transcripts before you change anything a caller hears.
Pricing shape
A managed agent prices the whole loop as one product, which keeps the invoice simple. Deepgram uses tiered, usage-based pricing and charges extra for concurrency. For a support line that spikes at peak hours, concurrency metering makes costs harder to predict and potentially more expensive exactly when you're busiest.
Building on AssemblyAI splits the bill into lines you can see. Streaming speech-to-text on Universal-3.6 Pro Realtime is $0.45/hr base, billed per second, pay as you go with no minimums. There's no cap on open sessions; new sessions per minute are rate limited (5 on the free tier, 100+ on paid) and scale automatically. Add-ons are priced individually: streaming diarization +$0.12/hr, Voice Focus +$0.10/hr, Medical Mode +$0.15/hr. LLM and text-to-speech usage are separate lines, so you know exactly which layer drives cost and can tune each one. Current numbers are on the pricing page, and the free tier includes up to 333 hours of streaming with no credit card.
Which should you choose?
Choose the Deepgram Voice Agent API if you want the shortest path to a working demo, you're already standardized on Deepgram, and your agent's success doesn't hinge on capturing names, numbers and codes.
Choose AssemblyAI if the agent has to get those entities right the first time, if you want to choose and change your LLM, or if you expect the agent to grow into something a single vendor's session wasn't designed for. The 5.19% vs. 13.50% WER gap and the 14.4% vs. 30.1% entity gap turn directly into calls that finish without a “sorry, can you repeat that?” For contact center agents taking payments or verifying identities, that's the deciding factor.
Either way, test it on your own audio. Build a quick agent on each and talk to it. Turn feel and entity capture are things you notice in the first thirty seconds. For a wider survey of options, see our guide to the best voice agent APIs.
Universal-3.6 Pro Realtime for listening, the Text-to-Speech API for speaking, and the orchestrator and LLM you choose. Start free with up to 333 hours of streaming.
Frequently asked questions
What's the difference between the Deepgram Voice Agent API and AssemblyAI?
The Deepgram Voice Agent API is a managed agent: one vendor runs speech recognition, reasoning and voice output behind one connection. AssemblyAI supplies the speech layer, Universal-3.6 Pro Realtime for speech-to-text and the Text-to-Speech API for voice output, and you run the agent loop in LiveKit, Pipecat, Vapi or your own code with the LLM you choose.
Which is more accurate on names and numbers?
AssemblyAI. On AssemblyAI's English voice-agent benchmark, Universal-3.6 Pro Realtime has a 14.4% entity error rate versus 30.1% for Deepgram Flux EN and 26.1% for Nova-3. On names it's 10.9% versus 29.0% for Flux EN, and on phone numbers 2.4% versus 11.5%.
Is a managed voice agent API faster to build with?
For a first demo, usually yes, because there are fewer components to wire. Building on AssemblyAI adds an orchestrator and an LLM choice, but LiveKit and Pipecat handle most of the plumbing, and Universal-3.6 Pro Realtime drops in as their speech-to-text. The extra setup buys control over every layer and the ability to swap one without rebuilding the agent.
How does pricing compare?
Deepgram uses tiered, usage-based pricing with concurrency charges. AssemblyAI's Universal-3.6 Pro Realtime is $0.45/hr base, billed per second with no minimums and no cap on open sessions; your LLM and text-to-speech costs appear as separate lines, so you can see and tune each layer.
Can I use AssemblyAI with LiveKit, Pipecat or Vapi?
Yes. Universal-3.6 Pro Realtime drops into LiveKit, Pipecat and Vapi as the speech-to-text layer, and you can also connect directly to the streaming WebSocket at wss://streaming.assemblyai.com/v3/ws. You keep your orchestrator, your LLM and your tool logic.
Does AssemblyAI handle multilingual callers?
Yes. Universal-3.6 Pro Realtime covers 32 languages with automatic language detection and mid-sentence code-switching, so a caller who switches from English to Spanish mid-sentence stays on the same session. On code-switched speech it posts 7.20% WER versus 17.13% for Deepgram Nova-3 Multi.
