ElevenLabs' ElevenAgents and AssemblyAI solve different parts of the voice-agent problem. ElevenAgents is a managed agents platform: you configure an agent in their interface, and their stack handles listening, reasoning and speaking. AssemblyAI is the speech layer you build on: Universal-3.6 Pro Realtime hears the caller, the Text-to-Speech API answers, and your orchestrator and LLM run everything in between. If your agent has to hear names, numbers and codes correctly at production scale, building on AssemblyAI gives you the more accurate input layer and full control over the rest.
If you want a model-against-model look at transcription alone, we cover that in Universal-3 Pro vs. ElevenLabs Scribe v2. This post is about the architecture decision: adopt a managed agents platform, or own the agent and buy the speech layer from a specialist.
The two approaches at a glance
ElevenLabs started as a text-to-speech company. Its agents platform, ElevenAgents (formerly Conversational AI), extends into a single managed platform that bundles speech input, reasoning and voice output. You configure agents in their interface, use their conversation flows, and deploy inside their ecosystem.
AssemblyAI's position is different: we're not the agent; we're the platform you build on. We supply the speech layer at both ends, and you own the orchestration and the LLM. Universal-3.6 Pro Realtime, our flagship streaming speech-to-text model, drops into LiveKit, Pipecat and Vapi. The Text-to-Speech API covers the voice. For reasoning, call any model directly or route through LLM Gateway, one API to GPT, Claude, Gemini and other models.
| Dimension | AssemblyAI (Universal-3.6 Pro Realtime + Text-to-Speech API, with your orchestrator) | ElevenAgents |
|---|---|---|
| What it is | Speech layer (speech-to-text and text-to-speech) you build on | Managed agents platform |
| Speech-to-text model | Universal-3.6 Pro Realtime | Scribe v2 |
| Voice-agent WER (AssemblyAI benchmark, lower is better) | 5.19% | 7.78% |
| Entity error rate | 14.4% | 18.5% |
| Where the agent logic lives | Your orchestrator: LiveKit, Pipecat, Vapi or your own loop | Inside the platform |
| LLM | Any model, called directly or via LLM Gateway | Platform-hosted models or your own LLM endpoint |
| Concurrency | No cap on open sessions; new-session rate limits auto-scale | Capped by plan (4 to 40 concurrent calls; burst up to 3× at double the rate) |
| Pricing shape | $0.45/hr base streaming STT, per second; LLM and TTS on separate lines | Plan subscription with included minutes, per-minute overage, plus LLM usage |
| Languages (speech-to-text) | 32, with automatic detection and mid-sentence code-switching | Check ElevenLabs' current docs |
| Contextual prompting | agent_context and keyterms prompting | ASR keywords; no conversation-context input like agent_context |
| Medical terminology | Medical Mode + BAA available | No medical vocabulary mode; BAA on Enterprise only |
The core question: who owns the agent?
A managed platform owns the agent. That's real value early on: fewer decisions, a working demo sooner, and one vendor to call when something breaks. If you want something live this week and your requirements fit the platform's defaults, that's a fair trade.
The cost shows up later. A managed platform is opinionated about conversation design, which leaves less room to shape edge-case behavior, tool integration and flow. Need a support agent that switches languages mid-conversation? A retrieval step before every answer? An agent wired into a niche CRM, or a human handoff that carries full context? On a platform, you work within its boundaries, and when a requirement falls outside them, you're looking at a migration.
Building on a speech layer flips that. The system prompt, tools, conversation state and turn-taking rules live in your orchestrator, in code you can read and version. AssemblyAI does the two jobs a speech layer should do: turn audio into accurate text with reliable turn boundaries, and turn the reply back into speech built for conversation. Swap the LLM in one line. Add a tool without asking a vendor. Your agent is yours.
The honest framing: if time-to-first-agent outranks everything and you don't need deep customization, a platform can get you live faster. If you're building a product you'll own and evolve for years, building on a speech layer gives you room a platform won't.
Input accuracy: where agents succeed or fail
A voice agent that mishears a prescription number or an email address fails the task, whatever it says next. Input accuracy is the foundation every other layer depends on, and it's the layer you can't fix downstream: no prompt recovers a word the speech model never heard.
On AssemblyAI's English voice-agent benchmark (12,460 scripted voice-agent scenarios, not clean read speech), Universal-3.6 Pro Realtime posts a word error rate of 5.19% versus 7.78% for ElevenLabs Scribe v2. On entities, the tokens that decide whether an order or a callback goes through, the gap holds:
Input-side accuracy (AssemblyAI English voice-agent benchmark, lower is better)
| Metric | Universal-3.6 Pro Realtime | ElevenLabs Scribe v2 |
|---|---|---|
| Word error rate | 5.19% | 7.78% |
| Entity error rate | 14.4% | 18.5% |
| Names | 10.9% | 14.8% |
| Codes / IDs | 10.0% | 12.0% |
| Phone numbers | 2.4% | 3.4% |
That's roughly 28% more entity errors for Scribe v2 on exactly the tokens a real conversation runs on. Names and dates are where the two models diverge most: Scribe v2's date error rate is 26.5% versus 18.6%, and its name error rate is 14.8% versus 10.9%. On the benchmarks page, Universal-3.6 Pro Realtime ranks 1st of 22 systems on the voice-agent suite, with the lowest entity error rate of the group.
For production use cases, customer support, phone agents, clinical workflows and order processing, the agent that captures “RX-7704132” or a spelled-out email correctly every time will outperform one that has to ask twice. We go deeper on why in choosing a speech-to-text API for voice agents.
Context: a lever platforms tend to hide
Voice agents don't process isolated utterances. They process conversations. When you own the pipeline, you can feed the speech model what it needs to hear each turn correctly.
- agent_context passes the agent's last question to the model, so it hears the caller's reply through that lens. Ask “what's your email address?” and the model leans toward an address instead of guessing at homophones. Across 20,000 voice-agent files, passing agent context cut word error rate by 10.2%.
- Keyterms prompting, included on streaming, biases recognition toward your product names, SKUs and customer vocabulary.
- Voice Focus (+$0.10/hr) isolates the caller's voice from background noise. It hears the caller, not the room.
This is a structural advantage of building on a speech layer. A managed platform that abstracts the transcription step away doesn't give you that lever, or gives you only the version its designers anticipated. See the voice agent best practices for how to wire each one.
Run a recording with names, emails and confirmation codes through Universal-3.6 Pro Realtime and see what your agent would hear.
Turn-taking: knowing when the caller is done
Turn detection decides whether your agent feels natural or interrupts people. Universal-3.6 Pro Realtime combines semantic context with voice activity rather than relying on silence alone, so a caller who pauses halfway through reading a card number isn't cut off. Because turn detection is built into the same model that transcribes, it draws on the same understanding of what was said.
The input side is fast, too. On Pipecat's open STT benchmark, Universal-3.6 Pro Realtime posts a 0.96% pooled semantic WER, and the final transcript lands a median 307 ms after the speaker stops. The more useful question than any single millisecond figure is whether the agent hears the caller correctly on the first try. Re-prompts and misheard entities cost far more real-world time than a few dozen milliseconds anywhere in the pipeline.
Scaling: concurrency ceilings vs. open sessions
This difference shows up fast at production scale. ElevenLabs caps concurrent calls by plan, from 4 on the free tier to 40 on Business, with burst capacity up to 3× the limit billed at double the rate. For a support operation fielding hundreds of simultaneous calls at peak, that ceiling isn't something you engineer around. It pushes you onto higher tiers or negotiated limits and constrains how you plan capacity.
AssemblyAI streaming puts no cap on open sessions. New sessions per minute are rate limited (5 on the free tier, 100+ on paid) and scale automatically, so traffic spikes are absorbed without manual intervention. If you're choosing infrastructure you'll grow with, hitting a concurrency ceiling six months in means a painful re-platform.
Pricing shape
ElevenAgents is sold as plan subscriptions with included minutes and per-minute overage, with LLM usage billed on top, which makes forecasting depend on both how much your agents talk and which tier you're on. Check ElevenLabs' pricing page for current figures.
Building on AssemblyAI breaks cost into visible lines. Universal-3.6 Pro Realtime is $0.45/hr base, billed per second, pay as you go with no minimums. Add-ons are priced individually: streaming diarization +$0.12/hr, Voice Focus +$0.10/hr, Medical Mode +$0.15/hr. LLM and text-to-speech usage appear as their own lines, so you can see which layer drives cost and tune it. Current numbers are on the pricing page, and the free tier includes up to 333 hours of streaming with no credit card.
| Dimension | AssemblyAI (speech layer + your orchestrator) | ElevenAgents |
|---|---|---|
| Pricing model | Per-second speech-to-text; LLM and TTS billed as separate lines | Plan minutes, overage and LLM usage |
| Concurrency ceiling | No cap on open sessions | 4 to 40 concurrent calls by plan (burst up to 3×) |
| Cost visibility | Per layer | Bundled by plan |
| Behavior at scale | Linear in usage | Tier changes as volume grows |
What about the voice?
The voice your caller hears matters, and building on a speech layer doesn't mean giving it up. AssemblyAI's Text-to-Speech API is built for conversation, not narration: it handles hesitations, restarts and interruptions, and reads digits and names correctly, which is what an agent reading back an order number or confirming an appointment needs. With it, speech in and speech out come from one vendor, on one account.
Because the orchestrator owns the loop, you're never locked into one voice. Keep the text-to-speech you already use and add AssemblyAI's speech-to-text on the input side, or run both ends on AssemblyAI. Either way, you can test changes one layer at a time.
Language coverage
Universal-3.6 Pro Realtime streams 32 languages with automatic language detection and mid-sentence code-switching, so a caller mixing Hindi and English, or Spanish and English, stays on one session. On code-switched speech it posts 7.20% WER versus 17.13% for Deepgram Nova-3 Multi. If you need the broadest possible language list for a niche market, check both vendors' current docs against your traffic.
Use case fit
Build on AssemblyAI when: your agent needs to capture entities accurately (account numbers, emails, addresses, medical terms); you want code-level control over prompts, tools and conversation behavior; you're planning for production scale without a concurrency ceiling; you want cost visible per layer; or you need Medical Mode for clinical vocabulary. AssemblyAI is considered a business associate under HIPAA and offers a Business Associate Addendum (BAA) that covered entities and their business associates can sign to process protected health information.
Choose ElevenAgents when: you prefer a fully managed platform and are early enough that time-to-launch outranks control; your use case fits the platform's conversation flows; or your deployment stays within its concurrency limits.
A useful frame: this isn't a question of which company is better, but who should own your agent. If it's you, and the agent has to hear callers correctly at scale, the speech layer plus your orchestrator is the stronger foundation. For a broader survey, see our guide to the best voice agent APIs.
Universal-3.6 Pro Realtime for listening, the Text-to-Speech API for speaking, and the orchestrator and LLM you choose. Start free with up to 333 hours of streaming.
Frequently asked questions
AssemblyAI vs. ElevenLabs for voice agents: which is better?
It depends on who should own the agent. ElevenAgents is a managed agents platform that's quick to launch within its defaults. AssemblyAI supplies the speech layer, Universal-3.6 Pro Realtime and the Text-to-Speech API, for teams that run their own orchestrator and LLM. For production agents that must capture names, numbers and emails correctly at scale, building on AssemblyAI gives you more accurate input and full control over every layer.
Which is more accurate on the input side?
AssemblyAI. On AssemblyAI's English voice-agent benchmark, Universal-3.6 Pro Realtime posts a 5.19% word error rate versus 7.78% for ElevenLabs Scribe v2, and a 14.4% entity error rate versus 18.5%. On names it's 10.9% versus 14.8%, and on phone numbers 2.4% versus 3.4%.
Do I need an orchestrator to use AssemblyAI for voice agents?
You need something to run the loop: LiveKit, Pipecat, Vapi or your own code. Universal-3.6 Pro Realtime drops into LiveKit, Pipecat and Vapi as the speech-to-text layer, or you can stream directly to wss://streaming.assemblyai.com/v3/ws. The orchestrator holds your prompts, tools and conversation state, which is what gives you control over the agent.
How do concurrency limits compare?
ElevenAgents caps concurrent calls by plan, from 4 on the free tier to 40 on Business, with burst up to 3× at double the rate. AssemblyAI streaming has no cap on open sessions; new sessions per minute are rate limited (5 free, 100+ paid) and scale automatically, so peak traffic doesn't hit a ceiling.
How does pricing compare?
ElevenAgents uses plan subscriptions with included minutes, per-minute overage and LLM usage on top. On AssemblyAI, Universal-3.6 Pro Realtime is $0.45/hr base, billed per second with no minimums, and your LLM and text-to-speech costs are separate lines you can see and tune. Check each vendor's pricing page for current figures.
Can I use AssemblyAI speech-to-text with a different text-to-speech provider?
Yes. Because your orchestrator owns the loop, you can pair Universal-3.6 Pro Realtime on the input side with the text-to-speech you already run, or use AssemblyAI's Text-to-Speech API to get speech in and speech out from one vendor. You can change either layer without rebuilding the agent.
