Best platforms for enterprise voice agents

Voice agent demos are easy. Almost suspiciously easy.

You can wire one up over a lunch break, hear it answer a question in a clean room on a good microphone, and walk into a review feeling great about it. Then it goes near production and meets its actual job: thousands of concurrent calls, telephony audio compressed to 8 kHz, a caller reading out a policy number in a car with the window down, and a security reviewer who wants to know where the audio is stored and who's signed what.

That's the gap this post is about. Not which platform has the nicest demo — which one survives the second week.

What enterprise voice agents actually need

Six requirements, and only two of them show up on a pricing page.

Speech accuracy on entities, not just words

Word error rate is the number every vendor leads with, and for enterprise agents it's close to useless on its own.

Here's why. WER averages across every token in the transcript, and most tokens are function words. Get "the" wrong and nothing happens. Get an account number wrong and the agent looks up the wrong customer, reads back the wrong balance, and does it all with total confidence — because the LLM downstream has no idea the input was corrupted.

Entity error rate is the number that predicts task completion. Account numbers, policy IDs, medication names, confirmation codes, street addresses, the letter-by-letter spelling of a surname. Those are the tokens your workflow actually branches on. A model with a slightly better WER and a much worse entity error rate will fail your users more often, not less. We've made this argument at length in word error rate is broken, and it's the single most consequential thing to get right in an enterprise evaluation.

Sub-second latency that holds under load

Everyone quotes latency from a benchmark run on a laptop with one stream open.

What matters is P95 at your concurrency, on your audio, with a tool call in the middle of the turn. A pipeline that averages 800ms with one session and spikes to three seconds at two thousand sessions isn't a sub-second pipeline. It's a sub-second demo.

Ask vendors for P95 under sustained concurrent load, not median in isolation. If they can't produce it, that's your answer.

Turn detection that works in real conversations

Most turn detection is a silence timer with a marketing name.

Real people pause mid-sentence to think. They say "um" and "uh-huh" and "right" while you're still talking, and none of that means "I'm done" or "stop talking." A voice-activity threshold fires on all of it. The result is an agent that interrupts people who were still speaking and sits silently through people who weren't.

Universal-3.5 Pro Realtime reads the punctuation it predicts rather than counting silence, and lands the end-of-turn decision at around 300ms. Three modes cover the range: min_latency for fast confirmations, balanced for normal conversation, max_accuracy for noisy and far-field audio. There's more on where naive pipelines break in the production ceiling.

Compliance that's actually enforceable

SOC 2 Type 2 is the floor, not the differentiator. Assume every serious vendor has it and move on to the questions that separate them.

For healthcare: can you get a signed Business Associate Addendum before any PHI touches the system, and how long does that take? AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) required under HIPAA to ensure PHI is appropriately safeguarded — signable in minutes, without a sales call.

For finance and anything regulated: audit trails, retention controls, and PII redaction in both the audio and the transcript.

For EU operations: where the audio is processed, not just where it's stored. AssemblyAI offers EU data residency at api.eu.assemblyai.com and streaming.eu.assemblyai.com, same price as US, with data staying in the EU. And when a review board won't accept any third-party processing, there's a self-hosted deployment into your own VPC.

Scalability without concurrency ceilings

This is the requirement that gets discovered in production, which is the worst possible time.

Some platforms cap concurrent sessions. Others meter billing against a concurrency tier you have to commit to in advance, which means your Black Friday costs you a renegotiation as well as a traffic spike. Either way, you find out during the event.

AssemblyAI has no concurrency caps and autoscaling rate limits, on pay-as-you-go pricing billed per second with no upfront commit. Ask every vendor on your list, in writing, what happens at 10x your normal volume on a Tuesday with no warning.

Mid-conversation flexibility

Enterprise agents change behavior mid-call. Routing to a specialist persona after intake. Tightening turn-detection thresholds when the caller is clearly reading a number aloud. Swapping the tool set once you know which product line you're in.

If that requires tearing down the connection and reconnecting, the caller hears it. Look for runtime updates to system prompt, tools, turn detection, and volume without dropping the session.

The major platforms compared

Accuracy figures below come from the Universal-3.5 Pro Realtime launch post, which reports Pipecat's open STT benchmark of real agent conversations, which is the only public benchmark we're aware of that measures entity accuracy on the kind of audio voice agents actually see. Where a vendor's stack hasn't been measured on it, the cell says so rather than carrying a number we can't source.

Lower is better in every row.

AssemblyAI Voice Agent API

One WebSocket at wss://agents.assemblyai.com/v1/ws handling speech-to-text, the LLM, and text-to-speech. Flat $4.50/hr, roughly one second end-to-end, standard JSON with no SDK required.

The speech layer is Universal-3.5 Pro Realtime — 6.99% pooled WER and a 15.31% entity error rate on Pipecat's benchmark. Tool calling is JSON Schema. Session resumption brings context back if a mobile connection drops within 30 seconds. Session config is mutable mid-call: system prompt, tools, turn detection, volume, all without reconnecting.

On the procurement side: SOC 2 Type 2, BAA available for customers processing PHI, EU data residency, self-hosted deployment for teams that need it, no concurrency caps, and no upfront commitment. Medical Mode adds clinical vocabulary accuracy with one parameter — domain: "medical-v1", +$0.15/hr — cutting missed medical entities by roughly 20% against the base model and the lowest MER across benchmarked providers. English, Spanish, German, and French. Details on the medical solutions page.

One thing that doesn't fit on a comparison table: forward-deployed engineers. For enterprise deployments we embed engineers with the customer's team through integration and scale-up. It's the difference between a support ticket queue and someone who knows your audio.

AssemblyAI Universal-3.5 Pro Realtime (bring your own stack)

Same model, sold on its own at $0.45/hr base from wss://streaming.assemblyai.com/v3/ws. This is the path for teams who already have orchestration they're not throwing away — LiveKit, Pipecat, Vapi, or something built in-house.

That's a real advantage worth naming. If you've built on an orchestration layer, you've invested in telephony routing, call flows, analytics, and a deployment story. Swapping the speech model underneath is a configuration change, not a rebuild. You keep the orchestration and upgrade what the agent hears.

The model's enterprise-relevant features: agent_context (pass the agent's own question in so short or mumbled replies resolve correctly), Context Carryover rolling memory on by default, voice_focus speaker isolation in near-field and far-field variants, live diarization with a single re-clustering correction at stream end for up to 10 speakers, keyterm prompting included, and 18 languages with mid-sentence code-switching. Add-ons stack as used: diarization +$0.12/hr, prompting +$0.05/hr, voice isolation +$0.10/hr, Medical Mode +$0.15/hr. More at streaming speech-to-text.

OpenAI Realtime API

An end-to-end multimodal model that hears and speaks in one pass. Broad language coverage and genuinely impressive when it works.

Two problems for enterprise. The first is cost: per-token billing, which is hard to forecast because it moves with conversation length and verbosity. The second is auditability. An end-to-end model doesn't hand you a transcript to inspect between hearing and responding, which means when it mishears a policy number you have no artifact showing what it thought it heard. For a regulated workflow that's not a performance issue, it's an evidence issue. The developer surface is also heavy — 30+ event types to handle.

Deepgram Voice Agent API

A cascaded architecture, so the transcript is there. Their realtime model, Flux, benchmarks at 15.58% WER and a 50.50% entity error rate on Pipecat's open STT benchmark — that entity number is the one to weigh, since it's more than three times ours on exactly the tokens enterprise workflows branch on.

Turn detection is traditional VAD. Mid-session reconfiguration is limited. And billing is metered against concurrency tiers, so scale planning becomes a commercial negotiation rather than a technical one.

ElevenLabs Conversational AI

Built outward from best-in-class text-to-speech, and it shows — the output quality is excellent. The input side is the weaker half: Scribe v2 posts 9.76% WER and a 39.70% entity error rate on the same benchmark.

The blocker for enterprise is structural, though. A published concurrency cap rules out many contact center and high-volume support deployments before accuracy even enters the conversation.

Why speech accuracy is the deciding factor

Errors don't stay where they start. That's the whole argument.

A voice agent is a chain, and the LLM has no way to know its input was wrong. It receives text, treats it as ground truth, and acts. If the transcription turned "RX-7704132" into "RX-770 for 132," the model doesn't hesitate — it confidently looks up a prescription that doesn't exist, tells the caller it can't find their record, and escalates a call that should have taken forty seconds.

That's the pharmacy case, and it generalizes. Insurance claim numbers. Order IDs. Dosage strings where "fifteen milligrams" and "fifty milligrams" differ by one phoneme and a great deal of liability. Bank account digits read out under road noise. In every one of them, the failure is invisible in the logs unless you're storing the transcript, and it's indistinguishable from a model quality problem unless you go listen to the audio.

This is also why context matters more than raw accuracy. agent_context lets you pass the agent's own question into the transcription request, so when the agent just asked for a date of birth, the model resolves an ambiguous string against that expectation rather than guessing in a vacuum. Across a benchmark of 20,000 voice agent audio files, passing agent context cut word error rate by 10.2% — fabrications down 18.3%, hallucinations down 17.2%, place-name entities down 15.5%, medical entities down 9.4%.

Gautam Pradeep, Tech Lead at Commure, framed the enterprise version of this well:

We've integrated the newest models from AssemblyAI for pre-recorded audio ASR in our ambient product, and it's been excellent. We're now exploring Universal-3.5 Pro for async and realtime speech-to-text capabilities for new use cases. What's been just as important is the reliability of the platform itself—both technically and in terms of partnership.

The two-path approach

You don't have to pick a lane permanently, and for enterprise that's more valuable than it sounds.

Path one: the managed Voice Agent API. One WebSocket, flat $4.50/hr, working agent the same afternoon. Right when the conversation is the product and you want to be in a pilot this quarter rather than next.

Path two: Universal-3.5 Pro Realtime on its own. $0.45/hr, your LLM, your voice, your orchestration. Right when you have an existing stack, a licensed voice, a fine-tuned model you're not giving up, or a use case that isn't a conversation at all — live call monitoring, compliance flagging, real-time captioning.

Both paths share the same Universal-3.5 Pro Realtime speech foundation, the same WebSocket-and-JSON conventions, and the same API key. So the accuracy validation you do in a pilot on one path carries over to the other. You're not re-running the evaluation because you changed your mind about orchestration — the thing you were measuring didn't change.

For recorded audio, the same generation covers it: Universal-3.5 Pro async at $0.21/hr, with the most accurate diarization we've shipped and native code-switching across 18 languages.

What procurement will ask you

You'll be asked to defend this internally, probably by someone who has never read a WER table. Have these ready:

• SOC 2 Type 2 — yes, and it's table stakes; don't let it be the whole answer.
• BAA — AssemblyAI is a business associate under HIPAA and signs a standard Business Associate Addendum. Signable without a sales call.
• Data residency — EU processing at api.eu.assemblyai.com and streaming.eu.assemblyai.com, same price as US. Self-hosted deployment into your own VPC when third-party processing is off the table entirely.
• PII and PHI handling — redaction across both audio and transcripts.
• Concurrency — no caps, autoscaling rate limits, no tier commitments. Get every other vendor's answer to this in writing.
• Commercial terms — pay-as-you-go, billed per second, no minimums and no upfront commit. Nothing to renegotiate when volume moves.
• Support model — forward-deployed engineers embedded with your team for enterprise deployments, not a ticket queue.

More on deployment options at enterprise.

Choosing the right platform for your team

Three questions, in this order.

Does it hear correctly on your audio? Not on a benchmark corpus — on recordings from your actual channel, with your accents, your vocabulary, your compression. Weight entity accuracy over word accuracy.

Can it clear your compliance review before the pilot ends? A platform you can't get a BAA for is a platform you can't use, regardless of how it benchmarks.

Does it break at your peak? Concurrency caps and tier commitments are the constraint that shows up last and hurts most.

Then, and only then, developer experience — which for voice agents means how fast someone on your team can get a real conversation running and start learning from it.

The failure mode nobody plans for

The enterprise voice agent projects that die don't usually die because someone picked the wrong platform. They die because nobody measured the platform's worst case before committing to it.

Every evaluation runs on clean audio, one session at a time, with the vendor's demo script. Every production deployment runs on compressed telephony, at peak concurrency, with a caller reading a sixteen-digit number in a moving car. Those are different systems, and the second one is the only one that matters — but it's the first one that gets benchmarked, budgeted, and signed off.

So flip the evaluation. Pull your fifty worst calls, the ones your current system already fails on, and run those. Test at your peak concurrency, not your average. Put a tool call in the middle of the turn and measure P95, not median. Ask each vendor to tell you their entity error rate and watch which ones change the subject to word error rate.

The platform that wins that evaluation is boring in the demo and reliable in production. For enterprise, that's the whole trade — and it's the right one, because your agent's ceiling isn't set by how good it is on a good day. It's set by what it does on the worst call of the week.

Frequently asked questions

What is the best API for building enterprise voice agents?

For most enterprise deployments, AssemblyAI's Voice Agent API is the strongest starting point: a cascaded STT, LLM, and TTS pipeline behind one WebSocket at a flat $4.50/hr, built on Universal-3.5 Pro Realtime, which posts 6.99% pooled word error rate and a 15.31% entity error rate on Pipecat's open STT benchmark of real agent conversations. If you already run your own LLM or orchestration, Universal-3.5 Pro Realtime is available as standalone streaming speech-to-text at $0.45/hr base. Both come with SOC 2 Type 2, BAA availability, EU data residency, and no concurrency caps.

How much does it cost to build an enterprise voice agent?

AssemblyAI's Voice Agent API is a flat $4.50/hr covering speech-to-text, the LLM, and text-to-speech; standalone streaming transcription is $0.45/hr base and pre-recorded transcription is $0.21/hr. OpenAI's Realtime API bills per token. The billing model matters as much as the rate — flat hourly pricing is forecastable, while per-token and concurrency-tier billing both move with variables you don't control, which is what turns a traffic spike into a budget conversation.

What compliance certifications should a voice agent platform have?

SOC 2 Type 2 is the baseline for any enterprise deployment. Healthcare requires a signed Business Associate Addendum (BAA) before any protected health information is processed — AssemblyAI is considered a business associate under HIPAA and offers a standard BAA, signable without a sales call. Also ask about PII and PHI redaction across both audio and transcripts, retention controls, and where audio is processed rather than just where it's stored.

Can I use my own LLM with AssemblyAI's voice agent infrastructure?

Yes. Universal-3.5 Pro Realtime is available as standalone streaming speech-to-text at $0.45/hr base, so you can pair it with your own LLM and text-to-speech through LiveKit, Pipecat, Vapi, or a custom pipeline. You get the same speech foundation the managed Voice Agent API runs on, including agent_context, Context Carryover, voice_focus speaker isolation, and live diarization. Swapping the speech layer under an existing orchestrator is a configuration change, not a rebuild.

Does AssemblyAI support EU data residency or self-hosted deployment?

Yes to both. EU data residency runs on api.eu.assemblyai.com and streaming.eu.assemblyai.com, priced the same as US, with data staying in the EU for GDPR purposes. For teams whose review boards won't approve third-party processing at all, AssemblyAI also offers self-hosted deployment into your own cloud environment — for example, a customer-managed AWS VPC.

What is the best one-API solution for voice agents?

AssemblyAI's Voice Agent API replaces three vendors with one WebSocket connection, one invoice measured in hours, and one set of logs, at a flat $4.50/hr. That covers speech-to-text, the LLM, and text-to-speech, with turn detection, barge-in, tool calling, and session resumption built in. It's positioned as infrastructure rather than an agent platform — you write the system prompt and the tool handlers, and your customers experience an agent that feels like you built it yourself.

How does turn detection work in enterprise voice agents?

Universal-3.5 Pro Realtime detects end-of-turn by reading the punctuation it predicts rather than waiting out a silence timer, landing the decision at around 300ms. That's what lets it distinguish a mid-sentence pause or a back-channel "uh-huh" from a genuine finished turn, so the agent neither talks over people nor sits silently waiting. Three modes cover the range of deployments — min_latency for quick confirmations, balanced as the default, and max_accuracy for noisy or far-field audio — and barge-in behavior is configurable per session.