Best voice agent API in 2026: 6 options compared for developers

The best voice agent API depends on how much of the stack you want to own. For developers building production agents, AssemblyAI supplies the full voice stack: Universal-3.6 Pro Realtime for streaming speech-to-text with turn detection built into the model, and the Text-to-Speech API for the voice, orchestrated with LiveKit, Pipecat, Vapi, or your own loop, with the LLM of your choice. OpenAI Realtime, Deepgram, ElevenLabs, Vapi, and Retell each fit different needs, compared below.

Most "best voice agent" lists rank options by how fast you get to a demo. That's useful, but it skips the question that decides production quality: does the agent hear your user correctly? A voice agent is a pipeline. Speech-to-text feeds the LLM, the LLM decides what to say, and the voice says it. If the first step gets a name or an account number wrong, everything downstream confidently answers a question the caller never asked.

So this guide compares six options on what holds up in production: accuracy on real agent audio, turn-taking, pricing model, concurrency, and how much control you keep. Every competitor price below comes from the vendor's public pricing page, checked October 28, 2026.

How the six options compare

Competitor pricing from public pricing pages as of October 28, 2026: OpenAI, ElevenLabs, Vapi, Retell. Carrier minutes are extra on every option.

One thing worth naming up front: LiveKit and Pipecat are open-source pipeline frameworks, not competitors on this list, and Vapi is both an option here and a place AssemblyAI runs. Universal-3.6 Pro Realtime drops into LiveKit, Pipecat, and Vapi, and the Text-to-Speech API has LiveKit and Pipecat integrations, so you can slot AssemblyAI in at both ends without rearchitecting anything.

What "best voice agent API" actually means

The LLM gets the attention, but it never hears your customer. It reads a transcript a speech model produced a few hundred milliseconds earlier and treats it as ground truth. If the speech model hears "Nguyen" as "when," the agent doesn't know it misheard. It just runs with it, and the caller has to repeat themselves.

A demo that works once in a quiet room tells you almost nothing. These are the criteria that separate an agent that ships from one that gets pulled after week one:

• Speech accuracy on real audio. Names, account numbers, accents, cross-talk, and background noise. Not clean read-aloud clips.
• Turn detection. Knowing when the caller has finished versus pausing mid-thought. Get it wrong and the agent interrupts or sits in dead air.
• Interruption handling. Stopping when the caller really cuts in, but not on every "mhm."
• Latency. Mouth to ear. Around a second keeps the conversation flowing.
• Billing model. How many meters, and how each one moves with call length. This decides whether you can price a call before you scale.
• Concurrency. Whether a Monday-morning spike hits a cap, a per-line fee, or neither.
• Developer experience. Plain protocols and readable docs, or an SDK and a conversation designer you have to bend your app around.

Weight these however your use case demands. But accuracy sits underneath most of them. Bad transcripts make turn detection harder, force retries that add latency, and create the "sorry, can you repeat that?" loops that wreck the experience.

Platform vs building blocks: which are you actually buying?

This is the real fork in the road. Platforms like Vapi and Retell give you an opinionated way to design a conversation: hosted orchestration, a dashboard, and often telephony built in. You give up control of each layer, and your flow has to fit the platform's conversation model.

Building blocks go the other way. You pick the speech layer, the LLM, and the orchestration, and you own the prompts, the tools, and the integration. That's the path if you're building something differentiated or if accuracy is load-bearing. The two aren't exclusive: you can run AssemblyAI as the transcriber inside Vapi, or as the speech-to-text and the voice under a LiveKit or Pipecat agent you control end to end. We broke down the day-to-day side of this in which voice agent platform has the best developer experience.

The six options, one by one

Each one optimizes for something different.

AssemblyAI: the full voice stack

AssemblyAI covers every speech job in the agent. Universal-3.6 Pro Realtime hears the caller and decides when they've finished talking, the Text-to-Speech API speaks the reply, and LLM Gateway is there if you want the LLM on the same account. Your orchestrator (LiveKit, Pipecat, Vapi, or your own loop) runs the conversation with whichever LLM you choose. Speech in, turn-taking, and speech out come from one vendor, and you keep control of everything in between.

What's included

• Universal-3.6 Pro Realtime streaming speech-to-text at $0.45/hr, with keyterms prompting included
• Turn detection inside the speech model: at each pause it checks whether what's been said reads as a complete turn, with min_latency, balanced, and max_accuracy presets
• agent_context: pass the agent's last reply so the model expects the answer to that question
• Mid-stream configuration updates for prompts, keyterms, turn timing, and agent_context, without reconnecting
• 32 languages with automatic language detection and mid-sentence code-switching
• Optional add-ons: Voice Focus voice isolation (+$0.10/hr) and streaming diarization (+$0.12/hr)
• The Text-to-Speech API (AssemblyAI-TTS-1), built for conversational voice agents and trained on millions of hours of audio, including real conversational data: frontier-level quality that runs faster and costs less than other production voice agent TTS models, unlimited concurrency, thousands of voices across six languages, instant voice cloning in the dashboard, one WebSocket session per conversation that starts speaking from streamed LLM text, and cancel() for barge-in
• LLM Gateway if you want one API to GPT, Claude, Gemini, and other models
• Telephony audio at both ends: pcm_mulaw at 8 kHz into speech-to-text, and μ-law or A-law out of text-to-speech

OpenAI Realtime

OpenAI built a multimodal model that also does voice. That's the framing to keep in mind. GPT-Live-1 adds another session shape alongside the Realtime API. The cost is a larger event surface to manage and billing that spreads across audio and text token meters, plus a per-minute session fee on GPT-Live-1 with backend model and tool usage charged separately. More in our roundup of OpenAI Realtime API alternatives.

Deepgram

Deepgram's Voice Agent API is a unified agent API built on its own speech models. The difference shows up in accuracy on real agent audio: on AssemblyAI's 12,460-scenario English voice-agent benchmark, entity error is 14.4% for Universal-3.6 Pro Realtime versus 26.1% for Nova-3 and 30.1% for Flux. The full breakdown is in our AssemblyAI vs Deepgram comparison.

ElevenLabs

ElevenLabs Agents is a hosted platform billed by call minutes on monthly plans, with LLM usage passed through separately. Concurrency is set by plan, from 4 to 40 concurrent calls, with burst capacity above that at double the per-minute rate. That's worth modeling if your traffic is spiky.

Vapi

Vapi is orchestration middleware. You configure assistants, bring the STT, LLM, and voice providers you want, and pay a $0.05/min hosting fee plus each provider's cost. Your agent's accuracy, latency, and cost are the sum of the providers you pick, plus the hosting fee. AssemblyAI is available as a Vapi transcriber, so you can keep Vapi's orchestration and put Universal-3.6 Pro Realtime underneath it.

Retell

Retell is a phone-first platform with built-in telephony and pricing that adds up by component, from $0.07 to $0.31 per minute. You work inside its conversation model, so flows that need something it doesn't expose have nowhere to go.

The best voice agent API for each use case

Best for accuracy-critical production agents: AssemblyAI with your orchestrator

When names, order IDs, and phone numbers decide whether the call succeeds, start with the most accurate realtime speech model, add the LLM you trust, and run it in LiveKit, Pipecat, Vapi, or your own loop.

Best for multimodal prototyping: OpenAI Realtime

If your product mixes voice with images or text in one model session, OpenAI's realtime models keep them in one session, billed on token meters that move with how much the call says.

Best for a hosted agents platform with bundled minutes: ElevenLabs Agents

Plan-based minutes and a managed builder. Concurrency is capped by plan, and calls above the cap bill at double the per-minute rate, so model your peak before you pick a plan.

Best for no-code phone agents: Vapi or Retell

Both host the agent and the conversation design. Vapi passes provider costs through on top of its hosting fee (AssemblyAI is available as the transcriber); Retell bundles telephony and prices by component.

Best for teams assembling their own pipeline: LiveKit or Pipecat with Universal-3.6 Pro Realtime

If you need a specific LLM or a cloned voice, own every layer in a framework and plug in Universal-3.6 Pro Realtime for speech-to-text and the Text-to-Speech API for the voice, including voices cloned in the AssemblyAI dashboard.

Best by vertical

• Startups shipping a first voice product: see the best voice agent API for startups.
• Support and contact-center automation: see the best voice agent API for contact centers.
• Patient intake and scheduling: see our healthcare voice agent guide. AssemblyAI offers a Business Associate Addendum (BAA) for customers processing PHI.

Choosing only the speech-to-text layer? That's a narrower decision, covered in how to choose a speech-to-text API for voice agents. This guide is about the whole stack.

Why speech accuracy is the tiebreaker

When two options both promise low latency and a nice developer experience, accuracy on real audio is what's left to choose on.

AssemblyAI's English voice-agent benchmark runs 12,460 scripted caller scenarios voiced by real speakers across three noise environments. Universal-3.6 Pro Realtime posts a 5.19% mean word error rate, first of the 22 systems on our benchmarks page, against 7.78% for ElevenLabs Scribe v2 Realtime, 8.64% for Deepgram Nova-3, and 13.50% for Deepgram Flux EN.

Entity error rate matters even more for agents. It measures how often the things that carry meaning come back wrong: names, account numbers, confirmation codes. Universal-3.6 Pro Realtime lands at 14.4%, versus 18.5% for Scribe v2, 26.1% for Nova-3, and 30.1% for Flux EN. When a caller reads back a card number, that's the metric deciding whether the agent charges the right account.

For an independent check, on Pipecat's open STT benchmark of real agent turns, Universal-3.6 Pro Realtime posts a 0.96% pooled semantic word error rate, and the final transcript lands a median 307 ms after the speaker stops. Passing the agent's spoken reply as agent_context gives the model the question the caller is answering, and the benchmarks page charts entity error with agent context on and off.

"We were searching for the best realtime ASR model for our voice agent pipeline in Fireflies. The new Universal 3.5 Pro speech model from Assembly is best so far in terms of accuracy, latency and language switching."

— Foysal Osmany, Software Engineer at Fireflies

What you'll pay

With AssemblyAI as the speech layer, the speech-to-text line is $0.45/hr for Universal-3.6 Pro Realtime, pay-as-you-go with no minimums. Optional add-ons stack on that line: streaming diarization at $0.12/hr and Voice Focus at $0.10/hr. The Text-to-Speech API is $15 per 1M characters, billed on the input characters you send (rates are on the pricing page), and your LLM, any orchestration platform fee, and your carrier minutes come on top.

That's more than one line item, but each one moves predictably. Speech-to-text scales with call duration, not with how chatty the context gets. Platforms and token-metered APIs bill on different meters, so the same call can cost more or less depending on how long and how talkative it runs. We priced the same call under each model in AI voice agent pricing, compared.

Before you commit: test it on your own calls

Run your 50 worst calls through every finalist: the accents, the noisy lines, the callers who spell their names. Measure entity error and P95 latency under realistic load, not on a single quiet session. Then price cost per completed task, not cost per minute, and load test at your expected peak.

Getting started

The speech-to-text side is one WebSocket with standard JSON messages. Open a connection with your API key in the Authorization header, stream audio in, and act on each Turn event where end_of_turn is true. The voice side is a second session on the same API key: the Python SDK (1.8.2 or later) opens it, and the TTS docs cover the raw WebSocket and HTTP options. The LiveKit and Pipecat integrations handle both for you if you'd rather not touch the sockets.

Here's the shape of a minimal loop against the streaming WebSocket API. The LLM call and audio output are stubs for whatever you plug in, and the reply is spoken by the Text-to-Speech API:

import asyncio
import json
import os

import assemblyai as aai  # pip install -U assemblyai (1.8.2 or later)
import websockets  # pip install websockets

API_KEY = os.environ["ASSEMBLYAI_API_KEY"]

URL = (
    "wss://streaming.assemblyai.com/v3/ws"
    "?speech_model=universal-3-6-pro&sample_rate=16000&encoding=pcm_s16le"
)

TTS_CONFIG = aai.TTSConfig(
    voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3",  # Audrey
    language="english",
)


async def llm(text):
    ...  # your LLM call (or LLM Gateway)


async def play(chunk):
    ...  # your audio output: raw 24 kHz 16-bit mono PCM


async def run(audio_chunks):
    headers = {"Authorization": API_KEY}
    async with websockets.connect(URL, additional_headers=headers) as ws:
        # One TTS session for the whole call, one request per agent turn
        async with aai.AsyncTTSSession(TTS_CONFIG, api_key=API_KEY) as tts:

            async def send_audio():
                async for chunk in audio_chunks:  # 50-1000 ms of 16 kHz PCM per message
                    await ws.send(chunk)
                await ws.send(json.dumps({"type": "Terminate"}))

            sender = asyncio.create_task(send_audio())

            async for message in ws:
                event = json.loads(message)
                if event["type"] == "Turn" and event["end_of_turn"]:
                    reply = await llm(event["transcript"])
                    # Give the speech model the agent's reply as context for the next turn
                    await ws.send(json.dumps({"type": "UpdateConfiguration", "agent_context": reply}))
                    async for chunk in tts.speak(reply):
                        await play(chunk)
                elif event["type"] == "Termination":
                    break

            await sender

Three details carry most of the quality. Send audio in 50 to 1,000 ms chunks at real-time pace. Update agent_context after every agent reply, because each update replaces the last one. And send Terminate when the call ends so the session closes cleanly. When the caller cuts in, call cancel() on the TTS session to stop the reply in flight. From there, move the loop into LiveKit or Pipecat when you need rooms, telephony, or interruption handling around the TTS output.

The bottom line

Every option on this list will get you a demo. The differences show up on your worst calls, at your real volume, on an invoice someone has to forecast. Platforms hand you a hosted setup and take control of the layers in exchange. Frameworks leave the layers to you. Token-metered models bill on meters that move with how talkative the call gets.

But the agent can only act on what it heard. If accuracy is what your product lives or dies on, choose the speech layer first and build the rest on top of it.

Frequently Asked Questions (FAQs)

What is the best API for building voice agents?

For accuracy-critical production agents, the strongest setup is AssemblyAI's voice stack, Universal-3.6 Pro Realtime for streaming speech-to-text and turn detection and the Text-to-Speech API for voice output, orchestrated with LiveKit, Pipecat, Vapi, or your own loop and paired with the LLM you choose. OpenAI Realtime suits multimodal prototyping, and Vapi and Retell suit no-code phone agents. The right pick depends on how much control you need and how much your calls depend on accuracy.

What is the best one-API solution for voice agents?

The closest thing to a one-API voice agent is one vendor for both speech ends on one key: AssemblyAI's Universal-3.6 Pro Realtime for streaming speech-to-text and turn detection, AssemblyAI Text-to-Speech for the reply, and LLM Gateway for the model in between. All three run on the same API key and account, while LiveKit, Pipecat or your own loop runs the conversation. You can swap the LLM without touching either speech layer, which all-in-one agent platforms usually don't allow.

Which voice agent API is best for multilingual customer support?

For multilingual support, pick a stack that follows a caller who switches languages mid-call instead of locking one language per session. Universal-3.6 Pro Realtime transcribes 32 languages in streaming, all with native code-switching, so a caller who moves between English and Spanish stays in one session. AssemblyAI Text-to-Speech speaks six languages (English, Spanish, French, German, Italian and Portuguese), and every voice can speak all six. The TTS API doesn't detect language, so set language for each session to match the text your agent sends.

Is a managed all-in-one voice agent API better than building your own STT-LLM-TTS stack?

A managed all-in-one API gets you to a demo with fewer decisions, but it fixes the speech model, the LLM, and the voice for you. Building on a framework like LiveKit or Pipecat lets you pick each layer, swap the LLM without touching speech, and tune turn detection for your callers. In either case, the speech-to-text layer sets the accuracy ceiling for the whole agent.

Which voice agent API has the best speech accuracy?

Universal-3.6 Pro Realtime posts a 5.19% word error rate and a 14.4% entity error rate on AssemblyAI's 12,460-scenario English voice-agent benchmark. On the same benchmark, ElevenLabs Scribe v2 is at 7.78% WER and 18.5% entity error, Deepgram Nova-3 at 8.64% and 26.1%, and Deepgram Flux EN at 13.50% and 30.1%. Entity error is the number to watch, because names and account numbers are what agents act on.

How much does a voice agent cost to run with AssemblyAI?

Universal-3.6 Pro Realtime streaming speech-to-text is $0.45/hr, pay-as-you-go with no minimums. The Text-to-Speech API is $15 per 1M characters, billed on input characters, with rates on the pricing page, and your LLM, any orchestration platform fee, and carrier minutes come on top. Compare cost per completed call on your own traffic, because call length, talk ratio, and LLM choice move the total more than any single list rate.

Does AssemblyAI work with LiveKit, Pipecat, and Vapi?

Yes. Universal-3.6 Pro Realtime drops into LiveKit and Pipecat as the speech-to-text layer, and it's available as a transcriber in Vapi. The Text-to-Speech API also has LiveKit and Pipecat integrations, so both ends of the agent can run on AssemblyAI inside those frameworks. They run the conversation, and AssemblyAI supplies the speech underneath.

Which voice agent API handles enterprise-scale, high-call-volume production?

Choose an API that puts no cap on concurrent sessions and lets capacity grow with traffic. Universal-3.6 Pro Realtime and AssemblyAI Text-to-Speech count only new sessions per minute (200 each to start on a paid plan), raise that allowance by 10% every minute you use 80% or more of it, and never count calls already in progress. Both run in US and EU data zones, with no concurrency fees or tier commitments. Load test at your expected peak before launch so you see P95 latency and error behavior under real traffic.