Using the Voice Agent API alongside an existing voice stack

The standard sales playbook for a new voice agent platform tells you your existing architecture is wrong and you should rebuild from scratch. This post doesn't do that.

If you've already shipped a voice agent on LiveKit, Pipecat, or Vapi, you've done real work. You tuned turn detection thresholds. You wrote glue code that handles the ugly cases. You debugged interruption handling at 11pm before a customer demo. None of that is wasted, and nobody should tell you it is.

But there is one change worth making, and it takes about ten minutes.

The two lanes, named honestly

AssemblyAI gives you two ways in, and they're built for different situations. Being straight about which one you're in saves everyone a discovery call.

Lane 1: Voice Agent API ($4.50/hr). One WebSocket. Everything managed. AssemblyAI handles speech-to-text, turn detection, LLM routing, and text-to-speech behind a single connection. You send audio, you get conversation.

Lane 2: Universal-3.5 Pro Realtime ($0.45/hr). Plug the streaming model into your pipeline as the STT layer. Keep everything else — your LLM, your TTS, your turn detection logic, your existing integrations, your glue code.

That last row is the one people miss. Both lanes listen with the same model. The difference isn't accuracy — it's how much of the stack we run for you.

If you're on an orchestration framework today, you're in Lane 2. Read on.

Why STT is still the lever worth pulling

You control your LLM. You control your TTS. You can swap either one this afternoon and the agent still works, roughly.

Your STT model is different. It determines what your LLM gets to reason about. If the transcription is wrong, everything downstream inherits the error — and inherits it silently, because the LLM never sees the audio. It only sees your transcript, and it has no way to know the transcript is wrong.

So the question isn't "which model has the best word error rate." It's which model captures the words your users actually say that matter: the account ID, the surname, the medication, the callback number. Those are a small fraction of the tokens in a conversation and close to all of the business value. A model can post an excellent WER by getting the and and and um right while dropping one digit in six of a policy number.

Here's what that looks like on Pipecat's open STT benchmark, which runs on real agent conversations rather than read-aloud audiobook corpora:

Broken out by entity type, Universal-3.5 Pro Realtime misses 16.92% of names, 6.28% of places, and 3.55% of phone numbers. Phone numbers and account IDs are where a voice agent either books the appointment or doesn't, so that 3.55% is the number worth arguing about.

Beyond raw accuracy, three things come with the swap that you can't get by tuning your existing pipeline harder:

• agent_context — you pass your agent's own question into the stream, and the model hears the caller's reply through the lens of what was just asked. Ask "what's your date of birth" and the model biases toward dates. Across a 20,000-file voice-agent benchmark, this cut WER 10.2%.
• Context Carryover — rolling conversation memory that applies earlier turns to later transcription. It's on by default. There's nothing to configure and nothing to predefine.
• 18 languages with mid-sentence code-switching, including Hinglish. Not per-turn language detection — a caller can switch languages inside a single sentence and the transcript follows. If your users do this, you already know how badly most models handle it.

Keyterms are included at no extra cost on Universal-3.5 Pro Realtime — up to 100 terms, each 50 characters or fewer, and updatable mid-stream. If your agent knows the caller's name before the call connects, you can push it into the recognizer before they say a word.

How to evaluate it without disrupting your stack

Nothing here touches production traffic. You run the new model in parallel, compare, and decide.

Step 1: Grab an API key

Free tier, no credit card. It takes two minutes. The free tier covers 333 hours of streaming and 185 hours of pre-recorded audio, which is far more than a shadow test consumes.

Step 2: Update your STT configuration

This is the whole change. Point your streaming connection at AssemblyAI's v3 endpoint and set one parameter.

The endpoint and the parameter:

wss://streaming.assemblyai.com/v3/ws?speech_model=universal-3-5-pro&sample_rate=16000

Two things to note. The parameter on streaming is speech_model, singular — the plural speech_models array belongs to the pre-recorded API and will be rejected here. And universal-3-5-pro is already the streaming default, but set it explicitly anyway; never let a production pipeline ride on an account default that could change under you. If you need EU data residency, the host is streaming.eu.assemblyai.com at the same price.

For language biasing, streaming uses language_codes — a plural list, unlike the singular language_code on pre-recorded. The older language parameter is deprecated.

LiveKit Agents:

from livekit.plugins import assemblyai

stt = assemblyai.STT(
    model="universal-3-5-pro",
    language_codes=["en", "es"],
)

Pipecat:

import os
from pipecat.services.assemblyai.stt import AssemblyAISTTService

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
    ), vad_force_turn_endpoint=True)

In both cases the rest of your agent definition is untouched. Your LLM service, your TTS service, your VAD, your interruption handling, your function calling — none of it changes. You swapped one service object.

If you want the accuracy lift from agent_context, pass your agent's last utterance alongside the audio on each turn. Full parameter reference is in the streaming model selection docs.

Formatting is always on. Turn detection defaults are min_turn_silence and max_turn_silence set by the mode preset, 128 ms and 1280 ms on balanced, with a vad_threshold of 0.2 — leave them alone for the first run so you're comparing models, not tuning. If your existing pipeline owns turn detection, keep it — you can drive endpointing yourself with ForceEndpoint.

Step 3: Run both in parallel for a week

Fork your audio stream. Shadow the Universal-3.5 Pro Realtime output against your current STT output on the same audio, in production, with no user-facing change. Log both transcripts side by side.

A week is usually enough to hit the cases that matter — the caller with the accent, the noisy warehouse, the person who spells their surname. Those are the calls you're actually buying a new model for.

Step 4: Measure what matters for your use case

Not aggregate WER. Three things:

• Entity capture on your specific entity types. Pull the last 200 calls where an entity mattered and count the misses on each transcript. This is the number that maps to revenue.
• Latency, end to end. Time-to-final on Universal-3.5 Pro Realtime is a median of 285 ms, p95 374 ms, p99 443 ms. Measure it inside your own pipeline, because your VAD and network add their own budget.
• Turn detection behavior. Count barge-ins the agent handled wrong and pauses it mistook for end-of-turn. This is where a model that looks worse on a spreadsheet can feel dramatically better on a call.

There's a fuller version of this evaluation method in our guide to choosing an STT API for voice agents.

If you're coming from OpenAI's Realtime API

This one's different, and it's worth calling out because the migration path isn't the same shape.

OpenAI's Realtime API is a speech-to-speech model: audio in, audio out, with the reasoning fused into the model. That's elegant, and it's also why teams leave it. You can't inspect the transcript the model reasoned over, you can't swap the LLM, and you can't put your own logic between hearing and speaking.

Two migration targets, depending on what you're optimizing for.

If you want to keep a managed single-socket experience, the Voice Agent API is the closest analogue — one WebSocket, roughly one second end to end, PCM16 at 24 kHz. The difference is that it's a cascaded STT → LLM → TTS pipeline inside one connection rather than a fused model, so you get the transcript, you get recordings, and you can prompt each stage. Sessions survive 30 seconds after a disconnect, so a dropped mobile connection can resume with session.resume and the prior session_id. That window is billable, so send session.end when the call is genuinely finished.

If what you actually wanted was your own agent logic all along, go to Lane 2 and put Universal-3.5 Pro Realtime under LiveKit or Pipecat. Your prompts and tool definitions move over as-is — they were never coupled to the audio layer. We wrote up the tradeoffs across the category in OpenAI Realtime API alternatives.

Either way, your agent logic doesn't get rewritten. That's the part teams brace for and it isn't where the work is.

When the Voice Agent API becomes interesting to you

File this away for later.

If you ever start a new project from zero — a new product line, a proof of concept for a different use case, a V2 where you're rethinking the infrastructure — the Voice Agent API is the other lane. Flat $4.50/hr, billed per second on connected conversation time, with prompting, Voice Focus, advanced turn detection, interruption detection, recordings and transcripts, hosting, and BYO-Twilio SIP trunking all included. Every feature is included in the $4.50/hr rate. There are no per-layer add-ons, concurrency fees, or per-agent subscriptions.

That pricing model matters more than it sounds. Most of the cost surprises in a voice agent stack come from the seams between vendors, not from any one line item — we broke that down in the hidden cost of a voice agent stack.

But you don't have a greenfield project. You have a running one. Which is the point of this post.

The work you did doesn't need to be undone

You built something real. The glue code, the threshold tuning, the 2am debugging — that's infrastructure with value, and it encodes things about your users that no vendor knows. Nobody should tell you to throw it away.

Lane 2 exists specifically for you. Same model as Lane 1. Same accuracy. Plugs into what you already have, in one parameter.

This is also how our partners see it. LiveKit made Universal-3.5 Pro available on LiveKit Inference for exactly this reason:

"We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms."

— David Zhao, Co-founder at LiveKit

Retell took the same path from the other direction. They evaluated on their own production traffic and made AssemblyAI their default high-accuracy option, replacing Deepgram — without rebuilding their agent platform. The orchestration stayed. The listening layer changed.

Here's the thing worth sitting with: the reason a one-parameter migration is even possible is that the industry finally agreed on the boundary between hearing and thinking. Five years ago, swapping a speech model meant rewriting your application, because the model and the app were the same artifact. That's over. The layers are separable now, which means the cost of being wrong about your STT vendor has collapsed to about ten minutes and a week of shadow traffic.

Which means there's no longer a good reason to stay on a model you haven't tested lately.

Frequently asked questions

Can I run AssemblyAI's Voice Agent API side-by-side with my current provider during migration?

Yes — and running both in parallel is the recommended way to migrate. Fork your audio stream, send the same audio to both providers, and log the transcripts side by side with no user-facing change. There are no minimums, no upfront commits, and unlimited concurrency, so a shadow test costs only the streaming audio it consumes. Most teams get a decision out of about a week of production traffic.

How do I migrate my voice agent from Deepgram to AssemblyAI's Voice Agent API?

If you're on an orchestration framework, it's one parameter: point at wss://streaming.assemblyai.com/v3/ws and set speech_model: "universal-3-5-pro". Your LLM, TTS, turn detection, and function calling stay exactly as they are. If you'd rather hand off the whole pipeline, the fully managed Voice Agent API replaces your orchestration with a single WebSocket at a flat $4.50/hr — but that's a bigger change and it's optional, not a prerequisite for getting the accuracy.

How do I migrate from OpenAI's Realtime API to AssemblyAI without rewriting my agent logic?

Your prompts and tool definitions carry over unchanged, because they were never coupled to the audio layer. Choose the Voice Agent API if you want to keep a single managed socket with roughly one second end-to-end latency, or put Universal-3.5 Pro Realtime under LiveKit or Pipecat if you want your own LLM and your own logic between hearing and speaking. The gain over a fused speech-to-speech model is that you get the transcript back, so you can log it, audit it, and reason over it.

AssemblyAI vs Deepgram for voice agents: Voice Agent API vs Nova-3

The difference shows up on entities, not on aggregate word error rate. On Pipecat's open STT benchmark of real agent conversations, Universal-3.5 Pro Realtime posts a 15.31% entity error rate and 6.99% pooled WER, against 50.50% and 15.58% for Deepgram Flux. On pricing, AssemblyAI streaming is $0.45/hr and the fully managed Voice Agent API is a flat $4.50/hr with every feature included — no per-layer add-ons or concurrency fees. Full methodology is on our benchmarks page.

Is there a free tier for AssemblyAI's Voice Agent API I can use before committing to production volume?

Yes. The free tier covers 333 hours of streaming and 185 hours of pre-recorded audio, with no credit card required. That's enough to run a full shadow evaluation against your current provider and still have room for a pilot. After that it's pay-as-you-go, billed per second, with no minimums and no upfront commitment — see pricing for the current rates.

Is AssemblyAI suitable for healthcare voice agents that handle PHI?

Yes. AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI. The BAA can be signed in minutes without a sales call. For clinical vocabulary, Medical Mode activates with domain: "medical-v1" as a $0.15/hr add-on — details on the medical transcription page.

‍