If you've already shipped a voice agent on Vapi, Retell, LiveKit or Pipecat, you don't need to rebuild anything to add AssemblyAI. You swap Universal-3.6 Pro Realtime into the speech-to-text slot, run it in shadow next to your current provider for a week, measure entity capture, latency and turn behavior on your own calls, and cut over with a one-object change you can roll back. The Text-to-Speech API can be evaluated the same way for the voice slot, on its own schedule.
This post doesn't tell you your architecture is wrong. You tuned turn detection thresholds. You wrote glue code that handles the ugly cases. You debugged interruption handling at 11pm before a customer demo. None of that is wasted. For a tour of the orchestrators themselves, see our roundup of voice agent orchestration tools. This one is about changing a single layer underneath them, safely.
What changes and what stays
AssemblyAI is the speech layer, not the agent. Your orchestrator keeps the loop, your LLM keeps the reasoning, and your tools, prompts and state don't move. Here's the scope of the change on each stack:
| Your stack | Where the STT slot lives | What you change | What stays |
|---|---|---|---|
| LiveKit Agents | The stt= argument of your agent session |
One plugin object (livekit-agents 1.8.0+) | LLM, TTS, VAD, tools, room logic |
| Pipecat | The STT service in your pipeline | One service object (pipecat-ai 1.9.0+) | LLM, TTS, processors, transport |
| Vapi | The assistant’s transcriber object |
Provider and speech model | Model, voice, tools, phone numbers |
| Retell | The agent’s Transcription Mode setting | Custom Settings, with AssemblyAI as the provider | Agent logic, voice, telephony |
| Your own loop | Your streaming WebSocket client | Endpoint, auth and message handling | Everything else |
The reason a one-object swap is possible is that the industry finally agreed on the boundary between hearing and thinking. Five years ago, swapping a speech model meant rewriting your application. Now the layers are separable, so the cost of testing a new STT vendor has collapsed to an afternoon of setup and a week of shadow traffic.
Why the STT slot is the lever worth pulling
You control your LLM and your TTS. Swap either one and the agent still works, roughly. Your STT model is different: it determines what your LLM gets to reason about. If the transcription is wrong, everything downstream inherits the error silently, because the LLM never sees the audio.
So the question isn't which model has the best word error rate. It's which model captures the words that matter: the account ID, the surname, the medication, the callback number. Those are a small fraction of the tokens in a conversation and close to all of the business value. Here's what that looks like on AssemblyAI's English voice-agent benchmark, which runs 12,460 scripted voice-agent scenarios rather than read-aloud audiobook corpora:
| Model | Entity error rate | WER |
|---|---|---|
| Universal-3.6 Pro Realtime | 14.4% | 5.19% |
| Deepgram Flux EN | 30.1% | 13.50% |
| ElevenLabs Scribe v2 | 18.5% | 7.78% |
| Deepgram Nova-3 | 26.1% | 8.64% |
Broken out by entity type, Universal-3.6 Pro Realtime misses 10.9% of names, 10.0% of codes and IDs, and 2.4% of phone numbers. On Pipecat's open STT benchmark it posts a 0.96% pooled semantic WER, and the final transcript lands a median 307 ms after the speaker stops.
Three things come with the swap that you can't get by tuning your existing pipeline harder:
- agent_context: pass your agent's last question into the stream, and the model hears the reply through that lens. Ask “what's your date of birth?” and it leans toward dates. Across 20,000 voice-agent files, this cut WER 10.2%. LiveKit and Pipecat both support Context Carryover, so you get it without extra plumbing.
- 32 languages with mid-sentence code-switching, with automatic language detection. A caller can switch languages inside a sentence and the transcript follows.
- Keyterms prompting, included on streaming: up to 100 terms, updatable mid-stream. If your agent knows the caller's name before they speak, push it into the recognizer first.
The free tier covers up to 333 hours of streaming with no credit card, more than a week of shadow traffic on most agents.
Step 1: Shadow-test without touching production
Nothing in this step changes what a caller hears. You fork the audio your current STT already receives, send the copy to Universal-3.6 Pro Realtime, and log both transcripts side by side.
Where to fork depends on your stack. On LiveKit or Pipecat, add a second consumer of the inbound audio track or frames. On Vapi or Retell, the cleanest fork is usually from call recordings or a media stream you already capture, replayed through the same streaming endpoint so you test real-time behavior rather than batch accuracy. If you run your own loop, tee the frames before they reach your current client.
The streaming API is a single WebSocket at wss://streaming.assemblyai.com/v3/ws. Authenticate with your API key in the Authorization header (no Bearer prefix), send binary PCM16 audio in 50–1000 ms chunks, and read Turn messages back. A minimal shadow tap in Python:
# pip install "websockets>=14"
import asyncio, json, os
import websockets
URL = ("wss://streaming.assemblyai.com/v3/ws"
"?speech_model=universal-3-6-pro&sample_rate=16000&encoding=pcm_s16le")
async def shadow(frames, log):
"""frames: async iterator of 16 kHz mono PCM16 chunks (50-1000 ms each),
forked from the audio your current STT already receives."""
async with websockets.connect(
URL, additional_headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}
) as ws:
async def send():
async for chunk in frames:
await ws.send(chunk)
await ws.send(json.dumps({"type": "Terminate"}))
async def receive():
async for raw in ws:
msg = json.loads(raw)
if msg["type"] == "Begin":
log("session", msg["id"])
elif msg["type"] == "Turn" and msg["end_of_turn"]:
log("assemblyai", msg["turn_order"], msg["transcript"])
elif msg["type"] == "Termination":
break
await asyncio.gather(send(), receive())
# Optional: after each agent reply, tell the model what was just asked.
async def set_agent_context(ws, last_agent_line):
await ws.send(json.dumps({"type": "UpdateConfiguration",
"agent_context": last_agent_line}))Two notes. The streaming parameter is speech_model, singular; the plural speech_models array belongs to the pre-recorded API. And universal-3-6-pro is already the streaming default, but set it explicitly so a production pipeline never rides on a default. If you need EU data residency, use streaming.eu.assemblyai.com.
Leave turn-detection settings at their defaults for the first run, so you're comparing models, not tuning. Run the shadow for about a week. That's usually enough to hit the cases that matter: the caller with the accent, the noisy warehouse, the person who spells their surname. Those are the calls you're testing a new model for.
Step 2: Measure what matters
Not aggregate WER. Four things:
- Entity capture on your entity types. Pull the calls where an entity mattered (account numbers, dates of birth, emails, order IDs) and count the misses in each transcript. This is the number that maps to revenue.
- Turns your LLM would have handled differently. Feed both transcripts of the same turn to your LLM offline and diff the responses. Where they diverge, listen to the audio and decide which transcript was right.
- Latency inside your pipeline. Measure time from end of speech to final transcript in your own environment, because your VAD and network add their own budget. Watch p95, not just the median: the tail is what callers hear as awkward silence.
- Turn behavior. If you plan to use AssemblyAI's turn detection, count barge-ins handled wrong and pauses mistaken for end of turn. Universal-3.6 Pro Realtime combines semantic context with voice activity, so a caller pausing mid-way through a card number shouldn't get cut off. This is where a model that looks similar on a spreadsheet can feel very different on a call.
There's a fuller evaluation method in our guide to choosing an STT API for voice agents.
Step 3: Cut over
Once the shadow numbers hold up, the production change is small. Make it behind a flag so you can route a slice of traffic first and roll back in one config change.
LiveKit Agents (livekit-agents 1.8.0+):
from livekit.plugins import assemblyai
stt = assemblyai.STT(
model="universal-3-6-pro",
language_codes=["en", "es"],
)Pipecat (pipecat-ai 1.9.0+):
import os
from pipecat.services.assemblyai.stt import AssemblyAISTTService
stt = AssemblyAISTTService(
api_key=os.getenv("ASSEMBLYAI_API_KEY"),
settings=AssemblyAISTTService.Settings(
model="universal-3-6-pro",
),
vad_force_turn_endpoint=True, # keep Pipecat's turn detection; False uses AssemblyAI's
)On Pipecat, vad_force_turn_endpoint=True keeps your existing Pipecat turn handling, which is the lowest-risk first cut. Set it to False later to use AssemblyAI's built-in turn detection and compare.
Vapi: set the transcriber in the dashboard's Transcriber tab or in the assistant JSON:
{
"transcriber": {
"provider": "assembly-ai",
"speechModel": "universal-3-6-pro"
}
}Set speechModel explicitly, because Vapi's default is a different model. If you update with PATCH, send the whole transcriber object, since PATCH replaces it. Leaving Vapi's smart endpointing plan unset lets AssemblyAI's end-of-turn detection drive the turn. Details are in the Vapi guide.
Retell: set the agent's Transcription Mode to Custom Settings and choose AssemblyAI as the provider (in the API, stt_mode "custom" with custom_stt_config.provider "assemblyai"). Prompts, functions, voice and telephony stay as they are. The orchestration stays. The listening layer changes.
In every case the rest of your agent definition is untouched: LLM, TTS, VAD, interruption handling and function calling. Roll out to 5–10% of calls, compare task completion and re-prompt rates against the control group for a few days, then widen. If anything regresses, the rollback is reverting one object.
Run a recording with names, phone numbers and confirmation codes through Universal-3.6 Pro Realtime before you change anything in production.
Evaluating the voice slot
Once the input side is settled, the same method works for output. AssemblyAI's Text-to-Speech API is built for conversation, not narration. It handles hesitations, restarts and interruptions, and reads digits and names correctly, which is what an agent needs when it reads back an order number or confirms an appointment. Using it alongside Universal-3.6 Pro Realtime gives you speech in and speech out from one vendor, on one account.
Treat it as a separate experiment, after the STT cutover, so you can attribute any change to one layer. Route a slice of calls to the new voice and measure what callers do, not what reviewers think: how often they ask the agent to repeat itself, how often they correct a read-back number or name, and whether calls complete. Keep your current TTS on the rest of the traffic as the control. Check the orchestrator docs for how to wire a TTS service in your stack.
If you're coming from OpenAI's Realtime API
This migration has a different shape. OpenAI's Realtime API is a speech-to-speech model: audio in, audio out, with the reasoning fused into the model. That's why teams leave it. You can't inspect the transcript the model reasoned over, you can't swap the LLM, and you can't put your own logic between hearing and speaking.
Moving to a cascaded pipeline means putting Universal-3.6 Pro Realtime under LiveKit or Pipecat, with your LLM and a text-to-speech service. Your prompts and tool definitions move over as-is, because they were never coupled to the audio layer, and you get a transcript you can log, audit and evaluate. We covered the tradeoffs in OpenAI Realtime API alternatives.
The work you did doesn't need to be undone
The glue code, the threshold tuning, the 2am debugging encode things about your users that no vendor knows. Adding AssemblyAI keeps all of it and changes the one layer every other layer depends on.
Our partners see it the same way. LiveKit made Universal-3.5 Pro available on LiveKit Inference for exactly this reason:
“We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms.”
— David Zhao, Co-founder at LiveKit
Pricing stays simple during the test and after: Universal-3.6 Pro Realtime is $0.45/hr base, billed per second, with no minimums and no cap on open sessions. A shadow test costs only the audio it consumes. See pricing for add-ons like streaming diarization and Voice Focus.
Which means there's no longer a good reason to stay on a speech model you haven't tested lately.
Running high call volume, a regulated workload or a multi-region deployment? Talk to our team about shadow testing and rollout on your stack.
Frequently asked questions
Can I run AssemblyAI side by side with my current STT provider?
Yes, and it's the recommended way to evaluate. Fork your audio stream, send the same audio to both providers, and log the transcripts side by side with no user-facing change. There are no minimums and no cap on open sessions, so a shadow test costs only the streaming audio it consumes. Most teams have enough data after about a week of production traffic.
How do I switch my LiveKit or Pipecat agent to Universal-3.6 Pro Realtime?
Replace one object. On LiveKit Agents 1.8.0+, use assemblyai.STT(model="universal-3-6-pro"). On Pipecat 1.9.0+, use AssemblyAISTTService with model="universal-3-6-pro" in its Settings. Your LLM, TTS, VAD, interruption handling and tools stay exactly as they are.
How do I use AssemblyAI with Vapi or Retell?
On Vapi, set the assistant's transcriber provider to assembly-ai and speechModel to universal-3-6-pro, in the dashboard or the assistant JSON. On Retell, set Transcription Mode to Custom Settings and pick AssemblyAI as the provider. In both cases the rest of the agent configuration is unchanged.
What should I measure during a shadow test?
Entity capture on the entity types your agent depends on, turns where the two transcripts would lead your LLM to different responses, end-of-speech-to-final-transcript latency inside your own pipeline (including p95), and turn-detection behavior such as premature cut-offs. Aggregate WER alone hides the errors that cost you calls.
Can I evaluate AssemblyAI's Text-to-Speech API at the same time?
You can, but run it as a separate experiment after the speech-to-text cutover so you can attribute changes to one layer. Route a slice of calls to the new voice, keep your current TTS as the control, and measure repeat requests, corrected read-backs and call completion.
Is AssemblyAI suitable for healthcare voice agents that handle PHI?
Yes. AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI. For clinical vocabulary, Medical Mode is a $0.15/hr add-on on streaming.
