What's inside a voice agent pipeline: every stage from audio in to audio out

A voice agent pipeline has seven stages: noise handling, speech-to-text, turn detection, the LLM, tool calls, text-to-speech, and transport or telephony. Audio comes in at one end and audio goes out at the other, and the latency a caller feels is the sum of every handoff in between. AssemblyAI covers the speech layer at both ends (Universal-3.6 Pro Realtime for what the agent hears, the Text-to-Speech API for what it says), and your orchestrator and LLM own the loop in the middle.

"Managed" is one of the most abused words in voice AI tooling. It usually means: someone made decisions about your pipeline, they’re not going to tell you what they are, and when it breaks you get to guess. If you’ve been burned by a black box before, that instinct is correct and you should keep it.

So this post does the opposite. Every stage is named, every knob is listed, every number is sourced, and for each stage we say plainly which part AssemblyAI handles and which part belongs to your orchestrator, whether that’s LiveKit, Pipecat, Vapi, or a loop you wrote yourself.

The pipeline, named

A voice agent turn is a relay race. Audio comes in, seven stages hand off to each other, audio goes out. Latency is the sum of the handoffs, which is why streaming every stage into the next matters more than any single stage being fastest.

Here’s the short version of who does what before we go stage by stage:

1. Noise handling and voice focus

The first stage suppresses background noise and isolates the primary speaker before any transcription happens. This ordering is deliberate: speech models are trained on speech, and every unfiltered dog bark, keyboard clatter, or second conversation in the room is an opportunity for the model to transcribe something that was never said.

What AssemblyAI covers. Voice Focus runs inside the streaming session and isolates the primary voice before transcription. It has two profiles. near-field is tuned for headsets and phone handsets, where the speaker is close to the mic and the noise is mostly ambient. far-field is tuned for rooms, kiosks, and drive-thrus, where the speaker is at a distance and competing voices are the dominant problem. It’s off by default and costs +$0.10/hr when you turn it on. Pick the profile that matches your deployment, not the one that sounds more capable, and leave it off if nobody else is talking near the caller.

Even without Voice Focus, the model is built to keep background speech out of the transcript. The short version: it hears the caller, not the room.

What your stack owns. Echo cancellation and audio capture. If the agent’s own voice leaks back into the microphone, that’s a client or media-server problem, and WebRTC stacks like LiveKit handle it on their side.

2. Speech-to-text: Universal-3.6 Pro Realtime

Universal-3.6 Pro Realtime is our flagship streaming speech-to-text model. Transcripts stream out as the user talks rather than arriving after they stop, which is the only way the downstream LLM can start reasoning before the turn ends.

Entity accuracy is the number that determines whether a voice agent works. A word error rate of 5% sounds fine until the 5% is the caller’s last name, their street, or the last four digits of their account. Here is Universal-3.6 Pro Realtime on AssemblyAI’s English voice-agent benchmark, which is scored on 12,460 scripted voice-agent scenarios rather than read audiobooks. Lower is better.

Independent benchmarks point the same way: on Pipecat’s open STT benchmark, it posts a 0.96% pooled semantic word error rate, and the final transcript lands a median 307 ms after the speaker stops. On AssemblyAI’s benchmarks page, it ranks 1st of 22 systems on the voice-agent suite, with a 5.19% WER averaged over no, low, and heavy noise and the lowest entity error rate of the 22 at 14.41%.

Keyterms prompting is included on streaming at no extra charge. Pass your SKU list, drug names, or account prefixes and the model biases toward them. You can set them at connection time or swap them mid-session with an UpdateConfiguration message, which is useful when the conversation moves from “who are you” to “which medication”:

{
  "type": "UpdateConfiguration",
  "keyterms_prompt": ["Okonkwo", "periodontal", "Dr. Reyes"]
}

agent_context lets you pass the agent’s own last reply into the speech model, so short or mumbled answers resolve against what was asked. Ask “what’s your date of birth?” and a mumbled “oh three fourteen eighty-seven” is a date, not noise. Across a benchmark of 20,000 voice agent audio files, agent_context cut WER by 10.2%, with fabrications down 18.3%, hallucinations down 17.2%, place-name entities down 15.5%, short-utterance errors down 13.7%, and name and medical entities each down 9.4%.

The model also keeps a short rolling conversation memory and uses it as context for each new turn, so recognition leans toward what has already been said in the session.

“We’re excited to make AssemblyAI’s Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don’t, removing the need for users to predefine key terms.”

— David Zhao, Co-founder at LiveKit

Languages: 32. Universal-3.6 Pro Realtime runs in 32 languages with automatic language detection and mid-sentence code-switching. A caller can start a sentence in Spanish and finish it in English and the transcript follows them. On code-switched audio it posts 7.20% WER against 17.13% for Deepgram Nova-3 Multi. If you know the caller’s language, the language_codes parameter steers the model toward it, which prevents wrong-language slips on short audio. Details are in the multilingual transcription docs.

What your stack owns. Which audio you send, in what encoding, and when sessions open and close. Streaming bills per second of session time at $0.45/hr base, so send a Terminate message when a call ends instead of letting the socket idle.

3. Turn detection and interruption handling

End-of-turn detection decides when the user has finished speaking. Silence-threshold VAD gets this wrong constantly, because humans pause mid-thought and then keep going. Universal-3.6 Pro Realtime combines semantic context with voice activity: when a pause occurs, the model looks at what has been said so far and judges whether it sounds complete.

It’s also entity-aware. If the transcript ends partway through something that looks like a phone number or an account code, the model holds the turn open rather than cutting the caller off after the third digit. Median endpoint latency is 537 ms. Each final Turn message carries an end_of_turn_confidence score your orchestrator can read.

What AssemblyAI covers. The end-of-turn decision itself, and the knobs to tune it. The turn detection docs list min_turn_silence (silence before an end-of-turn check runs), max_turn_silence (the most silence allowed before a turn is forced to end), and vad_threshold, with defaults that vary by preset. You can change any of them mid-stream. The docs’ own example raises the silence window before the agent collects an entity, then re-applies the preset afterwards:

# Before capturing the entity
websocket.send('{"type": "UpdateConfiguration", "min_turn_silence": 1000}')

# After the response, re-apply your mode to restore its defaults
websocket.send('{"type": "UpdateConfiguration", "mode": "balanced"}')

If your application already knows the turn is over (the caller pressed a key, or your UI has a “done” button), you can end it immediately:

websocket.send('{"type": "ForceEndpoint"}')

What your orchestrator owns. Barge-in. When a SpeechStarted message arrives while the agent is still talking, something has to decide whether that’s a real interruption or a backchannel. When a user says “uh-huh” or “right” while the agent is talking, that is encouragement, not a request to stop. Treating it as an interruption makes an agent that trails off every three seconds. LiveKit and Pipecat both ship interruption handling you can configure; if you wrote your own loop, this policy is yours.

4. The LLM

Once the turn closes, the transcript goes to a model. This stage owns system prompt injection, conversation history, and tool definitions, and it is the stage you configure most.

What AssemblyAI covers. Optionally, model access. LLM Gateway gives you one API to GPT, Claude, Gemini, and other models, so switching models is a parameter change rather than a new integration. The streaming API can also send each finalized turn to LLM Gateway for you and return the response on the same socket as an LLMGatewayResponse message, which saves a hop when you want it.

What your stack owns. Everything about the conversation: the prompt, the history window, which model, and how you stream tokens to TTS. To keep latency down, start synthesizing as soon as the first sentence of the reply is ready instead of waiting for the full completion.

5. Tool calls

Tool calling is where voice agents fail most visibly. The LLM decides it needs to look up an appointment, and the arguments it passes are only as good as the transcript they came from. A tool call with a misheard surname or a wrong digit in a policy number fails, or worse, succeeds against the wrong record.

What AssemblyAI covers. The input side. That’s why the entity rows in the benchmark table matter more than the headline WER: names at 10.9% error, codes and IDs at 10.0%, phone numbers at 2.4%. Keyterms and agent_context push those numbers further in your favor for your own vocabulary.

What your orchestrator or LLM owns. Tool definitions, execution, argument validation, retries, and what the agent says while a slow lookup runs. Silence is what makes people hang up, so have the agent say something like “still pulling that up” while your backend works. Validate arguments against the transcript before you act on them, and gate write actions (rescheduling, refunds) behind an identity check.

6. Text-to-speech

The model’s reply is synthesized and streamed back as audio. Phone and web agents need a voice built for conversation, not narration: one that handles hesitations, restarts, and interruptions, and reads digits and names correctly.

What AssemblyAI covers. The Text-to-Speech API, launching October 28, 2026, is built for exactly that, and it means speech in and speech out from one vendor: one contract, one bill, one support channel for both ends of the conversation.

What your orchestrator owns. Streaming playback, chunking the LLM output into speakable pieces, and flushing buffered audio the moment a real interruption is detected. After each reply, send the text back to the speech model as agent_context so the next user turn is heard in context.

7. Transport and telephony

The last stage gets audio to and from the caller. On the web that’s usually WebRTC; on the phone it’s SIP or a carrier media stream from a provider like Twilio.

What AssemblyAI covers. The streaming endpoint at wss://streaming.assemblyai.com/v3/ws accepts the encodings telephony uses, including pcm_mulaw, at sample rates from 8,000 Hz up, so you don’t need to transcode phone audio. There’s no cap on open sessions; new sessions per minute are rate limited and scale automatically.

What your stack owns. Phone numbers, SIP trunks, DTMF, call transfer, and session lifecycle. Orchestrators like LiveKit and Vapi offer telephony integrations, and the LiveKit, Pipecat, and Vapi guides show how Universal-3.6 Pro Realtime drops into each. Sessions have a 3-hour maximum; if you set an inactivity_timeout, a KeepAlive message resets it.

Connecting: the actual code

If you use LiveKit or Pipecat, the plugin handles the socket and you set the model name. If you want to see what’s underneath, here’s the speech-to-text side of a minimal loop against the streaming WebSocket API. It authenticates with an Authorization header, streams 16 kHz PCM, waits for a final Turn, hands the transcript to your LLM, and feeds the reply back as agent_context. The LLM and TTS steps are labelled stubs for your own providers.

import asyncio
import json
from urllib.parse import urlencode

import websockets

API_KEY = "<YOUR_API_KEY>"
PARAMS = {
    "speech_model": "universal-3-6-pro",
    "sample_rate": 16000,
    "encoding": "pcm_s16le",
}
URL = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(PARAMS)}"


async def call_llm(transcript):
    ...  # your LLM, or a model through LLM Gateway; returns the reply text


async def synthesize(text):
    ...  # your TTS provider; streams audio back to the caller


async def send_audio(ws, audio_frames):
    # audio_frames: async iterator of 50 ms frames, 16 kHz PCM16 mono
    async for frame in audio_frames:
        await ws.send(frame)
    await ws.send(json.dumps({"type": "Terminate"}))


async def receive(ws):
    async for raw in ws:
        msg = json.loads(raw)
        if msg["type"] == "Begin":
            # Unrecognized parameters are ignored silently, so check the model
            print("session", msg["id"], msg["configuration"]["model"])
        elif msg["type"] == "Turn" and msg["end_of_turn"]:
            reply = await call_llm(msg["transcript"])
            # Give the speech model the agent's reply as context for the next turn
            await ws.send(json.dumps({
                "type": "UpdateConfiguration",
                "agent_context": reply,
            }))
            await synthesize(reply)
        elif msg["type"] == "Termination":
            break


async def run(audio_frames):
    async with websockets.connect(
        URL, additional_headers={"Authorization": API_KEY}
    ) as ws:
        await asyncio.gather(send_audio(ws, audio_frames), receive(ws))

A few details that matter in production. Each Turn message re-transcribes the full turn, so replace earlier partial text rather than appending to it. Unrecognized query parameters are ignored silently, which is why the code prints the model from the Begin message. And browsers can’t set headers, so browser clients connect with a temporary token generated on your server instead of the API key.

Where you steer the pipeline

These are the settings that change agent behavior most, and each one is a single field on the speech-to-text side.

agent_context

Send the agent’s last reply with every turn. It’s the cheapest accuracy win in the pipeline: a 10.2% WER reduction across 20,000 voice-agent files for one extra message per turn.

voice_focus

Set near-field for headsets and phones, far-field for rooms, kiosks, and drive-thrus. This is the highest-leverage single setting for agents deployed in noisy physical environments, and getting it wrong is a common cause of “the model transcribes the person behind me.”

Turn silence windows

Instead of guessing at VAD thresholds, start from the preset defaults in the turn detection docs and adjust two numbers: min_turn_silence and max_turn_silence. Short windows suit quick, transactional exchanges. Longer windows suit turns that contain an address, a dosage, or a policy number. Raise them just before you ask for one, then put them back.

keyterms_prompt

Up to 100 terms per session, updatable mid-stream. Put your product names, staff names, and account prefixes here, and refresh the list when the conversation changes topic.

Observability: where to look when it breaks

When an agent gives a wrong answer, you need to see in one place whether the transcript was wrong, the model reasoned badly, or the tool returned bad data. Debugging that across separate providers means correlating several sets of logs on different clocks, and it’s where teams lose days.

The speech layer gives you plenty to log: every Turn message carries the transcript, word-level timestamps and confidence, and an end_of_turn_confidence score; Begin echoes the configuration the server applied; Termination reports audio and session duration. Log those next to your LLM requests and tool results with a shared call ID, and most “the agent misheard” tickets resolve in minutes.

What a speech layer doesn’t do

A clear list is more useful than a feature grid, so here’s what sits outside AssemblyAI’s part of the pipeline.

It isn’t the agent. AssemblyAI is the platform you build on, not an agent platform you build inside. Conversation state, prompts, and the turn loop live in your orchestrator.

It doesn’t run your tools. Tool definitions, execution, and guardrails on arguments belong to your LLM and backend.

It doesn’t own your phone numbers. Telephony, SIP, and DTMF come from your transport layer or orchestrator.

It doesn’t decide barge-in. The model tells you when speech starts and when a turn ends. Whether to stop the agent mid-sentence is your policy.

If you’re comparing full stacks rather than stages, our roundup of the best voice agent APIs covers the trade-offs between managed platforms and composed pipelines.

Healthcare and PHI workloads

AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI. Details are on our BAA legal page.

Scope matters in a composed pipeline: the BAA covers the AssemblyAI services you use, and every other provider that touches PHI in your stack (LLM, telephony, logging) needs its own agreement. For clinical vocabulary, Medical Mode adds +$0.15/hr to streaming. AssemblyAI also holds SOC 2 Type 2, ISO 27001:2022, and PCI DSS v4.0.

What each stage costs

Speech-to-text is the line item you can predict exactly. Universal-3.6 Pro Realtime is $0.45/hr base, billed per second, pay-as-you-go with no minimums. Keyterms prompting is included. Add-ons are priced separately: streaming diarization +$0.12/hr, Voice Focus +$0.10/hr, Medical Mode +$0.15/hr.

The LLM is billed by whichever provider you use, directly or through LLM Gateway. The Text-to-Speech API comes from the same vendor as your speech-to-text. Your orchestrator and telephony provider bill on their own terms. Current pricing for every AssemblyAI product is on our pricing page.

That’s the real argument for composing the pipeline yourself: each stage has its own price, its own knob, and its own owner, and when one underperforms you swap that stage instead of migrating everything.

Frequently asked questions

What are the stages of a voice agent pipeline?

A voice agent pipeline runs seven stages in order: noise handling, streaming speech-to-text, turn detection, the LLM, tool calls, text-to-speech, and transport or telephony. Each stage streams into the next, so total latency is the sum of the handoffs. AssemblyAI covers the speech stages at both ends; an orchestrator such as LiveKit, Pipecat, Vapi, or your own loop connects them to the LLM.

Which parts of a voice agent does AssemblyAI provide?

AssemblyAI provides the speech layer. Universal-3.6 Pro Realtime handles noise isolation through Voice Focus, streaming speech-to-text, and end-of-turn detection, and the Text-to-Speech API handles the agent’s voice. LLM Gateway is available if you want one API to multiple LLMs. Your orchestrator owns the conversation loop, tool execution, barge-in policy, and telephony.

How does turn detection work in Universal-3.6 Pro Realtime?

Universal-3.6 Pro Realtime combines semantic context with voice activity rather than relying on silence alone. When the caller pauses, the model judges whether what they said sounds complete, and it holds the turn open if they stopped partway through an entity like a phone number. Median endpoint latency is 537 ms, and you can tune min_turn_silence and max_turn_silence or force an endpoint mid-stream.

What is agent_context and why does it matter?

agent_context is a streaming parameter that passes the voice agent’s last spoken reply to the speech model as context for the next user turn. It helps the model resolve short or mumbled answers against the question that was asked. Across 20,000 voice-agent audio files, passing agent_context cut word error rate by 10.2%.

How accurate is Universal-3.6 Pro Realtime on voice agent audio?

On AssemblyAI’s English voice-agent benchmark of 12,460 scripted scenarios, Universal-3.6 Pro Realtime records a 5.19% word error rate and a 14.4% entity error rate, compared with 13.50% and 30.1% for Deepgram Flux EN, 7.78% and 18.5% for ElevenLabs Scribe v2, and 8.64% and 26.1% for Deepgram Nova-3. On Pipecat’s open STT benchmark it posts a 0.96% pooled semantic WER.

How much does the speech layer of a voice agent cost on AssemblyAI?

Universal-3.6 Pro Realtime costs $0.45/hr base, billed per second with no minimums, and keyterms prompting is included. Optional add-ons are streaming diarization at +$0.12/hr, Voice Focus at +$0.10/hr, and Medical Mode at +$0.15/hr. The free tier includes up to 333 hours of streaming with no credit card. LLM, text-to-speech, and orchestration costs depend on the providers you choose.