A speech-to-speech voice agent is any system where a person speaks, the system understands and responds, and the response comes back as speech. That's the whole definition—it describes the input and output, not the internals.
The internals are where the two architectures diverge:
- Cascaded (STT → LLM → TTS): three specialized components in a chain. Speech-to-text transcribes, an LLM reasons, text-to-speech responds.
- End-to-end (duplex): a single model that takes audio in and emits audio out directly, with no explicit text stage.
Both are "speech-to-speech." Only one of them is something you should put in front of customers today.
The cascaded architecture, step by step
In a cascaded pipeline, each stage does one job well:
- Streaming STT turns the caller's audio into text in real time, emitting partial and final transcripts as they speak.
- An LLM takes each finalized turn and decides what to say—looking up an account, answering a question, calling a tool.
- TTS converts the answer back into natural speech.
The advantage is control and swappability. You choose the LLM, the voice, and the logic in between, and you can upgrade any one component without rebuilding the others. The knock on cascaded architectures used to be latency—three hops instead of one—but modern streaming speech-to-text has closed that gap to the point where a well-built cascade responds in about a second end-to-end.
Here's the speech layer of that cascade in Python. Note the streaming parameter is speech_model (singular) and auth is the raw API key with no Bearer prefix:
# pip install "assemblyai>=1.0.0"
import os
from assemblyai.streaming.v3 import (
RealTimeTranscriber, RealTimeTranscriberOptions, RealTimeEvents,
RealTimeParameters, RealTimeSessionParameters, TurnEvent,
)
def on_turn(_, event: TurnEvent):
tag = "FINAL" if event.end_of_turn else "partial"
print(f"{tag}: {event.transcript}")
client = RealTimeTranscriber(RealTimeTranscriberOptions(),
api_key=os.environ["ASSEMBLYAI_API_KEY"])
client.on(RealTimeEvents.Turn, on_turn)
client.connect(RealTimeParameters(sample_rate=16000,
speech_model="universal-3-6-pro"))
# feed 16kHz mono PCM16 chunks (50-1000ms) via client.stream(chunk)
client.disconnect(terminate=True) # ALWAYS terminateEach FINAL turn is your handoff point to the LLM. The connection runs over wss://streaming.assemblyai.com/v3/ws—the v3 endpoint; the old /v2/realtime/ws path is inactive and returns HTTP 410.
The end-to-end architecture—and the reality check
End-to-end duplex models are genuinely exciting. One model, audio in, audio out, potentially lower latency and more natural prosody because nothing gets flattened into text and back.
But here's the reality: "native duplex speech-to-speech is a research and roadmap direction—a 2027 bet—not production infrastructure you should build a business on in 2026." The models that exist are early, hard to steer, and difficult to make compliant or debuggable. Every serious production voice agent today runs cascaded or direct-API. So if a vendor is selling you an "end-to-end speech-to-speech model" as your production stack right now, ask hard questions.
Where AssemblyAI fits: Universal-3.6 Pro Realtime + Text-to-Speech API
AssemblyAI offers both speech ends of the cascaded architecture: Universal-3.6 Pro Realtime for streaming speech-to-text, and AssemblyAI's Text-to-Speech API for the voice coming back. You bring the LLM and orchestrate the loop with LiveKit, Pipecat, or your own code.
When AssemblyAI talks about the fastest way to build a speech-to-speech voice agent, it means building the cascaded architecture on those two APIs, not an end-to-end S2S model.
Skip the plumbing. Drop Universal-3.6 Pro Realtime and AssemblyAI's Text-to-Speech API into LiveKit or Pipecat and the cascaded STT-LLM-TTS architecture wires itself.
The cascaded advantage nobody talks about: agent_context
Cascaded architecture offers something end-to-end models cannot easily provide: a place to inject live context into the speech layer. Universal-3.6 Pro Realtime takes an agent_context parameter—you tell the model what the agent just asked, so it knows what the caller is likely about to say. Across 20,000 voice-agent files, that cut WER by 10.2%.
client.connect(RealTimeParameters(
sample_rate=16000, speech_model="universal-3-6-pro",
agent_context="What's your email address?",
))
# after each agent reply:
client.set_params(RealTimeSessionParameters(agent_context="Sure, what date would you like to book?"))
You seed it at connect time, then push updates mid-stream with set_params after each agent turn. That feedback loop—agent tells STT what it expects—is only possible because the stages are separate. It's a concrete reason cascaded still wins in production.
Latency and accuracy: what the numbers say
The old assumption that cascaded is too slow doesn't survive contact with current benchmarks. Independent testing from Coval gives Universal-3.6 Pro Realtime the lowest WER on its STT benchmark: 2.2% over a 7-day window. On Pipecat's open STT benchmark, the final transcript lands a median 307ms after the caller stops speaking. End-to-end, a well-built cascade lands around a second.
On accuracy, AssemblyAI's English voice-agent benchmark of 12,460 scripted voice-agent scenarios puts Universal-3.6 Pro Realtime at 5.19% WER—versus Deepgram Flux EN at 13.50%, ElevenLabs Scribe v2 at 7.78%, and Deepgram Nova-3 at 8.64%. That 5.19% is the accuracy floor your cascaded agent inherits, and it's the number that decides how often the LLM downstream gets clean input. Full detail lives on the benchmarks page and in the Universal-3.6 Pro Realtime research write-up.
5.19% WER is the number your whole cascade inherits. Run Universal-3.6 Pro Realtime on your own audio and see the streaming accuracy and latency in action.
Interruptions, barge-in, and end-of-turn
A real conversation isn't tidy. People interrupt, trail off, and talk over each other. Two capabilities make an agent feel human here.
Barge-in lets the caller cut off the agent mid-sentence, and the agent stops and listens. End-of-turn detection decides when the caller has actually finished, so the agent responds at the right moment instead of stepping on them. Universal-3.6 Pro Realtime exposes latency modes—min_latency, balanced (default), and max_accuracy—to tune that tradeoff. If you're building on LiveKit or Pipecat, Context Carryover works out of the box, and on LiveKit you should use its English turn-detection model rather than turn_detection="stt".
Cascaded vs end-to-end: which in 2026?
| Criterion | Cascaded (STT-LLM-TTS) | End-to-end duplex |
|---|---|---|
| Production-ready in 2026 | Yes — the standard | No — 2027 roadmap direction |
| Control over LLM & voice | Full, swappable | Limited |
| Live context injection | Yes (agent_context, −10.2% WER) | Hard |
| Debuggability & compliance | Strong (inspect each stage) | Immature |
| Latency | ~1s end-to-end; 307ms median STT final (Pipecat) | Potentially lower, unproven at scale |
For 2026, cascaded is the answer for anything you're putting in production. You can build it yourself with Universal-3.6 Pro Realtime and AssemblyAI's Text-to-Speech API on LiveKit, Pipecat, or your own loop—or go through a partner platform like Vapi, which many teams build through. Teams like Vapi and LiveKit already run production voice agents on this architecture.
Want to build one hands-on? The Python voice agent tutorial and the vibe-coding walkthrough take you from zero to a working agent, and the production-ceiling piece covers what breaks at scale.
Start building
Whether you assemble the cascade yourself or lean on LiveKit or Pipecat to wire it, the speech layer is the input everything else depends on.
Explore Voice AI solutions to see the full stack, see how teams build voice agents on it, or get your free API key and wire up the streaming code above.
Cascaded is the production answer today. Get a free API key, wire up the streaming code above, and pair it with AssemblyAI's Text-to-Speech API for the voice coming back.
Frequently asked questions
What is speech-to-speech?
It's an architecture, not a product: audio in, audio out, with a conversation in between. Today, production speech-to-speech agents are built as cascaded STT-LLM-TTS pipelines rather than single end-to-end models.
What's the fastest way to build a speech-to-speech voice agent?
Drop Universal-3.6 Pro Realtime and AssemblyAI's Text-to-Speech API into an orchestration framework like LiveKit or Pipecat. The framework handles transport, turn-taking, and interruptions; you plug in your LLM and ship the cascaded architecture without hand-wiring three vendors.
What's the best streaming model for a voice agent?
Universal-3.6 Pro Realtime: 5.19% WER on AssemblyAI's English voice-agent benchmark, the lowest WER on Coval's independent STT benchmark (2.2%), and final transcripts a median 307ms after the speaker stops on Pipecat's open benchmark. It also takes agent_context to cut WER a further 10.2% using live conversation context.
What is the cascading architecture?
It's the STT → LLM → TTS chain: speech-to-text transcribes the caller, an LLM decides the response, and text-to-speech speaks it back. Each stage is separate and swappable, which is why it dominates production voice agents.
Does AssemblyAI offer an end-to-end speech-to-speech model?
No. The company's offering is Universal-3.6 Pro Realtime plus the Text-to-Speech API—both speech ends of the cascade. "Native duplex speech-to-speech is a roadmap direction for 2027, not a product" AssemblyAI sells today.
Can a speech-to-speech agent handle interruptions?
Yes. Barge-in lets callers interrupt the agent, and end-of-turn detection—tunable via latency modes—decides when they've finished so the agent responds at the right moment.
