Now live New models for transcription and text-to-speech. AssemblyAI-STT-3.7 AssemblyAI-TTS-1
Insights & Use Cases

Voice agent architectures explained: STT→LLM→TTS vs. speech-to-speech vs. orchestrated pipeline

Every voice agent does the same three things: it listens, it thinks, and it talks back. The interesting part is how — and that splits into three architectures, each with different trade-offs on latency, control, cost, and accuracy. Here are the three patterns you can build on, and why the step everyone treats as plumbing is actually the ceiling on how good your agent can be.

Abstract green sphere illustration

Written by

Kelsey Foster

Published on

18 November 2026

Every voice agent does the same three things. It listens, it thinks, and it talks back. The interesting part isn't the what — it's the how. And the "how" splits into three very different architectures, each with its own trade-offs on latency, control, cost, and the one thing that quietly decides whether your agent is any good: accuracy.

So before you wire anything together, it's worth understanding the three patterns you can choose from — and why the step everyone treats as plumbing is actually the ceiling on how smart your agent can be.

The job of a voice agent: hear, think, speak

Strip away the framework names and a voice agent is a loop. A person says something. The agent turns that audio into text (speech-to-text, or STT). A language model reads the text and decides what to say (the LLM). That response gets turned back into audio (text-to-speech, or TTS). Then it waits for the next thing the person says and does it all again — ideally in about a second, because anything slower feels like a bad phone connection.

Here's the thing most architecture debates miss: the model that thinks is only ever reacting to what the first step heard. If the STT layer mishears "my account ends in 4-0-1-5" as "4-0-1-8," the LLM confidently helps with the wrong account. Get the input wrong and everything downstream is wrong too. Keep that in mind — it's the theme that runs through all three architectures below.

Architecture 1 — the chained STT→LLM→TTS pipeline

This is the classic approach, sometimes called the cascading or chained architecture. You pick a best-in-class model for each stage and connect them: a streaming speech-to-text API feeds transcripts to an LLM, and the LLM's response goes to a text-to-speech engine. Turn detection decides when the user is done talking so the LLM knows when to respond.

The upside is control. You choose the most accurate STT model, the LLM that best fits your use case, and the voice you want. You can tune each stage independently — swap the LLM without touching transcription, add domain vocabulary at the STT layer, change voices at the TTS layer. For teams that need a genuinely differentiated product, that granularity matters.

The cost is plumbing. You're now integrating three providers, reconciling three sets of latency, and debugging across three surfaces. Every millisecond of end-to-end latency is the sum of each hop plus the handoffs between them. It's the most flexible architecture and the most work.

AssemblyAI can fill both speech slots in a chained pipeline: AssemblyAI-STT-3.7-Realtime-Agent as the ears and the AssemblyAI Text-to-Speech API as the voice, with whichever LLM you choose in between. If your stack already calls an OpenAI-style speech endpoint, AssemblyAI TTS answers the same request shape through its OpenAI compatibility route, so swapping the voice stage comes down to a new base URL and API key. TTS is billed at $15 per 1M characters (see pricing).

Architecture 2 — a single speech-to-speech model

The newer approach collapses the pipeline into one multimodal model that takes audio in and emits audio out, reasoning over speech directly. It's simple to start with: one API, one round trip.

But simplicity has a price, and it's control. When a single model does everything, you can't swap the "ears" for a more accurate transcription model, you can't inspect the transcript to see what it actually heard, and you can't tune turn-taking or entity recognition stage by stage. These models are also built primarily around conversational fluency rather than getting a spelled-out email address or a 16-digit card number exactly right. When the same black box handles listening and speaking, an input error is invisible until the agent says something wrong.

You also inherit the model's own voices and its pricing for both directions of audio, with no way to change one stage without changing the whole thing. Speech-to-speech can work for casual, low-stakes conversation. It is a poor fit when your agent has to capture names, numbers, and identifiers correctly on the first try.

Architecture 3 — the orchestrated pipeline on one provider

There's a third option that's easy to miss because it looks like the simple one from the outside and behaves like the flexible one on the inside: the same STT→LLM→TTS pipeline, with all three stages from one provider on one API key, and an open orchestration framework like LiveKit or Pipecat (or your own loop) doing the wiring.

On AssemblyAI, that stack is AssemblyAI-STT-3.7-Realtime-Agent for speech-to-text, LLM Gateway for the language model, and the Text-to-Speech API for the voice. LLM Gateway is OpenAI-compatible: you pick a model from its list of available models, point an OpenAI-style client at it with the same AssemblyAI API key, and keep tool calling in the LLM request where it already lives. If you'd rather use your own OpenAI or Anthropic account, bring it; the speech ends stay the same. Drop the speech pieces into a LiveKit or Pipecat pipeline and the framework owns transport, turn-taking, and interruption handling.

The voice is AssemblyAI-TTS-1, built for conversational voice agents. It delivers frontier-level quality and runs faster than other production voice agent TTS models. It takes the LLM's text as it streams and starts speaking once it holds complete sentences, cancels cleanly when the caller barges in, and outputs μ-law audio for phone lines.

It's still a real pipeline — real speech-to-text, a real LLM, real text-to-speech — but you get one key, one bill, and one set of logs instead of three dashboards: streaming speech-to-text at $0.45/hr, TTS at $15 per 1M characters, and LLM Gateway billed on the input and output tokens of the model you pick (see pricing). The plugins are drop-in, and the docs work natively with tools like Claude Code so you can go from docs to a working agent the same afternoon.

The framing that matters here is "invisible infrastructure." You're not adopting an opinionated agent platform where every agent ends up sounding the same. You keep control over conversation design, tool calling, and turn timing — you just skip the plumbing. With AssemblyAI-STT-3.7-Realtime-Agent as the ears, the listening step is handled by a model designed for exactly the hard tokens that trip agents up.

Build Your Voice Agent Faster

Skip the three-provider plumbing. AssemblyAI-STT-3.7-Realtime-Agent, LLM Gateway, and AssemblyAI's Text-to-Speech API run on one API key and drop into LiveKit or Pipecat—with clear docs and a working agent the same afternoon.

Sign up free

Choosing an architecture

There's no single right answer — there's a right answer for your product. Here's how the three patterns compare on the dimensions that usually decide it.

Dimension Chained STT→LLM→TTS Speech-to-speech Orchestrated pipeline (one provider)
Control over each stage Highest Lowest High
Time to first working agent Slowest (three integrations) Fast Fast
Transcript transparency Full None Full
Billing & debugging surfaces Three One One with LLM Gateway; two if you bring your own LLM
Entity accuracy on hard tokens Depends on chosen STT Weaker Strong (AssemblyAI-STT-3.7-Realtime-Agent)

If you're a large team building a deeply custom product and you want to own every layer, the chained pipeline earns its complexity. If you're prototyping something casual, a speech-to-speech model gets you talking fast. And if you want production-grade accuracy and control without hand-wiring three vendors, an orchestrated pipeline on one provider is the pragmatic middle path — which is why it's where most teams building their first serious voice product end up.

Why the listening step is the real ceiling

Notice what every row in that table has in common: none of them can fix a transcription mistake after the fact. The LLM can only reason over the words it was handed. So the accuracy of the speech-to-text layer sets the ceiling for the entire agent, no matter which architecture you pick.

That's why the input model matters more than its "plumbing" reputation suggests. AssemblyAI-STT-3.7-Realtime-Agent is built for the tokens agents get wrong most often — emails, names, phone numbers, account IDs — with the final transcript landing 86 milliseconds after speech ends on AssemblyAI's latency benchmark, versus 172 ms for Universal-3.6 Pro Realtime, and native code-switching across 32 languages. Its turn detection combines semantic context with voice activity so the agent knows the difference between a pause and a finished thought. In a hand-built pipeline you'd wire it in as your STT layer; in LiveKit or Pipecat it's a drop-in plugin. Either way, it's doing the job that decides whether the rest of your stack has a chance.

For a closer look at where DIY stacks tend to strain under real traffic, our write-up on where voice agent stacks start showing their limits is a useful companion read.

See Voice AI in Action

Experience natural, real-time conversation that goes far beyond IVR menus. Test streaming transcription speed and accuracy on your own audio.

Try playground

The takeaway

Architecture is a series of trade-offs, but they all bottom out in the same place. Control versus simplicity, three bills versus one, transparency versus a black box — those are real decisions. What isn't up for debate is that the model doing the listening determines how good everything after it can be. Pick the architecture that fits your team, then give it the best possible ears. That's the part your customers will actually hear.

Want the hands-on version? Our Python voice agent tutorial walks through a working agent, and what are AI voice agents covers the fundamentals if you're just getting oriented.

Start Building Your Voice Agent

The fastest way to judge an architecture is to build it. See how to build voice agents on AssemblyAI-STT-3.7-Realtime-Agent and AssemblyAI's Text-to-Speech API, then start building your own.

Explore voice agent solutions

Frequently Asked Questions (FAQs)

What is a voice agent architecture?

A voice agent architecture is the way you connect the three core steps of a spoken conversation: speech-to-text (listening), a language model (thinking), and text-to-speech (speaking). The three common patterns are a chained STT→LLM→TTS pipeline, a single speech-to-speech model, and an orchestrated pipeline on one provider, where speech-to-text, the LLM, and text-to-speech share one API key and a framework like LiveKit or Pipecat (or your own loop) does the wiring.

What is the difference between a cascading (STT→LLM→TTS) pipeline and a speech-to-speech model?

A cascading pipeline uses a separate, best-in-class model for each stage, giving you full control and a visible transcript but requiring you to integrate and debug three providers. A speech-to-speech model handles audio-in to audio-out in one model, which is simpler to start with but hides the transcript and gives you less control over per-stage accuracy and turn-taking.

Which voice agent architecture has the lowest latency?

All three can reach roughly one-second end-to-end latency when tuned well. A single speech-to-speech model removes handoffs between stages, while an orchestrated pipeline on LiveKit or Pipecat with AssemblyAI-STT-3.7-Realtime-Agent as the ears gets there because the framework handles turn-taking and transport.

Why does speech-to-text accuracy matter so much for voice agents?

The language model can only respond to the words the speech-to-text layer produces, so any transcription error propagates through the entire agent. This is why entity accuracy on emails, names, phone numbers, and account IDs is the ceiling on agent quality — and why models like AssemblyAI-STT-3.7-Realtime-Agent are built specifically for those hard tokens.

What is the best API for building a voice agent?

The best choice depends on how much control you need. Teams that want production accuracy without hand-wiring three providers often run all three stages on AssemblyAI with one API key: AssemblyAI-STT-3.7-Realtime-Agent for speech-to-text, LLM Gateway for the language model, and the Text-to-Speech API for voice output ($15 per 1M characters), orchestrated with LiveKit, Pipecat, or their own loop. AssemblyAI-STT-3.7-Realtime-Agent gives the agent high entity accuracy on the tokens that matter.