What it takes to build smarter voice agents: lessons from Retell and Super

A voice agent demo on a quiet MacBook is one thing. The same agent in production — bad connections, background noise, a caller who has already answered this question three calls ago — is another thing entirely. At a recent meetup and live panel at AssemblyAI's San Francisco office, two teams who ship voice agents for a living talked candidly about the gap between the two.

The panel brought together Zhongren Shao, a senior software engineer at Retell (a platform for building any kind of voice agent), and Adam Schuld, CTO of Super (the voice layer for property management), moderated by AssemblyAI's VP of Customer Solutions, Ryan Seams. Here's what stood out.

Why both teams still run a cascading architecture

The first question was the one everyone building voice agents wrestles with: cascading pipeline (separate speech-to-text, LLM, and text-to-speech) or an end-to-end speech-to-speech model? For both teams, the answer is overwhelmingly cascading — roughly 99% of Retell's agents.

The reason is control. A cascading stack lets you configure and optimize each layer independently, and — crucially — swap in a new model the day it launches without re-architecting. As Adam put it, if voice infrastructure is core to your IP, the deconstructed pipeline gives you the granular control to benchmark a new model and ship it the next day. It also opens up small but useful tricks: because Super can listen to the LLM's token stream, it can catch a URL the model is describing mid-sentence and text it to the caller in the moment.

The takeaway for builders weighing speech-to-speech: it's a real tradeoff, not a clear win. A single S2S provider can be convenient and shave a network hop, but it tends to lock you in until that provider's next model. If world-class voice is your product, the flexibility to pick the best model at each node usually wins. (We've written more about where voice agent stacks start showing their limits.)

The latency budget rules everything

Voice is unforgiving about time. Both teams anchor to a latency budget of roughly one to one-and-a-half seconds for a response — and, counterintuitively, faster isn't always better. Below about one second, Adam noted, the agent starts replying while you're still thinking, which feels unnerving rather than snappy.

Inside that budget, you make your own accuracy tradeoffs. Super optimizes aggressively for low latency everywhere and papers over minor LLM slips with better prompting and harness engineering. On the transcription side, latency is rarely the bottleneck — Zhongren pegged Retell's P50 speech-to-text time at around 500–700 milliseconds, well within budget. That headroom is exactly why the accuracy and context of your speech-to-text layer matter more than shaving another few milliseconds off it.

Evaluation is built in layers — and a lot of it is still vibes

How do you decide which STT, LLM, or TTS to use, and when to switch? Both teams evaluate layer by layer. For transcription, they run word error rate benchmarks on every new model — but both were quick to say WER isn't enough. Getting a number wrong in a phone number or an order ID is far worse than missing an ordinary word, so entity-level accuracy matters more than the aggregate. (This is a theme we keep coming back to: word error rate is broken as a single measure of quality.)

For LLMs, they run curated hard cases and use an LLM as judge. For TTS, it comes down to human vibe checks — as the panel joked, "vibes, and then more vibes." Layered on top are deterministic metrics like call duration and number of turns. The honest through-line: evaluation is as deep as the time you invest in it.

Turn-taking: the hardest thing to measure

Making an agent feel natural is largely a vibe-driven process, and the panel had the scar tissue to prove it. Super once added a typing sound to mask latency; early demo users liked it, until a wave of later users felt actively deceived — it read as trying to trick them into thinking a human was on the line. Lesson learned, feature removed.

On the mechanics, the advice for new builders was refreshingly practical. Start with the turn detection your ASR provider already ships — AssemblyAI's realtime API includes end-of-turn detection that reads tonality, pacing, and rhythm rather than just silence. From there, Retell trained its own turn-taking model on past conversations, and tunes for three behaviors: never interrupt the human, treat human speech during the agent's turn as an interruption to yield to, and don't let filler words like "um" count as interruptions.

Context is the difference between a demo and a product

Asked to name the single biggest success factor for a production voice agent, Adam didn't hesitate: context. The demo wins on wow factor; the product wins when the agent has intuition — awareness of who the caller is and what they've already said.

The data backs him up. Super found a 95% negative sentiment when a caller has to repeat information they already gave — tell the agent your name and that you're a tenant, then get asked again three calls later, and that call is going south. Both teams have built machinery to prevent it: Super runs a "scratchpad," a structured store that listens in parallel and fills out a customer profile the agent can draw from; Retell extracts dynamic variables from earlier turns and connects a mini-CRM to HubSpot or Salesforce for persistent caller identity.

This is the same problem AssemblyAI's Universal-3.5 Pro Realtime attacks at the transcription layer. Passing the agent's own question to the model as context — so a mumbled reply, a spelled-out account ID, or an address resolves correctly — cut word error rate by 10.2% across 20,000 voice-agent audio files. Context isn't only an application-layer concern; it starts with what your speech-to-text model knows about the conversation.

Getting to production: fallbacks, routing, and your real moat

Both teams agreed the boring engineering fundamentals carry the day. Measure before you ship, ground model choices in quantifiable metrics, and — as Adam admitted, to the discomfort of some engineers in the room — sometimes you have to test in production, because you can't anticipate every accent, demographic, and edge case in advance. His sharper point: those hard-earned production lessons are your moat, and they should feed back into the product.

Retell's approach to reliability is defense in depth: fallbacks for every layer. If a TTS, ASR, or LLM provider degrades, monitoring jobs detect it and route traffic away with a hard cutover — deliberately not running providers in parallel, which would blow up cost. LLM calls are latency-routed across deployments on OpenAI, Azure, and others. Retell's live ASR stack, for the record, runs AssemblyAI alongside Azure, Deepgram, and Sonix.

Controlling cost without wrecking the experience

An audience founder asked how to keep voice costs sane at scale. The panel's advice: measure first — back-of-napkin estimates rarely match production, where some calls run 20–30 minutes you never anticipated. Once you see where the tokens go, you have options. Turning reasoning down to near-nothing had only minor impact on Retell's agents, and reasoning tokens are a big driver of ballooning LLM cost. Older or open-source models can cover a surprising amount — Retell found open models nearly as capable as Claude for its agent-building "Conductor" feature — though GPU prices make self-hosting its own tradeoff.

What the panel is excited about next

Two forward-looking threads stood out. Adam pointed to "loop engineering" — a term coined by former Google Chrome lead Addy Osmani — where instead of hand-iterating on prompts, you define an achievable metric and let an agent run the improvement loop itself. Zhongren described the same idea applied to voice: self-evaluating agents that surface the calls that went badly, diagnose them, and improve the deployed agent without a human in the loop.

Both visions depend on the same foundation the whole panel kept circling back to: accurate transcription, rich context, and the flexibility to swap in the best model at every layer. That's the layer AssemblyAI is built to be — invisible infrastructure the rest of your agent can rely on.

Frequently asked questions

Cascading vs. speech-to-speech: which should I use for a voice agent?

Both teams on the panel run cascading pipelines (separate STT, LLM, and TTS) for about 99% of their agents, because it lets them optimize and swap each layer independently. Speech-to-speech can be convenient and marginally lower-latency, but tends to lock you into one provider. If flexibility and best-in-class quality at each layer matter, cascading usually wins.

What's a good latency target for a voice agent?

The panel anchors to roughly one to one-and-a-half seconds for a response. Faster than one second can actually feel unnerving, because the agent replies before the user has finished thinking. Speech-to-text is rarely the bottleneck — Retell reported a P50 around 500–700ms.

How do teams evaluate voice agents beyond word error rate?

They benchmark WER on every new STT model but weight errors by impact — a wrong digit in a phone number or order ID matters more than a missed common word. LLMs are evaluated with curated hard cases and LLM-as-judge; TTS relies on human vibe checks; and deterministic metrics like call length and turn count round it out.

How do you stop a voice agent from asking for information the caller already gave?

Maintain conversation context. Super uses a parallel "scratchpad" that builds a caller profile; Retell extracts dynamic variables and connects a CRM for persistent identity. At the transcription layer, AssemblyAI's Universal-3.5 Pro Realtime carries conversation context automatically and accepts the agent's question as context, cutting WER by 10.2% across 20,000 agent audio files.

How do production voice teams handle provider outages?

With fallbacks for every layer. Retell monitors each provider's health and does a hard cutover when one degrades, rather than running providers in parallel (which would multiply cost), and latency-routes LLM calls across multiple cloud deployments.

‍