Build smarter voice agents: 7 takeaways from our July 23rd San Francisco meetup

We packed our San Francisco office for an evening on one question: what does it actually take to build a voice agent that survives contact with the real world?

The night opened with a live demo of Universal-3.5 Pro Realtime, our newest realtime speech-to-text model, showing how features like Context Carryover, conversation memory, multilingual code-switching, and Voice Focus help agents hold onto what a caller actually said. Then we handed the mic to two people shipping voice agents in production: Zhongren Shao, Sr. Software Engineer at Retell (an early adopter of Universal-3.5 Pro Realtime), and Adam Schuld, CTO at Super, the voice layer for property management. Ryan Seams, our VP of Customer Solutions, moderated.

The conversation was refreshingly unglamorous — less “look what voice can do” and more “here’s what broke, and here’s what we did about it.” If you’re building voice agents today, these are the takeaways worth stealing.

1. Both builders bet on cascading architectures — for control

The first question out of the gate: cascading pipeline (separate speech-to-text, LLM, and text-to-speech) or a single speech-to-speech model?

Retell runs cascading for roughly 99% of its agents. The reason is customization — being able to swap and individually optimize each layer. “It was also easier to individually optimize the different parts of the stack,” Shao said. When a new model drops, you test it, benchmark it, and ship it the next day without re-architecting anything.

Schuld framed it as a question of where your IP lives. If voice infrastructure is core to your product, the deconstructed pipeline buys you granular control over every node. He gave a nice example: because Super listens to the LLM’s event stream, it can catch a URL mid-sentence and fire it off as an SMS in the moment — the kind of thing a sealed speech-to-speech box won’t let you do.

The through-line: flexibility wins. Nobody wanted to be married to a single provider’s release cycle.

2. Everything lives inside a 1–1.5 second latency budget

Schuld’s rule of thumb: you have roughly one to one-and-a-half seconds to respond. Inside that budget, you can spend on accuracy, reasoning loops, or a more elaborate script — your call. Go over it and the conversation feels broken.

The counterintuitive part? Faster isn’t always better. Under about one second, “it becomes unnerving for customers,” Schuld said — the agent replies while a human would still be thinking. Natural beats instant.

Retell optimizes hard for low latency everywhere, then papers over small LLM inaccuracies with better prompting and harness engineering. Worth noting where the time actually goes: Shao pegged transcription P50 at 500–700ms and pointed out that text-to-speech latency barely matters once you’re streaming the first token. The bottleneck is getting that first LLM token out the door.

3. Evals come in layers — and the top layer is still vibes

Ask how these teams evaluate models and you get a surprisingly honest answer: it depends on the layer.

For speech-to-text, Retell runs an automated word error rate benchmark on every new model — then goes beyond it, because not all errors are equal. “If you get a number wrong, that’s usually much worse than getting a particular word wrong,” Shao noted — a good argument for looking past a single WER number. For LLMs, both teams lean on curated hard cases and LLM-as-judge scoring. Super adds deterministic signals: call length, number of turns, whether the agent is looping.

And for text-to-speech? “Vibe checks by people.” There’s no clean number for whether a voice sounds right, so humans still decide. As Seams put it, “vibes and then more vibes.” It’s a good reminder that evaluating speech models is part science, part judgment — and pretending otherwise doesn’t ship better agents.

4. Sounding human is mostly turn-taking — and you can overdo it

The mechanics of a good conversation are surprisingly concrete: acknowledgments, light repetition of what the caller said, expectation-setting. Retell layers on optimizations to avoid interrupting the human, to treat human speech as a real interruption of the agent, and to ignore filler words like “um” so they don’t trigger a false turn. Many teams start with the turn detection built into their speech-to-text provider — Universal-3.5 Pro Realtime reads tonality, pacing, and rhythm rather than just silence — then tune from there.

But Schuld’s best story was a cautionary one. To mask latency, Super once added a typing sound during pauses. Early demo users loved it. Then it reached people who felt deceived — it read as a trick to make them think a human was on the line. They pulled it. The lesson: natural is the goal, not the illusion of a human.

5. Context is the moat

If there was one word the panel kept circling back to, it was context.

Super found a 95% negative sentiment when callers have to repeat information they already gave — name, address, whether they’re a tenant or owner. So the agent needs a mechanism to carry what matters across a conversation (and across calls). Super built a “scratchpad,” a structured store that listens in parallel and fills out a caller profile. Retell does dynamic variable extraction plus a mini-CRM that connects to HubSpot or Salesforce, so an agent can greet you by name on the next call.

When Seams asked for the single biggest driver of a successful call, Schuld didn’t hesitate: “For me, it’s context.” It’s what turns a flashy demo into a system that feels like it has intuition — and it’s the piece most teams underinvest in. This is exactly the problem Context Carryover in Universal-3.5 Pro Realtime was built to solve at the transcription layer: passing the agent’s own question in as agent_context cut word error rate by 10.2% across a 20,000-file benchmark, because the model resolves a mumbled account number or one-word confirmation in the context of what was actually asked.

6. Your production failure modes are your product

The gap between a quiet MacBook demo and a real deployment — bad connections, background noise, accents, edge-case callers — is where voice agents earn their keep.

Retell’s answer is hard engineering: fallbacks for every layer, so if a text-to-speech, speech-to-text, or LLM provider degrades, monitoring jobs detect it and route traffic away mid-call. They run latency routing across multiple LLM deployments too — best-effort on KV cache, but fallback availability comes first. Notably, they don’t run providers in parallel, because that would blow up cost; they assume the primary works until something says otherwise, then hard-cut.

Schuld’s take was the quote of the night: those hard-won lessons from production are the moat. “If you look into what makes your product differentiated, it should be using those hard-earned lessons.” The production ceiling is real — and the teams that climb it turn every failure into a feature.

7. On cost: measure first, then turn down the knobs

An audience member building AI learning experiences asked how to keep voice costs sane. The panel’s advice was practical.

First, measure. Back-of-napkin estimates rarely survive real conversations that run 20 or 30 minutes. “You can’t improve what you don’t measure first,” Schuld said. Once you know where the token spend goes, the cascading pipeline gives you options — trade a little latency for a cheaper model configuration where it’s acceptable.

Shao’s tactical tip: turn reasoning down. For many newer models, dialing reasoning to near-zero had only a minor impact on Retell’s agents — and reasoning tokens are a big reason LLM costs have ballooned. Older, cheaper models can quietly do the job in plenty of cases too.

What they’re watching next

Asked what excites them, both pointed at self-improving systems. Schuld described “loop engineering” — instead of hand-tuning prompts, you define a metric an agent can optimize against, and let it run the iterative loop. Shao wants self-evaluating agents that surface their own bad calls, diagnose what went wrong, and improve without a human in the loop. Same instinct, different entry point: take the human out of the tuning cycle.

The thread that tied it together

Strip away the specifics and one idea ran through every answer: the model has to listen well before anything downstream can be good. Flexible pipelines, layered evals, turn-taking, context stores, fallbacks — they’re all ways of protecting the quality of what the agent hears. If the transcription is wrong, the LLM confidently answers the wrong question, and no amount of prompt engineering saves the call. It’s the same conclusion we reached in our 2026 report on what actually makes a good voice agent.

That’s the bet behind Universal-3.5 Pro Realtime: 6.99% pooled word error rate on Pipecat’s open STT benchmark, agent_context and rolling conversation memory on by default, Voice Focus to isolate the primary speaker, and code-switching across 18 languages — the input layer, built to hold up on real calls.

Want the whole conversation? Watch the full panel above. And if you’re ready to build, grab a free API key or see how the Voice Agent API works — one WebSocket for speech-to-text, LLM, and text-to-speech, flat rate, no juggling providers.

Thanks to Zhongren, Adam, and everyone who came out. If you missed our last one, here’s the recap from our NYC voice agents meetup — and keep an eye out for the next.