Building voice agents for the real world: 7 takeaways from our NYC meetup with LiveKit

Two agents, one problem list

Building a voice agent that demos well takes an afternoon. Building one that survives contact with real people is a different job entirely — voicemail beeps, phone screeners, interrupted instructions, and evals nobody has figured out yet.

That was the theme of our latest NYC meetup, which we co-hosted with LiveKit at our office in New York. The room was full of builders who had already shipped something and were mostly there to compare scar tissue.

On stage:

• Owen Gretzinger, Head of Engineering at Boardy — think LinkedIn, except it's a person you can call and text. Boardy makes intros between founders, investors, customers, and talent across a network of roughly 200,000 people. It also called a good chunk of the room before the event to register them, which became clear when Owen asked for a show of hands and a lot of the room raised theirs.
• Dylan Wight, Software Engineer at Flagler Health — an AI platform for musculoskeletal care. Their voice agents answer inbound scheduling calls and place outbound pre-procedure calls, walking patients through which medications to stop, when to stop eating, and what to expect on the day.
• Ryan Seams, VP of Customer Solutions at AssemblyAI, moderating.

Before the panel, Ryan ran a live demo of Universal-3.5 Pro Realtime in the playground — including agent context, where you pass the agent's own question into the transcription request so short or mumbled replies resolve correctly. Across a benchmark of 20,000 voice agent audio files, passing agent context cut word error rate by 10.2%.

Then the interesting part. Boardy is open-ended, conversational, and talks to tech founders who may be wary of an AI caller at first and tend to warm up a few minutes in. Flagler talks to older patients who called a clinic because that's how you reach a clinic, and who are not expecting a synthetic voice on the other end. Different users, different regulatory surface, different definitions of success.

Same problem list: trust, turn-taking, interruptions, latency, evals, memory, and the endless inventiveness of real humans on the other end of the line.

Both teams also build on overlapping infrastructure — LiveKit for the real-time layer, with Boardy running AssemblyAI for streaming speech-to-text and Flagler running a full AssemblyAI transcription pass on every call — which made the contrast in what they built on top of it that much sharper.

1. There is no one right voice agent experience

The panel opened on the most useful disagreement of the night: how much freedom should a conversation have?

Boardy leans all the way in. The more someone talks, the more Boardy learns about their goals and priorities, and the better the intros get. Letting the conversation wander isn't a side effect — it's the product. "When you talk with someone over voice, you just say so much more," Owen said. "Whereas over text you're very dry."

Flagler does close to the opposite. Their calls follow a flowchart: collect the email, the phone number, the insurance information, or deliver this specific set of pre-op instructions. Wandering is a bug. "It's a big issue if it starts trying to give them medical advice," Dylan said. When a patient goes off script, the call escalates to a human.

The two teams even define success differently. Boardy tracks whether its intros actually convert — recently past 50% of proposed intros turning into a real meeting. Flagler's bar is lower and harder: don't make this person unhappy. "We're not really trying to delight this user," Dylan said. They want to get to their goal and not come away wishing they'd reached a human.

If you're picking an architecture or writing a system prompt based on someone else's voice agent demo, this is the thing to internalize first. The right conversational behavior is a function of who's calling and why — not a best practice you can copy. We saw the same split at our San Francisco meetup in July, and it keeps showing up.

2. Say it's an AI in the first sentence

Flagler debated whether to disclose. They landed on yes, immediately, every call: "Hey, I'm Sarah, an AI assistant from this clinic, calling about your upcoming procedure."

The reasoning is practical rather than principled. People figure it out anyway. Flagler had a handful of early calls where the patient asked, near the end, whether they were talking to a person — and that moment costs more trust than disclosure ever would.

Boardy discloses too, just with more swagger. Its opening line is often a question: is this your first phone call with an AI? "That breaks the fourth wall almost immediately," Owen said. Four minutes into a good conversation, most people forget to care.

Same conclusion from opposite directions. Disclose up front and spend your effort on making the next four minutes good, rather than on staying undetected.

There's an unglamorous half to trust, too. Flagler's calls are outbound to a number the patient doesn't recognize, so a real chunk of the work is telephony hygiene — Twilio configuration, caller ID that shows the clinic's name, and everything required to not get flagged as spam. A perfect agent that reads as "Unknown Caller" never gets to talk.

3. For Boardy, one-on-one turn-taking is largely solved. Multi-party is harder.

Ask a room of voice builders about turn-taking and you usually get war stories. Owen's answer was blunt: for one-on-one calls, it's basically solved. Between LiveKit and AssemblyAI, it works.

The hard problem moved. Boardy now joins Google Meet calls — it schedules the intro it made, then shows up to the meeting. Owen was on a video call with Boardy from the stage while he explained it.

Once there are three or four humans plus an agent in a room, the agent has to answer a question that never comes up on a phone call: is anyone talking to me right now? "False negatives and false positives are both catastrophic," Owen said. Ignore someone who addressed you directly and you look broken. Jump into a conversation that wasn't yours and you never get invited back.

Boardy's approach is a classifier that decides whether it should speak based on the conversation so far, running alongside a pipeline that preemptively generates responses so one is ready the moment the classifier says go. That matters because of how people actually talk: you start your reply as the other person is landing their last few words. Wait for silence, then run a full pipeline, then respond three seconds later, and you've already lost the turn.

Which reframes latency. It isn't an infrastructure metric you optimize because lower numbers look good on a dashboard. Below about a second, an agent can be a participant in the conversation — and sub-second is the bar Owen said Boardy holds itself to. Boardy's own take, delivered live on the panel through the agent itself: most AI interactions feel like sending a letter and waiting for a reply.

4. Your agent knows what it said, not what the user heard

This was the most quietly unsettling insight of the night, and it came out of Flagler's pre-op calls.

There's a specific block of instructions a patient needs to hear. The LLM generates it, the audio starts playing — and the patient interrupts partway through. The model knows exactly what it intended to say. It has no idea how much of it made it into the patient's ear. "The agent as we have it right now doesn't listen to itself speak," Dylan said.

In a cascading stack, nothing in the pipeline closes that loop by default. So Flagler does two things: break instructions into smaller chunks, and restart the whole block if it got interrupted. Both are workarounds, and Dylan named the tradeoff directly — long monologues are dangerous, because an interruption leaves the agent with no idea where it was. Restart and you're repetitive. Skip ahead and the patient may miss part of the instruction.

The distinction between what an agent said and what a user heard barely matters for a scheduling bot. For clinical instructions it's the entire ballgame. If your agent delivers information someone has to act on, this is worth designing for explicitly rather than discovering later.

5. The unglamorous telephony is the product

Ryan asked what surprised each team most about going from demo to production. Dylan's answer was, essentially, the phone.

More than half of Flagler's outbound calls go to voicemail. So the agent has to know it reached voicemail, know when the beep happened, and time its message correctly — which means signal detection on the beep, audio event detection, and a fair amount of retry logic when it starts talking too early. Over 50% of the time, the outcome of the call is a good voicemail that gets the patient to call back.

Then there's everything they didn't see coming. Flagler tested by calling everyone at the company. It went fine. They went live with patients and hit AI phone screeners on day one — the kind that ask the caller to state who they are and why they're calling, then go check whether the human wants to talk. Now the agent has to introduce itself to another agent, wait, and branch depending on whether a person ever picks up.

Even "did the patient answer?" turned out to be real engineering. Twilio initiates the outbound call and dials into LiveKit over a SIP trunk, so getting pickup state back to the agent — so its silence timers start at the right moment — meant finding the LiveKit room, updating its metadata, and watching for the change. None of this appears in a demo. All of it determines whether the product works.

This is the part of the stack we've written about as the production ceiling — the point where the model is fine and the surrounding system is what breaks.

6. Evals are a production bottleneck — and agents testing agents isn't enough

Both teams flagged evaluation as a major production challenge, for the same underlying reason: task completion alone can be misleading.

Flagler started with humans reading every transcript, because the system would report a completed task on a call that had clearly gone sideways. "We think it achieved the goal and it didn't," Dylan said. That review loop is how they built their eval set: catch the mismatch, figure out what happened, adjust the prompt, then test the change against the library of previous transcripts. They use Braintrust for LLM observability and a voice-agent testing service to reduce the amount of time an engineer spends personally re-running call scenarios on the phone. Good transcripts are load-bearing here — you can't evaluate what you can't read, which is also why how you evaluate transcription accuracy ends up upstream of everything else.

Boardy has about 200 messaging agent simulation scenarios — a specific person, a specific situation, and exactly what Boardy should do. For text, that works. For voice, Owen was skeptical of the whole category: even a purpose-built agent calling your agent behaves differently than a human will. "There's always things that go wrong which are really hard to catch ahead of time with evals and simulations."

Worth sitting with, because both teams are adding tools, memory, and autonomy to their agents — which widens the space of things that can go wrong faster than the eval tooling is closing it. Our 2026 voice agent insights report found the same gap across 450+ builders.

One note for anyone building in healthcare: Flagler's eval work happens inside a regulated envelope. Everything they do sits under HIPAA, every vendor that touches call data has to sign a BAA, and the platform goes through third-party security testing. That shapes what you're allowed to store, which shapes what you can evaluate. (AssemblyAI is considered a business associate under HIPAA and offers a standard Business Associate Addendum for customers processing PHI; Medical Mode is available on both async and streaming models.)

7. Memory is a retrieval problem, not a context-window problem

An audience question about how Boardy decides what to remember produced the most reusable architecture of the night.

Boardy keeps a contact file on each person — a compact document of what matters about them, always loaded into context, and directly editable by the agent. Owen's framing: Boardy is at his desk, and when you call, he pulls your sheet out of the drawer. That file is also what powers the matching, so the agent can read your stated preferences before proposing an intro.

Underneath that, every past conversation is indexed. If Boardy doesn't know something, it searches semantically across your history, pulls the relevant conversations, reads them properly, and answers from there.

The alternative is what they tried first: stuffing everything into one giant system prompt, which still couldn't surface a conversation from thousands of messages ago. Loading context on demand beats stuffing it, and Owen pointed the room at Anthropic's published guidance on exactly that pattern.

The complication is voice. Boardy's text and email harness already does compaction, long-term memory, and history search well. Doing the same over voice means a tool call can't leave a caller listening to silence for several seconds. That's the piece Boardy is rebuilding now, and Owen pointed at duplex architectures — a fast front-end model that keeps talking while a slower back end goes and does the work — as the shape of the answer.

If you're weighing that same tradeoff, how you choose a speech-to-text API for voice agents matters more than it looks: every millisecond you spend on transcription is a millisecond you can't spend on retrieval.

What they want fixed in the next 12 months

Instead of predictions, Ryan asked what each of them would fix with a magic wand.

Dylan wants evals native to the platform. Right now Flagler runs a full post-call transcription through AssemblyAI on top of the live session, because the live stream only gives them one side of the conversation while the call is happening. The eval then runs on their own backend, decoupled from the call. What he wants: the full transcript and a suite of evals attached to the call record itself, in the platform, so re-evaluating a call is one request instead of a pipeline.

Owen wants the pipeline to stop being the ceiling. He's watching what people now expect from coding agents and assuming those expectations arrive in voice next. Two specific asks: speech-to-speech models that keep getting smarter, and the front-end/back-end split that lets an agent keep talking while it works. His view is that cascading stacks are inherently bottlenecked by transcribing to text first — and that the ceiling moves when that changes.

The thread that tied it together

Two teams, opposite users, near-identical pain. And when you line up the seven takeaways, almost none of them are about getting speech recognition, an LLM, and text-to-speech to talk to each other. That part works now.

What's left is everything around the pipeline. Whether the agent knows it's being addressed. Whether it knows what the caller actually heard. Whether the caller ID says the clinic's name. Whether you can tell a good call from a bad one after the fact. Whether the agent can remember you without stalling.

The one dependency running under all of it is the transcript — whether you assemble your own voice agent stack or run a managed one. Flagler can't collect an insurance number or run an eval without it. Boardy can't decide whether to speak, or search two years of conversation history, without it. Every layer above transcription inherits its errors — which is why the boring work of accuracy on real-world audio keeps turning out to be the interesting work.

Thanks to LiveKit for co-hosting, to Owen and Dylan for being unusually candid about what broke, and to everyone who stayed until 8pm to keep arguing about it. More NYC events are coming — follow us on LinkedIn to hear about the next one first.