How intelligent turn detection (endpointing) solves the biggest challenge in voice agent development

There's a specific moment that makes a voice agent feel broken, and it isn't a wrong answer.

It's the pause. You finish a sentence, and the agent just… sits there. One second. Two. You start to wonder if it heard you, so you say "hello?" — and of course that's the exact moment it starts talking, and now you're both talking, and the whole thing collapses into the conversational equivalent of two people trying to get through the same doorway.

Or the opposite failure, which is worse: you pause mid-sentence to think, and the agent barrels in over the top of you.

Both failures come from the same thing. The agent has to decide when you're done speaking, and that decision — endpointing, or turn detection — is one of the hardest problems in voice agents. It's also one of the least discussed, because it doesn't show up in a word error rate table. It shows up in whether people hang up.

Latency in voice agents

Before getting into turn detection specifically, it's worth being precise about what "latency" means in a streaming speech-to-text system, because there are three of them and they get conflated constantly.

Partial transcript latency is how quickly a model returns a partial transcript after someone starts speaking. This is the number most vendors put in their marketing, and it's the least important of the three for a voice agent.

Final transcript latency is how quickly a finalized transcript arrives after someone finishes speaking. This matters, because it's what your LLM actually acts on.

Endpointing latency is how quickly the model decides someone is done speaking. This is the one that determines whether your agent feels natural, and it's the one nobody advertises.

Here's why the third one dominates. If your endpointing decision takes 800ms, it doesn't matter that your final transcript arrives 200ms after that — your caller has already been sitting in silence for a second before your LLM has even seen the text. Endpointing latency sits at the front of the chain, and every millisecond spent there is a millisecond of dead air.

The trap is that you can't just make it smaller. Endpointing latency and endpointing accuracy are in direct tension: decide faster and you interrupt people mid-thought, decide slower and you feel unresponsive. Which is exactly why how you make the decision matters more than how fast you make it.

Three ways to decide someone stopped talking

Manual endpointing. The user presses a button or says a wake phrase to signal they're done. It's fast and unambiguous, and it's a terrible user experience that will kill adoption. Nobody wants a walkie-talkie. This is fine for a kiosk with a physical button and nowhere else.

Silence detection. Watch for N milliseconds of silence, then declare the turn over. Enormously better than manual, and it's still what most systems run on. The problem is that N is impossible to set correctly, because silence means completely different things in different moments. The pause between "my email is john" and "at gmail dot com" is not the end of a turn. The pause after "yes" is. Silence detection can't tell them apart, so you're forced to pick between an agent that interrupts and an agent that lags. Most teams pick lag, and their callers assume the line dropped.

Semantic endpointing. Instead of measuring silence, understand what was said. A model that knows "my account number is four seven" is an incomplete thought, and "that's all I needed, thanks" is a complete one, can stop guessing from proxies. It's more complex to build. It's also the only approach that gets out of the accuracy-versus-latency trap instead of just sliding along it.

That's the approach worth building on, and the interesting part is how much it's changed in the last year.

How intelligent turn detection works now

The first generation of semantic endpointing worked roughly like this: the model predicted an end-of-turn token alongside the transcript, you got a confidence score for it, and you set a threshold. Above the threshold, end the turn. Below it, keep waiting. It was a real improvement over pure silence detection, and it came with a set of knobs you were expected to tune.

Universal-3.5 Pro Realtime works differently, and the difference is worth understanding.

It doesn't endpoint on silence, and it doesn't reduce the decision to a single token score you threshold. When a speaker pauses, the model looks at the transcript — the punctuation it predicts for what was just said — and judges whether this is a finished thought or a breath. That decision lands at around 300ms.

The reason that matters is that the punctuation the model predicts depends on how a sentence was delivered, not just on its words. Say "my name is Sarah" as an answer and your pitch falls at the end. Say the same four words as the start of "my name is Sarah, and I'm calling about my bill" and your pitch stays level. Identical text, opposite meanings, and a silence timer cannot tell them apart. A model that predicts a period rather than a comma can.

Two silence thresholds still bound the behavior:

Both are set by the mode you pick, and both can be overridden on the connection or updated mid-stream. All of it runs on the base $0.45/hr streaming rate — turn detection isn't an add-on, and neither is agent context or keyterm prompting. The full breakdown is on the pricing page.

One more thing worth flagging if you're migrating: end_of_turn_confidence_threshold does not exist on Universal-3.5 Pro Realtime. It was a Universal-Streaming parameter, it's officially deprecated there, and it was never part of this model's API. If you have it in a config, delete it and set mode instead.

Modes replaced the knobs

This is the biggest practical change from the previous generation. You no longer tune raw thresholds as a matter of course. You pick a mode:

Setting it is one connection parameter on the v3 streaming endpoint — note that streaming still uses the singular speech_model, unlike our pre-recorded API:

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "mode": "balanced",  # min_latency | max_accuracy
}
API_ENDPOINT = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(CONNECTION_PARAMS)}"

Worth being clear about what the modes are trading, because the names invite a misreading. min_latency doesn't transcribe faster. It makes the model less patient — quicker to call a pause the end of a turn, and quicker to let a barge-in through. That's a great trade for an agent confirming a delivery window and a bad one for an agent taking a credit card number.

Which brings us to the pattern that matters most, and the one that quietly fixes the single most common turn-detection complaint.

Switch modes by call stage. People pause between the groups of a phone number, a credit card, or an address. Each of those pauses can read as a completed thought, and one entity gets split across three turns. So when your agent knows a value is coming — it just asked for the callback number — raise the floor for the duration of that answer, then put it back:

# Before capturing the entity
websocket.send('{"type": "UpdateConfiguration", "min_turn_silence": 1000}')

# After the answer, re-apply your mode to restore its defaults
websocket.send('{"type": "UpdateConfiguration", "mode": "balanced"}')

Changes take effect immediately, no reconnect. This is a much better tool than picking one conservative threshold and living with a sluggish agent for the whole call — you spend the patience only where it buys you something.

And if you run your own turn detection model and just want us out of the way, send a ForceEndpoint and the final transcript comes back immediately without waiting for silence:

websocket.send('{"type": "ForceEndpoint"}')

The other half: knowing when a turn starts

Turn detection gets discussed as an end-of-turn problem, but half of what makes an agent feel human is how it handles being interrupted.

Universal-3.5 Pro Realtime emits a SpeechStarted message once per turn, carrying a timestamp and a confidence, and — importantly — it only fires once the model produces an actual transcript. Background noise alone won't trigger it. That makes it a reliable barge-in signal: when it arrives, stop your agent's audio and start listening.

Two parameters shape how eagerly that fires. vad_threshold (default 0.2) is the confidence required to classify a frame as speech. interruption_delay is how long after speech begins the first partial is emitted — 0ms in min_latency, 500ms in the other two.

The failure modes are symmetric and both common. If quiet or distant speech is being missed, lower vad_threshold. If the agent keeps interrupting itself, it's picking up its own TTS through the mic — raise vad_threshold, raise interruption_delay, or turn on voice_focus to isolate the primary speaker before the audio reaches the model at all.

That last one is underrated. A lot of what gets diagnosed as bad turn detection is actually a second voice in the room — which is also why streaming speaker diarization is worth turning on for any agent that runs in a shared space.

Comparing turn detection approaches

Three approaches worth knowing, because they represent genuinely different architectural bets.

LiveKit: semantic-only

LiveKit's turn detection model analyzes transcribed text. It reads the words and predicts whether the utterance is complete. Because it works on text, it needs a VAD to tell it when to run — the VAD decides there's been a pause, then the model judges whether the pause is meaningful.

The strength is that it generalizes well across speakers; text doesn't care about your accent or your pitch. The weakness is that it inherits every flaw of the VAD in front of it, and it's blind to the acoustic cues that distinguish a falling "…Sarah" from a level one.

Pipecat: audio-centric

Pipecat's smart turn detection goes the other way — it reads audio features like prosody and intonation directly, no transcript required. That makes it fast, and it catches exactly the acoustic signals text-only models miss.

The tradeoff is that prosody varies enormously between speakers, and audio features degrade in noise. A model reading pitch contours has a harder time when the pitch contours are buried under a car engine.

AssemblyAI: Audio, via its own transcript

We do both, in one model, as one decision. The transcript and the punctuation predicted for it are one decision rather than one gating the other, which means no separate VAD in the path and no dependency on a transcript arriving first.

The practical payoff is robustness to the two things that break the single-modality approaches: background noise (where the transcript carries the decision) and speaker variation (where the words carry it when the prosody is unusual). And because we're already transcribing, there's no additional model in the pipeline to add latency.

The comparison matrix

That last row changed since we first published this comparison. It used to say "High" for us, and that was a fair criticism — the previous generation exposed raw confidence thresholds and silence timers, and getting them right took real experimentation. Three named modes replaced that, and mid-stream switching replaced the rest.

Which approach should you choose?

LiveKit if your audio is clean, your priority is semantic understanding of complete thoughts, and a little extra latency is acceptable.

Pipecat if your speakers are consistent, your audio is clean, and you specifically want to experiment with audio-only detection.

AssemblyAI if your callers are diverse, your audio is real-world, and you want the end-of-turn decision made inside the model rather than bolted on beside it. Also if you'd rather set a mode than tune a threshold.

If you're building on an orchestration platform — LiveKit Agents, Pipecat, Vapi — check what it does with our end-of-turn signal before you configure anything else, because the common failure is a platform-level timer sitting on top of a model that's already made a better decision. Layering a wait timer over semantic endpointing doesn't add safety; it adds latency and overrides the better signal. Our guide to building the lowest latency voice agent in Vapi walks through exactly that mistake.

And there's a third option that skips the question entirely: our Voice Agent API has semantic end-of-turn and semantic barge-in on by default, with adaptive pacing that adjusts to each speaker over the course of a call and automatic waiting when a tool parameter expects a phone number or an email. Flat $4.50/hr, one WebSocket, nothing to tune. The way you improve turn-taking there is by writing better tool descriptions, not by touching VAD knobs — how to build with the Voice Agent API walks through it.

What this buys you in production

Turn detection is one of those things that's invisible when it works and is the entire experience when it doesn't.

Siro, which runs conversation intelligence for field sales teams, saw a 90% reduction in customer complaints and support tickets after switching to AssemblyAI. That's not a turn-detection metric on its own — it's what happens when transcription accuracy and conversational feel improve together, because callers don't file tickets about word error rates, they file them about being cut off and not being understood.

LiveKit's co-founder David Zhao made a related point about how much of this is now happening inside the model rather than in your config:

We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms.

The same shift applies to turn detection. The trend line is that decisions which used to live in your configuration are moving into the model, where they have more information to work with — which is also, not coincidentally, where assembled voice agent stacks tend to hit their ceiling.

Where turn detection goes next

The obvious next step is more signals: conversational context beyond the current turn, speaker identity, and in multimodal settings, visual cues. If a model knows you're mid-way through a five-step form, it should be more patient than if you just answered a yes/no question. If it can see you're still looking at your phone reading out a confirmation code, it shouldn't jump in.

Some of that is already here. Passing agent_context — the question your agent just asked — cut word error rate by 10.2% across a benchmark of 20,000 voice agent audio files, and it's the same signal that lets the model stay patient while you read out a number. Context Carryover keeps a rolling memory of the conversation without you managing it. Neither is a turn-detection parameter, and both make turn detection better.

Conclusion

Here's the thing that took us a while to internalize: turn detection isn't really a latency problem, and treating it as one is why so many agents feel wrong.

It's a rhythm problem. Human conversation runs on a shared beat — the gap between turns in natural speech is remarkably consistent, around 200ms across languages and cultures, and we're all unconsciously calibrated to it. An agent that responds in 400ms every single time will feel more natural than one that averages 300ms but occasionally takes 1,200ms, because the variance is what breaks the beat. Your caller can adapt to a slightly slow agent. They can't adapt to an unpredictable one.

Which reframes what you should be optimizing. Not the mean. The variance. When you instrument your agent, log the distribution of your endpointing latency, not the average — and pay particular attention to the tail, because that tail is where the "hello? are you there?" moments live.

That's also the real argument for putting the decision inside the model rather than in a timer. A timer has perfectly consistent latency and inconsistent correctness, which is exactly backwards. You want the thing that's right most of the time and steady about it.