Turn detection vs forced endpoints in voice AI: Why getting this wrong tanks your UX

Turn detection is how a voice agent decides the caller has finished speaking so it can respond. Three approaches exist: voice activity detection watches the audio signal for silence, a forced endpoint runs a timer, and semantic end-of-turn detection reads the speech itself. Universal-3.5 Pro Realtime uses the third—it reads tonality, pacing, and rhythm rather than silence or punctuation alone, and lands the decision in roughly 300ms.

Which is why it doesn't clip a caller who pauses to think. Get this wrong one way and your agent talks over people mid-sentence; wrong the other way and every exchange carries a second of dead air. It's the single trigger for the entire response pipeline, so it does more damage to perceived quality than almost anything else in the stack.

Does AssemblyAI still support hybrid turn detection?

Yes. This needs saying plainly, because the older version of this post left the opposite impression.

End-of-turn detection on AssemblyAI streaming is not a single silence timer and never was a punctuation-only heuristic in the current generation. It is a hybrid: voice activity detection, punctuation signals, and a dedicated turn-decision model that weighs how the caller is speaking, not just whether they have stopped. All three run together.

On top of that:

• Forced endpoints are still first-class. You can cut a turn on your own signal—a push-to-talk release, an orchestrator timer, your own VAD—and the model still processes the buffered audio behind that boundary.
• Mid-entity endpointing is on by default. Balanced-mode endpointing holds through numbers and proper nouns, including comma-separated and dotted spellings, so "555, 0123" or "j.smith@acme.io" no longer gets split into two turns.
• You can bring your own end-of-turn model. Plenty of teams run a tuned open-source detector, a framework-native detector, or something they trained on their own call recordings. That is a supported architecture, not a workaround, and there's a section on it below. It's also one of the places voice agent stacks hit their limits in production.

So if you read an older AssemblyAI post and came away thinking the hybrid approach had been retired in favor of a black-box detector—it hasn't. The detector got better and the tuning surface got simpler. Those are different things.

What is turn detection in voice agents?

Turn detection runs continuously while a caller speaks, answering one question over and over: is this person done?

In human conversation we answer it with everything at once—falling intonation, the shape of a sentence, how fast someone is talking, whether the last word landed like a conclusion or a comma. We're good at it. We interrupt each other constantly and it mostly works, because both parties are reading the same signals.

A voice agent has to reconstruct that from an audio stream, in real time, with no ability to go back and revise. And the cost of a wrong answer is asymmetric: a false positive clips the caller, a false negative makes the agent feel slow. Most teams tune for one and then discover they've made the other worse.

The mechanism matters here. Universal-3.5 Pro Realtime reads tonality, pacing, and rhythm to make the call, at roughly 300ms. It is not waiting out a silence window and it is not scanning for a period.

VAD, forced endpoints, and semantic endpointing: what's the difference?

These three get used interchangeably in conversation and they are not the same thing. Here's the honest version of each, including where each one breaks.

How semantic turn detection works

The model is listening to delivery, not just content. A sentence that trails off with rising intonation reads as unfinished even if the words form a complete clause. A clipped, falling "yep" reads as finished even though it's one syllable and lasted 200 milliseconds.

But turn detection isn't only about when the caller stopped. It's also about what they said—and short, low-information utterances are exactly where transcription and endpointing both get shaky. Three things help, and all three are worth turning on before you touch any timing config:

• agent_context. Pass your agent's own last question into the stream. Across a benchmark of 20,000 voice agent audio files, passing agent context cut word error rate by 10.2%, with short-utterance errors down 13.7% and fabrications down 18.3%. If your agent just asked "what's the last four of your card?", the model knows a four-digit string is coming.
• Context Carryover. A short rolling conversation memory, on by default. You don't configure it and you don't pay extra for it.
• voice_focus. Isolates the primary speaker and suppresses background speech—near-field for headsets and phones, far-field for rooms, kiosks, and drive-thrus. Background speakers are one of the most common causes of a turn firing when nobody actually finished talking.

The mid-entity behavior is worth calling out on its own. Endpointing now holds through numbers and proper nouns, including comma-separated and dotted spellings. Phone numbers, email addresses, and account IDs are the classic split case—a caller says "five five five" and takes a breath, and a silence-based detector fires. Holding the turn through the whole entity is what stops that. The trade-off is honest: holding through an entity adds some p90 latency on those specific turns. It's the right trade for most agents and the wrong one for a few.

When forced endpoints still make sense

Semantic detection is the default for a reason, but there are four cases where you should override it:

Push-to-talk. The button release already tells you the turn is over. Running a detector on top of that adds latency for zero information.

Compliance flows. When a caller has to hear a disclosure through to the end, or read something back in full, you want a guaranteed boundary rather than a probabilistic one.

Structured data collection. Field-by-field capture—date of birth, then policy number, then ZIP—benefits from explicit boundaries between items rather than one long inferred turn.

Very high noise, very short turns. In some drive-thru and factory-floor deployments a hard ceiling behaves more predictably than any detector. Try voice_focus: far-field first, but if it doesn't hold, a forced endpoint is a legitimate answer.

And forced endpoints have gotten materially faster. If you evaluated that path a while back and rejected it on latency, the evaluation is worth re-running before you rule it out.

What happens when turn detection gets it wrong?

Both failure modes are obvious to a user and invisible in aggregate metrics. Your word error rate can look fine while your agent is unusable.

And they are not edge cases. In AssemblyAI's 2026 Voice Agent Report, a survey of 455 practitioners fielded across Q4 2025 and Q1 2026, 47.5% reported that their voice agent interrupts users mid-sentence, and 55% named "having to repeat themselves" as the single biggest end-user frustration. Both of those are turn detection failing, described from the outside.

Cutting users off mid-sentence

The classic case is an entity split. A caller reads a phone number in the natural grouping—"five five five", pause, "oh one two three"—and a silence-based detector fires after the first group. Your agent now has "555" and starts responding to a partial. Email addresses are worse, because people spell them slowly and pause at the dot.

The second case is the thinking pause. "My account number is..." followed by two seconds of the caller finding their card. Every timer-based approach clips this, and no amount of threshold tuning fixes it, because the timer cannot tell a thinking pause from a finished thought.

Both of these are what mid-entity endpointing and semantic detection are for. If you're still seeing splits after moving to streaming speech-to-text on the current model, the next lever is agent_context—tell the model what kind of answer it's waiting for.

Awkward silence and response delays

The opposite failure is quieter and just as damaging. The caller finishes, and nothing happens. Half a second passes. They start to repeat themselves—and now the agent responds, on top of them.

Users read that delay as the product being slow, not as a tuning artifact. It's the same instinct that makes a 300ms web page feel snappy and a 900ms one feel broken. Perceived responsiveness is a product-quality signal, and it shows up in how people talk about the product.

Jonathan Kim, Software Engineer at Granola, put it this way:

"The speed difference is immediately noticeable — our users see their conversations transcribed almost instantaneously. It feels so much more responsive than what we were using before."

If your agent feels slow after the caller stops talking, the fix is usually not a shorter timer. It's a detector that recognizes a finished thought instead of counting silence.

How do you tune turn detection for your voice agent?

This is the part that changed most. If you learned this API a year ago, the mental model you have is wrong—not incorrect in detail, but pointed at a surface that no longer leads.

Pick a mode, not a threshold

Start with the mode preset instead of the individual timing flags. It's the top-level dial for the accuracy/latency trade-off, and it sets sensible defaults for everything underneath it:

• min_latency—fastest response. Use it for short, structured, predictable exchanges where callers give one- or two-word answers.
• balanced—the default, and the right answer for most agents. Mid-entity endpointing is on here.
• max_accuracy—for noisy rooms, distant microphones, hesitant speakers, and anything where a false interruption costs more than a few hundred milliseconds.

The fine-grained levers—min_turn_silence, max_turn_silence, vad_threshold, interruption_delay—are still there, and anything you set explicitly wins over the preset. But they're overrides now, not the starting point. That ordering matters: most teams who spent a week tuning thresholds landed somewhere close to what balanced does out of the box.

One flag to stop reaching for: end_of_turn_confidence_threshold has no effect on the U3 Pro family. It applies to the older universal-streaming models. If you're carrying it over from a previous integration, it's dead config.

Read the end-of-turn confidence value

Rather than setting a confidence threshold and hoping, the model emits end_of_turn_confidence on the Turn message itself. That's a better shape for a decision your orchestrator, not the STT layer, should own.

What to do with it: use it to gate the expensive part of your pipeline. On a high-confidence turn, fire the LLM immediately. On a marginal one, you have options your old threshold didn't give you—hold for a short grace window, start the LLM speculatively and cancel if more speech arrives, or fall back to a forced endpoint. The point is that the ambiguity is now visible instead of resolved for you behind a threshold you had to guess at.

Connecting on the current model

Here's the shape of a streaming connection with the pieces that matter for turn detection. This targets the AssemblyAI Python SDK 1.0.0 (released August 14, 2026), which unified the async, streaming, and sync clients behind one client shape.

Python:

import os
import pyaudio
from assemblyai.streaming.v3 import (
    BeginEvent,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    TerminationEvent,
    TurnEvent,
)

# The SDK doesn't capture audio -- supply 16-bit PCM chunks yourself.
def microphone_stream(sample_rate=16000, frames_per_buffer=800):
    audio = pyaudio.PyAudio()
    stream = audio.open(input=True, format=pyaudio.paInt16, channels=1,
                        rate=sample_rate, frames_per_buffer=frames_per_buffer)
    try:
        while True:
            yield stream.read(frames_per_buffer, exception_on_overflow=False)
    finally:
        stream.stop_stream(); stream.close(); audio.terminate()

def handle_user_turn(transcript: str, confidence: float):
    """Your orchestrator: fire the LLM, or hold if confidence is marginal."""
    print(transcript, confidence)

def on_begin(client: RealTimeTranscriber, event: BeginEvent):
    print(f"Session started: {event.id}")

def on_turn(client: RealTimeTranscriber, event: TurnEvent):
    if not event.end_of_turn:
        # Partial. Update the UI with it -- don't trigger the agent.
        return
    # The caller finished. end_of_turn_confidence tells you how sure
    # the model is, so your orchestrator can decide what to do about it.
    handle_user_turn(event.transcript, event.end_of_turn_confidence)

def on_terminated(client: RealTimeTranscriber, event: TerminationEvent):
    print(f"Session terminated: {event.audio_duration_seconds}s of audio")

client = RealTimeTranscriber(
    RealTimeTranscriberOptions(terminate_timeout=30.0),
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
)

client.on(RealTimeEvents.Begin, on_begin)
client.on(RealTimeEvents.Turn, on_turn)
client.on(RealTimeEvents.Termination, on_terminated)

client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
        mode="balanced",            # min_latency | balanced | max_accuracy
        voice_focus="near-field",   # near-field | far-field
        agent_context="What are the last four digits of your card?",
    )
)

try:
    for chunk in microphone_stream():
        client.stream(chunk)
finally:
    # Always terminate -- streaming bills on session duration.
    client.disconnect(terminate=True)

Three things to notice. agent_context is set at connect here to seed the first turn, but in a real agent it gets updated every time your agent finishes speaking—that's what client.update_configuration(agent_context=...) is for. Streaming uses the singular speech_model parameter, while the async API takes a speech_models array; that asymmetry trips people up constantly. And there's no format_turns here, because the U3 Pro family always returns formatted transcripts.

Not using an SDK? Connect straight to wss://streaming.assemblyai.com/v3/ws with your key in the Authorization header, no Bearer prefix, and manage the Begin / Turn / Termination messages yourself.

Running it inside Pipecat

Most production agents don't hand-roll the socket. On Pipecat the STT service owns the connection, and one flag decides the whole architecture:

Python:

from pipecat.services.assemblyai.stt import AssemblyAISTTService

stt = AssemblyAISTTService(
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        mode="balanced",             # min_latency | balanced | max_accuracy
        voice_focus="far-field",     # "near-field" for close-talking mics
        min_turn_silence=100,
        max_turn_silence=1000,       # respected in AssemblyAI turn-detection mode
    ),
    # False -> AssemblyAI's built-in turn detection owns the boundary.
    # True (the plugin default) -> Pipecat's VAD + Smart Turn own it and
    # send ForceEndpoint on silence. Both are supported; pick deliberately.
    vad_force_turn_endpoint=False,
)

That flag is the architectural question this whole post is about: who owns the turn decision, the STT layer or the orchestrator? Both are legitimate. Pipecat mode gives the framework's Smart Turn analyzer the call and is the plugin default; AssemblyAI mode uses the model's own detector and emits SpeechStarted for fast barge-in. What you should not do is leave it on the default without knowing which one you picked. Note that in Pipecat mode max_turn_silence is auto-synced to min_turn_silence, so raising the floor is how you widen the window for dictated entities. If you're still choosing a framework, we compared the major voice agent orchestrators separately.

On LiveKit the equivalent is turn_detection="stt" in TurnHandlingOptions, which hands the decision to AssemblyAI, versus LiveKit's own MultilingualModel(). Both work on livekit-agents 1.6+. One thing to skip: don't pass a context-carryover parameter by hand. From 1.6.6 the session forwards each assistant reply into agent_context automatically, and manual values get overwritten on the next reply.

How does AssemblyAI compare on turn-taking accuracy?

Turn detection accuracy is hard to benchmark on its own, because a false turn and a transcription error look the same downstream. What you can measure is how a model does on real agent conversations, and specifically on the tokens that break an agent when they're wrong.

On the Pipecat open STT benchmark of real agent conversations:

The entity row is the one that matters for turn detection. An entity error rate three times higher than ours mostly isn't the model mishearing a digit—it's the model ending the turn in the middle of one. That's the same defect from the other side.

And if you'd rather not assemble the latency budget yourself, the Voice Agent API runs the whole loop over one WebSocket at roughly 1 second end to end.

Can you bring your own end-of-turn model?

Yes, and a meaningful share of voice-agent teams do.

Some run a tuned open-source detector trained on their own call recordings. Some use their orchestration framework's native detector. Some run a small classifier over the transcript stream and make the decision themselves. All of these are compatible with AssemblyAI streaming, because the transcript stream and the turn signal are separable—you can take our transcripts and ignore our turn decision, or take both. The ForceEndpoint message exists precisely so an external detector can drive the boundary.

Provider-portable end-of-turn behavior is a legitimate requirement. If you're building on voice agent infrastructure you expect to run for years, not wanting your turn-taking logic welded to one vendor's detector is a reasonable position, and we'd rather you build it that way than pretend the constraint doesn't exist. AssemblyAI is also the default speech-to-text in LiveKit's starter voice agents, which is a decent signal of how portable the integration actually is in practice.

The recommendation is still to start with the built-in detector, because it's had more real agent audio through it than anything you'll train in a quarter. But "start with" is not "locked into."

Final words

Here's the thing nobody tells you about turn detection: the reason it's hard isn't that the signal is subtle. It's that the correct answer depends on what your agent is about to do with it.

An agent that's about to say "got it, one moment" can afford to be aggressive—the cost of a false turn is a filler phrase. An agent that's about to charge a card, book an appointment, or transfer a call cannot, because a false turn there means acting on half a sentence. Same audio, same model, different right answer.

Which means the most useful thing the current generation gives you isn't the improved detector, though that's real. It's that the confidence is now visible and the settings are per-session. You can run min_latency on the chatty parts of a flow and widen the window during entity dictation, in the same call. Turn detection stops being a global setting you argue about once and becomes something your agent reasons about in the moment—which is what it always should have been. For a deeper look at how this fits the wider pipeline, see our guide to real-time speech-to-text for voice agents, or the endpointing deep dive for the mechanics. And if you're still deciding which model to build on, we wrote up how to choose a speech-to-text API for voice agents.

Frequently asked questions

Does AssemblyAI still support a hybrid turn-detection approach?

Yes. End-of-turn detection combines voice activity detection, punctuation signals, and a dedicated turn-decision model rather than relying on silence alone. Forced endpoints remain fully supported for push-to-talk, compliance flows, and structured data capture, and customers can run their own end-of-turn model against the transcript stream using ForceEndpoint. Mid-entity endpointing, which holds turns through numbers and proper nouns, is enabled by default.

What is turn detection in voice agents?

Turn detection is how a voice agent decides the caller has finished speaking so it can start responding. It is the trigger for the entire response pipeline, so a wrong call either clips the caller mid-sentence or leaves dead air. Universal-3.5 Pro Realtime makes that call from tonality, pacing, and rhythm rather than silence alone, in roughly 300ms.

What is VAD in speech recognition?

Voice activity detection (VAD) is a lightweight check on whether the audio signal currently contains speech or silence. It is fast and cheap, but it only hears energy, not meaning, so it cuts a caller who pauses to think and triggers on background speech. Semantic end-of-turn detection reads the speech itself instead.

What is the difference between VAD and semantic turn detection?

VAD decides from silence in the audio signal; semantic turn detection decides from the content and delivery of the speech. VAD is faster but wrong more often on hesitant or long answers. Universal-3.5 Pro Realtime uses semantic end-of-turn detection reading tonality, pacing, and rhythm, at about 300ms.

How do I fix a voice agent that cuts users off mid-sentence?

Stop tuning silence thresholds first and switch to semantic end-of-turn detection. On Universal-3.5 Pro Realtime, choose the balanced or max_accuracy mode rather than min_latency, and pass agent_context so the model knows what kind of answer it is waiting for—that alone cut short-utterance errors 13.7% across 20,000 voice agent files. Mid-entity endpointing, on by default, handles the phone-number and email splits specifically.

Can I combine VAD forced endpoints with automatic model detection?

Yes, and in orchestration frameworks like Pipecat that is common: automatic detection handles normal turns while a forced endpoint acts as a backstop for audio the model never resolves. Set the forced endpoint well above your normal turn latency so it only fires as a safety net.

‍