Why your voice agent mishears "8" as "H"

Here’s a bug report you will eventually receive, more or less verbatim:

“The agent is great on the conversational stuff. But when customers read out their confirmation number, it drops the last digit about a third of the time. Can we get a better model?”

The instinct is to go shopping. Swap the speech model, run a benchmark, pick whoever wins on word error rate. We watch teams do this constantly, and it usually doesn’t work — because the premise is wrong.

If your agent handles free-flowing conversation well and only falls apart on digit strings, postal codes, and account IDs, you almost certainly do not have a model problem. You have an endpointing problem. The acoustic model never got a fair shot at the last digit, because the turn was already over by the time it was spoken.

This post is about how to tell the difference, and what to change once you have.

The tell: good conversation, bad sequences

Start with the diagnostic, because it’s cheap and it’s decisive.

Pull a sample of failed turns and sort them into two buckets: turns where the customer was talking, and turns where the customer was spelling or reciting. Then compare error rates.

A genuine model weakness is roughly uniform. Accents, noise, and unusual vocabulary degrade both buckets together, because they degrade acoustics. If your model is bad at Scottish English, it’s bad at Scottish English whether the speaker is telling a story or reading a serial number.

An endpointing failure is lopsided, and dramatically so. This pattern shows up repeatedly in head-to-head evaluations: a model comfortably ahead of the incumbent on contextual and accented speech, and behind it on alphanumeric strings from the same sessions. Same model, same audio pipeline, same day. When the gap is shaped like that, it isn’t acoustic. Single digits are finalizing mid-sequence.

That lopsided shape is the signature. When you see it, stop benchmarking models and go look at your turn boundaries.

Why sequences break turns

Turn detection has to answer one question continuously: has this person finished talking?

The naive answer is silence duration. Wait some number of milliseconds, and if nothing arrives, close the turn. That works acceptably for sentences, because people speak sentences with a rhythm — the pauses inside a sentence are shorter than the pause at the end of one.

Now listen to someone read out a confirmation number.

“It’s A... as in apple... 4... 7... 7... 0... uh... 4... 1... 8... 2.”

Those inter-digit gaps are enormous. They’re often longer than the natural pause at the end of a conversational sentence, because the speaker is reading, or recalling, or checking a screen. A silence threshold tuned for conversation will fire somewhere in the middle of that string — and everything after the fire point either lands in a new turn your agent has already stopped listening for, or gets discarded entirely.

This is why the failure mode is so specific and so recognizable. You don’t get a garbled confirmation number. You get a truncated one. The first several characters are perfect. The tail is missing.

Turn detection on Universal-3.5 Pro Realtime improves on raw silence by not endpointing on silence alone. When a speaker pauses, the model evaluates what has been said so far and judges whether it reads as a finished thought or a mid-thought pause. That helps a lot. It does not fully solve recitation, for a reason worth sitting with: a partial entity frequently does read as complete. “Four, seven, seven, zero” is a perfectly well-formed thing to have said. Only your application knows it was expecting nine characters.

Which is the actual insight here: your agent knows something the speech model doesn’t. It knows it just asked for a confirmation number. That knowledge is the fix.

Fix 1: tell the model what it’s about to hear

The cheapest intervention is also the most effective, and most teams skip it.

Streaming speech-to-text on Universal-3.5 Pro Realtime accepts agent_context — your agent’s most recent spoken reply, the text your TTS just said. Pass it, and the model resolves ambiguous audio against what was actually asked. “Can I get your member ID?” primes the model for an alphanumeric string. “What’s your name?” primes it for a name.

Set it when you open the WebSocket, then update it after every agent turn:

# At connection time
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "agent_context": "Thanks for calling. How can I help you today?",
}

# After each TTS reply, before the caller responds
await websocket.send(json.dumps({
    "type": "UpdateConfiguration",
    "agent_context": "What's your confirmation number?",
}))

Each agent_context value is capped at 1,750 characters.

Across a benchmark of 10,000+ voice agent audio files, passing agent context cut word error rate by 8.9% — and by 16.4% with a context prompt on top. That combined configuration is where the published per-class breakdown comes from, and the breakdown matters more than the headline: per the Universal-3.5 Pro Realtime release, agent context plus a context prompt dropped place-name entities 30.7%, fabrications 27.0%, medical entities 26.2%, name entities 21.8%, and short-utterance errors 20.5%, with entity errors overall down 11.1%. Short utterances and entities are precisely the turns that break.

There’s a rolling conversation memory too — Context Carryover — and it’s on by default. It carries the last five turns forward so the model isn’t reasoning about each utterance in isolation.

“We’re excited to make AssemblyAI’s Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don’t, removing the need for users to predefine key terms.”

— David Zhao, Co-founder at LiveKit

Note what Context Carryover replaces: predefining key terms. You can’t enumerate every possible confirmation number in a keyterms list. You can tell the model you’re expecting one.

Fix 2: relax endpointing while the entity is in flight

Context helps the model interpret what it hears. It doesn’t stop the turn from closing early. For that you need to move the boundary itself — temporarily.

The pattern is straightforward: when your agent asks for an entity, raise the silence thresholds. When the entity is captured, restore your mode preset.

Two parameters control this. Neither has a fixed default — both are set by the mode preset you select:

min_turn_silence is how long the model waits before running an end-of-turn check at all. max_turn_silence is the hard fallback that force-ends a turn regardless of what the content check thinks.

# Agent has just asked: "What's your confirmation number?"
# Widen the silence window so inter-digit pauses don't close the turn.
await websocket.send(json.dumps({
    "type": "UpdateConfiguration",
    "min_turn_silence": 1000,   # ms - balanced preset uses 128
    "max_turn_silence": 2400,   # ms - balanced preset uses 1280
    "agent_context": "What's your confirmation number?",
}))

# ... capture the entity ...

# Restore the preset's defaults by re-applying mode, rather than
# guessing at numbers - the defaults are mode-derived, not fixed.
await websocket.send(json.dumps({
    "type": "UpdateConfiguration",
    "mode": "balanced",
}))

UpdateConfiguration is a delta: fields you omit keep their current values. That’s why re-sending mode is the correct way to snap back — hardcoding numbers on the way out leaves the session permanently off-preset.

Most agent frameworks already have the hook you need for this: tool calling. The agent knows which slot it’s filling, so let the slot definition carry its own endpointing profile. Entity slots get the relaxed config. Everything else gets the conversational one.

If your agent is confident the entity is complete — the customer said “that’s it,” or the string hit the expected length — you can close the turn immediately rather than waiting out the timer:

await websocket.send(json.dumps({"type": "ForceEndpoint"}))

The server returns the turn’s final Turn message right away, without waiting for silence. That reclaims most of the latency the relaxed window would otherwise cost you.

About that latency cost

Be honest with yourself here, because this is a real tradeoff and pretending otherwise leads to bad configs.

Raising min_turn_silence to a full second measurably improves entity capture. It also adds close to a second of dead air before the model will even consider ending the turn — and if the content check keeps reading the turn as incomplete, max_turn_silence is what finally closes it, so the worst case with the config above is around 2.4 seconds. On a conversational turn that’s awful. On a confirmation-number turn it’s fine, because the customer just did something effortful and expects a beat. ForceEndpoint is how you get the tail of that window back when you know you’re done.

That asymmetry is the entire argument for dynamic configuration over a single global setting. Teams that set one relaxed threshold for the whole call fix their entity accuracy and break their conversational feel. Teams that set one tight threshold keep the feel and keep the bug. You don’t have to choose.

Fix 3: swap your keyterms as the conversation moves

There’s a second mid-stream lever most teams leave on the table, and it addresses the half of the problem agent_context doesn’t.

The docs are blunt about this: replacing keyterms_prompt mid-stream is the most effective way to improve recognition accuracy during a live session. The trick is that you’re not trying to enumerate every value — you’re narrowing the model’s search space for the stage of the flow you’re currently in.

# Caller-identification stage
await websocket.send(json.dumps({
    "type": "UpdateConfiguration",
    "keyterms_prompt": ["date of birth", "January", "February", "policy number"],
}))

This does nothing for an arbitrary nine-character confirmation code — nothing can enumerate those. But the same flow that asks for a confirmation number usually also asks for a surname, a city, a plan name, or a product SKU, and those are enumerable. Swapping the list per stage costs one message and is included in the base rate.

Streaming keyterms are capped at 100 terms of 50 characters each, so a per-stage list is the right granularity anyway — you couldn’t load your whole customer table if you wanted to.

Fix 4: stop counting the wrong thing

None of the above will stick unless you change what you measure, because pooled word error rate actively hides this failure.

Run the numbers on a transcript where the agent captured “RX-7704182” as “RX-770418.” One token wrong out of a forty-token turn. That’s 2.5% WER — a number most teams would celebrate. It’s also a completely failed transaction. The prescription doesn’t exist, the lookup fails, and the customer gets transferred.

Entity error rate is the metric that tracks what you actually care about. Here’s where Universal-3.5 Pro Realtime lands on Pipecat’s open STT benchmark, run on real agent conversations:

Look at the spread between the WER row and the entity row. Deepgram Flux is roughly 2x worse on words and roughly 3.3x worse on entities. If you had picked a model on WER alone, you’d have understated the gap on the thing that breaks calls.

Our own evaluation breaks this down further by entity class — alphanumerics, addresses, emails and names are each scored separately, with current figures on the benchmarks hub. Those are a different dataset and a different methodology from the Pipecat numbers above, so read them side by side rather than as one series. Track your own per-entity rates the same way. One number per entity type, measured on your production audio, is worth more than any vendor benchmark including this one.

A checklist you can run this afternoon

1. Split your failures into conversational turns and recitation turns. Compare error rates. Lopsided means endpointing.
2. Inspect the truncation pattern. Missing tails, not scrambled middles, confirms it.
3. Pass agent_context after every agent turn. It’s included in the base rate, and it’s the highest-leverage change available.
4. Make endpointing dynamic — raise min_turn_silence on entity slots via tool calling, re-send mode to restore the preset afterward, and ForceEndpoint when you know you’re done.
5. Check your vad_threshold. It defaults to 0.2. Raise it if background noise or your own TTS audio is triggering false interruptions; lower it if quiet or distant speech is being missed entirely.
6. Consider voice_focus if there’s background speech. It isolates the primary voice and suppresses background audio before it reaches the model — near-field for headsets and phones, far-field for rooms, kiosks, and drive-thrus. Note the hyphens.
7. Watch out for speaker_labels. Enabling it swaps in diarization-tuned defaults (min_turn_silence 640, max_turn_silence 768, continuous partials off), which changes your endpointing behavior underneath you.
8. Measure entity error rate per entity type. Retire pooled WER as your primary accuracy metric for agent work.

The part nobody tells you

The reason this bug survives so long in production is that it presents as a model problem and gets treated as a procurement problem.

A team notices digit capture is bad. They run an evaluation. The evaluation uses isolated audio clips, because that’s what’s easy to assemble — and isolated clips have no conversational context and no meaningful turn boundaries, so the endpointing behavior that caused the original bug is absent from the test entirely. The evaluation measures acoustics. The bug was configuration. Everyone concludes the model is worse, switches vendors, and ships the same bug on new infrastructure.

If you take one thing from this: evaluate voice agents end-to-end, on conversations, with your real turn-detection config in the loop. A clip-based benchmark will tell you something true about acoustic accuracy and nothing at all about whether your agent will capture a confirmation number.

That’s a harder test to build. It’s also the only one that predicts what your customers will experience.

Frequently asked questions

Why does my voice agent mishear numbers and letters?

Most often because the turn is being finalized in the middle of the sequence, not because the speech model misheard the audio. People pause much longer between spoken digits than between words in a sentence, and a partial entity often reads as a complete thought on its own, so the turn closes mid-string and the remaining characters are lost. The giveaway is that the beginning of the sequence is transcribed correctly and only the tail is missing.

How do I improve confirmation number accuracy in speech to text?

Pass your agent’s spoken question to the model as agent_context so it knows an alphanumeric string is coming, then raise min_turn_silence for the duration of that turn and re-send your mode preset once the entity is captured. On Universal-3.5 Pro Realtime, passing agent context cut word error rate by 8.9% across a benchmark of 10,000+ voice agent audio files, rising to 16.4% when combined with a context prompt. Combining context with dynamic endpointing addresses both halves of the problem.

What is entity error rate, and why isn’t word error rate enough?

Entity error rate measures how often a model gets critical values wrong — names, phone numbers, addresses, account IDs — rather than how often it gets any word wrong. A transcript can post an excellent 2.5% word error rate while corrupting the one confirmation number that determines whether the call succeeds. Universal-3.5 Pro Realtime records a 15.31% entity error rate against 50.50% for Deepgram Flux on Pipecat’s open benchmark, a much wider gap than the word error rate comparison suggests.

Should I just turn off turn detection entirely?

You can’t switch it off, but you can override it. Sending a ForceEndpoint message ends the current turn immediately, which lets teams with an existing VAD or push-to-talk stack drive boundaries themselves — max_turn_silence still acts as a backstop underneath. That moves the problem rather than removing it, since something still has to decide when the customer finished speaking, and dynamic configuration usually gets you the same entity accuracy without rebuilding turn detection from scratch.

Does relaxing endpointing hurt latency?

Yes, and you should account for it rather than absorb it globally. Raising min_turn_silence to around one second adds close to a second before the model will even check whether the turn ended, with max_turn_silence setting the worst case above that. It’s acceptable after someone recites an account number and noticeably sluggish in ordinary conversation. Applying the relaxed profile only to entity-capture turns — and calling ForceEndpoint once you have the value — keeps conversational latency at the balanced preset’s 128ms check interval.

How do I test this properly before switching providers?

Evaluate on full conversations with your production turn-detection configuration in the loop, not on isolated audio clips. Clip-based evaluations strip out the conversational context and turn boundaries that cause the failure in the first place, so they measure acoustic accuracy and miss configuration problems entirely. Track entity error rate per entity type on your own audio alongside any vendor benchmark.