Entity accuracy in speech-to-text: why word accuracy isn't enough

A voice agent transcribes a 90-second call perfectly. Every "um," every "you know," every filler word lands exactly right. Then it renders the caller's name as "John Smith" instead of "Jon Schmidt" and routes the account lookup to the wrong record.

That transcript has a great word error rate. It's also useless.

This is the gap nobody talks about when they benchmark speech-to-text. We've trained the whole industry to lead with WER, and WER rewards models for getting the easy, high-frequency words right. But the words that actually carry the meaning of a conversation, the names and numbers and medical terms that a workflow depends on, are rare. Miss one and the aggregate score barely moves. Miss one in production and the whole thing breaks.

So let's talk about entity accuracy, why it's the metric that actually predicts whether your voice agent works, and how to measure it honestly. If you want the broader context, we wrote a full breakdown of what "accurate" even means in speech-to-text that this post builds on.

Word accuracy vs. entity accuracy

Word error rate is simple. Take the reference transcript, take the model's output, count the substitutions, insertions, and deletions, divide by the total number of words. A WER of 7% means roughly 7 words out of 100 are wrong.

Here's the problem. WER treats every word the same. "The" and "amoxicillin" count equally. But in a real conversation those two words are not equally important, and they're not equally hard. Function words are everywhere and easy to predict from context. Entities are rare, and the model has almost no context to fall back on. That's exactly why they're hard, and exactly why they matter.

The thing is, a strong overall WER can hide a catastrophic entity failure. If a model nails 99 common words and botches the one phone number in the sentence, its WER looks fantastic and its transcript is worthless. We've argued before that word error rate as a single number is fundamentally broken for exactly this reason.

So how do you measure the thing that matters? You measure entities directly.

For production voice agents, the actual words that matter most are entities: names, organizations, emails, phone numbers, and medical terms. The Missed Entity Rate (MER) measures how often a model fails to correctly transcribe these high-stakes terms. It's the same idea as WER, scoped to the words your workflow can't afford to lose.

How to measure entity accuracy

Start by deciding what counts as an entity for your use case, then measure error rate within each category. The categories that show up over and over in production:

• Names — first names, last names, unusual spellings. The single hardest category.
• Places — cities, addresses, venue names.
• Phone numbers — long alphanumeric strings where one wrong digit fails the task.
• Emails — spelled-out addresses, domains, symbols.
• Alphanumerics — account numbers, confirmation codes, order IDs.
• Medical terms — drug names, dosages, procedures, anatomy.

The metric is straightforward. For each category, take the entities in the reference, check how many the model transcribed correctly, and report the error rate. Lower is better. If you want a rigorous walkthrough of the setup, we published a guide on how to evaluate speech recognition models that covers reference construction and scoring.

One thing to insist on: measure on real data. Read speech and clean studio audio will flatter every model. Measure on the messy, accented, cross-talk-heavy audio your users actually produce, because that's where entity accuracy falls apart.

The benchmark picture

The Pipecat open STT benchmark is a good one to reach for, because it runs on real agent conversations rather than curated read speech. Lower is better across the board. Here's how the entity error rate breaks down against other providers, alongside overall WER:

Look at the spread. On word error rate, Scribe v2 (9.76%) and Chirp3 (9.04%) aren't far off each other. But on entity error rate they're 18 points apart. WER alone would have told you those two models were roughly comparable. On the metric that decides whether your agent routes the right account, they're not close.

Broken out by category, Universal-3.5 Pro Realtime lands at 16.92% on names, 6.28% on places, and 3.55% on phone numbers. Names stay the hardest category for everyone, which is exactly what you'd expect and exactly why you should weight it most in your own evaluation. You can dig into the full methodology and numbers on our benchmarks page.

How to improve entity accuracy

Picking a strong base model is step one. But the real gains come from giving the model context, because entities are hard precisely when the model is guessing blind. Feed it hints and the numbers move fast.

Keyterms prompting. Give the model a list of the terms you expect: product names, drug names, the customer's own vocabulary. On Universal-3.5 Pro async you can pass up to 1,000 terms (a $0.05/hr add-on). On Universal-3 Pro Streaming, keyterms are free, up to 100, and you can update them mid-stream as a call evolves. On Universal-3.5 Pro Realtime, keyterm prompting is included.

Contextual prompting. Pass what you already know about the conversation. Metaview does this with meeting context, and their co-founder and CTO Shahriar Tajbakhsh put it well: "Since moving to AssemblyAI, we've seen a meaningful improvement in the confidence tail of our production transcripts....What stands out is not just the model quality, but the way [they] let us bring real meeting context into transcription, from calendar titles to organizations, domains, and participant names, so recruiting conversations come through with the nuance our customers depend on." Our guide to prompt engineering with Universal-3 Pro covers the mechanics.

Agent context. This one's underrated. With Universal-3.5 Pro Realtime you can pass the agent's own question as context for the user's reply. Across 20,000 voice agent files, doing that cut WER by 10.2%. Name entities improved 9.4%, place-name entities 15.5%, medical entities 9.4%, and errors on short utterances dropped 13.7%. It's included, not an add-on.

Stacking these works. In an independent Coval run, keyterms plus contextual prompting with min_latency mode drove WER down to 1.21%.

Custom vocabulary rounds it out for domain-specific jargon that shows up consistently. On its own, Universal-3 Pro already carries best-in-market entity accuracy across credit cards, phone numbers, emails, addresses, medication names, proper nouns, and rare words, with a hallucination rate roughly 30% lower than Whisper. The prompting layers build on top of that. The speech-to-text product page has the full feature set.

High-stakes entities: medical terms

Medical is where entity accuracy stops being a nice-to-have. A missed drug name or a wrong dosage isn't a transcription error, it's a patient safety and liability problem. And the cost of getting it wrong is real, which we broke down in the true cost of inaccurate transcription.

Medical Mode is built for this. You activate it with the parameter domain: "medical-v1", it's a $0.15/hr add-on, and it supports English, Spanish, German, and French. The numbers:

• 3.2% Missed Entity Rate on medical entities, the lowest MER across benchmarked providers including Deepgram, Speechmatics, AWS, and Google.
• ~20% fewer missed medical entities versus the base Universal-3 Pro model.
• 87% fewer entity errors versus the base model.

Contextual prompting compounds it. Feeding a patient's prior-visit note into the model cut missed medical terms by 31% in an internal healthcare test. Commure, which runs ambient clinical documentation through Athelas, has felt the difference. Their tech lead Gautam Pradeep: "We've integrated the newest models from AssemblyAI for pre-recorded audio ASR in our ambient product, and it's been excellent. We're now exploring Universal-3.5 Pro for async and realtime speech-to-text capabilities for new use cases. What's been just as important is the reliability of the platform itself—both technically and in terms of partnership."

For teams processing PHI, AssemblyAI signs a Business Associate Addendum (BAA), offers PHI redaction across audio and transcripts, and maintains SOC 2 Type 2. Details on the medical solutions page and our BAA FAQ.

One honest caveat. Keyterms and prompting are powerful, but they're not free. Load the model with too many biased terms and you can drag down general accuracy, because you're pushing it toward those terms even when they weren't spoken. Bias toward the entities you genuinely expect. Don't dump your entire product catalog in and hope.

An honest note on limits

No model is magic, and anyone who tells you otherwise is selling something.

Two failure modes worth knowing. First, models can hallucinate on entities that were spoken incorrectly in the first place. If a caller fumbles their own account number and self-corrects, the model may confidently render something that was never cleanly said. Second, very short utterances (roughly 3 to 10 seconds) are genuinely hard, because there's almost no acoustic or linguistic context to anchor on. This is exactly where agent context helps most, and it's why that 13.7% short-utterance improvement matters.

The point isn't that entity accuracy is unsolvable. It's that you should evaluate with eyes open, on your own audio, and design your workflow to confirm the highest-stakes entities rather than assuming any model gets them right 100% of the time.

Conclusion

Word error rate had a good run. It's a fine sanity check and a terrible north star. As voice agents move into medical scribing, PII redaction, and account-level automation, the failures that matter aren't distributed evenly across words, they're concentrated in the handful of entities each conversation turns on.

Siro sees it in the field: "On 10 out of 10 onboarding calls, our customers are at some point telling us 'wow that insight was crisp' — and that's because of the accuracy we're getting from AssemblyAI." Crisp is an entity-accuracy property, not a WER one.

So stop leading with WER. Measure Missed Entity Rate on the categories your workflow depends on, benchmark on real conversations, and stack the context tools that move the number. That's the metric production teams should evaluate on.

Frequently asked questions

What is entity accuracy in speech-to-text?

Entity accuracy measures how reliably a speech-to-text model transcribes high-stakes terms, the names, organizations, emails, phone numbers, account numbers, and medical terms that a workflow actually depends on. Unlike overall word accuracy, it focuses only on the rare, meaning-carrying words where a single error breaks the task.

How is entity accuracy different from word error rate?

Word error rate (WER) counts every wrong word equally across a whole transcript, so common words dominate the score. Entity accuracy scopes the measurement to entities only. A model can post a strong WER while still botching the one phone number or name that mattered, which is why WER alone can hide the errors that break production systems.

What is Missed Entity Rate (MER)?

Missed Entity Rate measures how often a model fails to correctly transcribe high-stakes terms like names, organizations, emails, phone numbers, and medical terms. Lower is better. AssemblyAI's Medical Mode posts a 3.2% MER on medical entities, the lowest across benchmarked providers including Deepgram, Speechmatics, AWS, and Google.

How can I improve entity accuracy in transcription?

Start with a strong base model, then give it context. Keyterms prompting supplies the terms you expect, contextual prompting passes what you already know about the conversation, and agent context feeds the agent's question alongside the user's reply. In one benchmark across 20,000 voice agent files, passing agent context cut WER by 10.2%, and combining keyterms with contextual prompting drove WER to 1.21% in an independent Coval run.

Which speech-to-text model is most accurate for medical terms?

AssemblyAI's Medical Mode, activated with domain: "medical-v1", delivers the lowest Missed Entity Rate on medical entities across benchmarked providers, with about 20% fewer missed medical entities and 87% fewer entity errors than the base Universal-3 Pro model. For teams processing PHI, AssemblyAI signs a Business Associate Addendum (BAA), provides PHI redaction across audio and transcripts, and maintains SOC 2 Type 2.

How do keyterms prompting and contextual prompting work?

Keyterms prompting gives the model a list of expected terms so it recognizes them more reliably. On Universal-3.5 Pro async you can pass up to 1,000 terms, Universal-3 Pro Streaming supports up to 100 free terms updatable mid-stream, and Universal-3.5 Pro Realtime includes keyterm prompting. Contextual prompting passes broader conversational context, like a meeting title or a patient's prior-visit note. The one caution: over-biasing with too many keyterms can hurt general accuracy, so bias only toward the entities you genuinely expect.