Medical terminology accuracy: Techniques for domain-specific transcription

A health system runs a quality audit on 200 ambient scribe notes. The vendor's contract promises 98% accuracy, and the audit confirms it — word-level agreement comes in just over that. The same audit finds eleven medication names wrong, two dosages transposed, and four instances where a negation flipped: "no history of atrial fibrillation" became "history of atrial fibrillation."

Both results are true simultaneously, and that's the problem. Word error rate and clinical usability are only loosely related. A transcript can be 98% word-accurate and clinically unreliable, because the 2% that's wrong isn't distributed randomly — it concentrates in exactly the rare, information-dense tokens that a general language model finds hardest to predict.

So the accuracy question worth asking is narrower than "how accurate is it." It's: what fraction of clinically meaningful entities survives? For Universal-3.5 Pro with Medical Mode enabled, that's a 3.2% Missed Entity Rate — the lowest across the providers we benchmarked, and 87% fewer entity errors than the same model without Medical Mode.

This post is about measurement: which metrics mean what, how our benchmark is constructed, how providers compare, and how to build a benchmark on your own audio so you're not taking anyone's word for it. For the mechanics of why specialist vocabulary breaks models and what fixes it, see the companion post on how accurate AI transcripts are on technical and medical terms.

"Accuracy" means at least four different things

When a vendor says 98% accurate, they could be describing any of these, and the numbers are not interchangeable.

Word error rate

WER counts substitutions, insertions, and deletions divided by the number of reference words. Every token weighs the same. In a clinical transcript, function words — the, and, a, of, patient — make up the bulk of the text, so WER is dominated by the words nobody would misread anyway. It's a legitimate metric for tracking whether a model is getting better in general. It is a poor specification for clinical safety.

Missed Entity Rate

MER restricts scoring to clinically meaningful entities: drug names, dosages, routes, frequencies, conditions, procedures, anatomy, and negations. If a transcript renders every function word perfectly and drops two medications, WER barely moves and MER moves sharply. That asymmetry is the point — MER tracks the errors that cause a note to be rewritten or, worse, to be trusted when it shouldn't be.

Diarization accuracy

In a two-speaker encounter, attribution is part of the meaning. "I stopped taking the lisinopril" from the patient is a medication reconciliation issue; the same sentence attributed to the clinician is nonsense. The conventional metric here, diarization error rate, measures how much audio time was assigned to the right speaker — which rewards systems that nail long monologues and shrug off short interjections. We optimize Universal-3.5 Pro's diarization for cpWER instead: concatenated minimum-permutation word error rate, which only improves when the actual words land under the correct speaker. It's the most accurate diarization we've shipped, and the improvement shows up most on short turns and overlapped speech.

Downstream note quality

The metric clinicians actually care about is whether the finished note is correct and requires no rework. That's a function of transcript accuracy plus whatever your summarization step does. It's worth measuring separately, because a great transcript can still produce a bad note — but a bad transcript cannot produce a good one.

How we measure Medical Mode

Medical Mode is a clinical domain adaptation applied with one parameter, domain: "medical-v1", on top of our flagship models. There's no model switch, no separate endpoint, and no vocabulary list to maintain, which makes the A/B comparison unusually clean: same model, same audio, one field different.

Against that baseline, Medical Mode delivers:

• 3.2% Missed Entity Rate, absolute — roughly three of every hundred clinical entities missed.
• About 20% fewer missed medical entities than the same base model with Medical Mode off.
• 87% fewer entity errors than the base model without Medical Mode.

Those three figures answer different questions and shouldn't be blended. The first tells you where the model sits in absolute terms. The second tells you what the add-on buys you on missed entities specifically. The third is the broader error reduction, and it's the largest because it counts entities that were transcribed but transcribed wrong, not just the ones dropped entirely.

Methodology, audio characteristics, and the provider-by-provider results live on our benchmarks page. Medical Mode itself is documented on the healthcare solutions page.

How the providers compare on Missed Entity Rate

The spread across the benchmarked set is not a rounding difference. At a high MER, every note needs a full read against the audio and the automation has produced no net saving. At 3.2%, review becomes a skim. That's the threshold where the economics of AI documentation actually work.

How to benchmark on your own audio

Any published benchmark is measured on someone else's audio. Yours has different accents, specialties, microphones, and room acoustics. Here's a benchmark process that takes a day and tells you more than any vendor datasheet.

1. Build a test set that looks like production

Pull 30 to 50 real recordings spanning your specialty mix, your worst microphones, your heaviest accents, and at least a few encounters with an interpreter or a family member present. Deliberately include the audio your team complains about. A test set of clean dictations will rank every provider as excellent.

2. Write reference transcripts, then extract the entity list

Have a human produce a verbatim reference for each file. Then, separately, list the entities in each: every drug, dose, route, frequency, condition, procedure, and negation. That entity list — not the full transcript — is what you'll score against. This is the step teams skip, and it's the step that makes the whole exercise meaningful.

3. Score entity-level, and count negations separately

For each provider run, count entities correct, entities missed entirely, and entities transcribed wrong. Report those separately rather than collapsing them into a single accuracy figure. Track flipped negations as their own category — they're rare and disproportionately dangerous.

4. Test the configurations, not just the providers

Most of the accuracy variance you'll find comes from configuration, not from provider choice. Run each provider with and without its clinical mode, with and without speaker labels, and with and without context passed in. Here's the AssemblyAI run:

import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"

def transcribe(path, medical=True, context=None):
    config = aai.TranscriptionConfig(
        speech_models=["universal-3-5-pro"],
        domain="medical-v1" if medical else None,
        speaker_labels=True,
    )
    if context:
        config = aai.TranscriptionConfig(speech_models=["universal-3-5-pro"], domain="medical-v1" if medical else None, speaker_labels=True, prompt=context)  # e.g. the patient's prior-visit note

    return aai.Transcriber(config=config).transcribe(path)

for path in test_set:
    baseline = transcribe(path, medical=False)
    medical = transcribe(path, medical=True)
    with_ctx = transcribe(path, medical=True, context=prior_notes[path])
    # score each against your entity list

Async transcription uses the plural speech_models field; streaming uses the singular speech_model. Full parameter reference is in the docs.

5. Check confidence scores against your own errors

Word-level confidence scores are useful for routing, not for truth. Once you have scored output, look at whether low-confidence spans actually correlate with your entity errors. If they do, you can build a review queue that surfaces only the risky segments — which is a far better use of clinician attention than uniform review.

What actually moves the number

Once you can measure, three configuration changes account for most of the achievable improvement.

Turn on clinical domain adaptation

The single largest step: 87% fewer entity errors from one parameter. It costs $0.15/hr on top of the base model — $0.36/hr combined for async, $0.60/hr for streaming. See pricing.

Pass context from the record

Feeding a patient's prior-visit note alongside the audio cut missed medical terms by 31% in an internal healthcare test. The model gets that patient's real medication list and diagnoses before decoding, which is strictly more information than any static vocabulary file can carry. If your application already has the chart open, this is free accuracy you're not collecting.

Match the model and settings to the audio

For post-visit processing, async with the full recording beats streaming on accuracy — the model sees everything. For live workloads, Universal-3.5 Pro Realtime exposes mode across min_latency, balanced, and max_accuracy, plus voice_focus with near-field and far-field profiles. Choosing far-field for a room microphone is worth real accuracy on the quieter speaker. In voice agent workloads, prompt cut word error rate 10.2% across 20,000 files, with detailed context cutting medical-term entity errors 43%.

Know your language coverage precisely

Universal-3.5 Pro code-switches natively across 18 languages with no configuration, so mixed-language encounters don't need a language hint. Medical Mode's clinical adaptation covers English, Spanish, German, and French. Those are separate capability lists — benchmark each language you actually see, and don't assume clinical adaptation extends to every language the base model handles.

Setting an accuracy target you can defend

"98% accurate" is a bad contract term because it doesn't say 98% of what. Replace it with something measurable: a maximum Missed Entity Rate on a named test set, measured on your audio, with medication names and negations reported separately. Add a re-measurement cadence, because models change and your audio mix changes too.

Then build the review process around the measured number rather than the promised one. If your MER is 3%, sample-based review with a low-confidence queue is defensible. If it's 15%, you need full review, and you should say so out loud rather than discovering it in an audit.

Related reading if you're choosing between vendors rather than measuring one: our guide to medical transcription services covers the human, hybrid, and API models and what each costs.

PHI in your benchmark data

A benchmark test set is a pile of PHI, and it tends to live in engineering environments with looser controls than production. Handle it accordingly. AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and operates as a business associate under HIPAA, so your evaluation traffic can be covered under the same terms as production. PHI redaction is available across both audio and transcripts, the platform is SOC 2 Type 2 audited, and self-hosted deployment and EU data residency are available where residency rules apply. See the BAA FAQ and the BAA terms. Teams running this kind of evaluation on our speech-to-text include Sully AI, Heidi Health, Knowtex, Deepscribe, and Magentus Healthcare.

Where clinical accuracy measurement goes next

MER is a better metric than WER, and it's still a proxy. What a health system actually wants to know is whether a note is safe to sign, and that's a claim about the finished document, not about the transcript underneath it. The measurement frontier over the next couple of years is end-to-end: scoring the generated note against the encounter, entity by entity and assertion by assertion, including whether a negation survived the summarization step as well as the transcription step. Expect vendor accuracy claims to migrate there, and expect the gap between providers to widen rather than narrow when they do — because a pipeline that loses entities early has nowhere to hide once you score the output instead of the intermediate.

Frequently asked questions

How accurate is AssemblyAI Medical Mode compared to other providers?

Universal-3.5 Pro with Medical Mode posts a 3.2% Missed Entity Rate, the lowest across the providers in our evaluation. Methodology is on our benchmarks page.

What's the difference between word error rate and Missed Entity Rate?

WER weighs every word equally, so it's dominated by function words that nobody would misread. MER scores only clinically meaningful entities — drugs, dosages, conditions, procedures, negations. A transcript can post an excellent WER while missing medication names, which is why MER is the metric to hold a vendor to for clinical work.

What accuracy do healthcare organizations need from AI transcription?

Set the target on entities rather than words. A defensible spec names a maximum Missed Entity Rate on a test set drawn from your own audio, reports medication names and negations as separate categories, and includes a re-measurement schedule. A blanket "98% accurate" clause doesn't tell you whether the misses are articles or antibiotics.

Is AssemblyAI more accurate than Deepgram for medical transcription?

On the entity-level metric that matters for clinical audio, yes — 3.2% MER for Medical Mode, the lowest in our benchmark. Both are strong general models, and the gap is specifically in how clinical entities survive. Run both on your own recordings before you decide; that's the only comparison that binds.

Do confidence scores tell me which parts of a transcript to review?

They're useful for triage, not proof. Word-level confidence correlates with error risk but imperfectly, so validate the correlation against your own scored output before you build a workflow on it. Once validated, a low-confidence review queue puts clinician attention where the risk actually is instead of spreading it evenly.

How does AssemblyAI handle HIPAA and PHI?

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and acts as a business associate under HIPAA. That's backed by PHI redaction across audio and transcripts, SOC 2 Type 2 audited infrastructure, and self-hosted or EU-resident deployment where required. Details in the BAA FAQ.