Medical transcription that actually works — Beyond generic STT

A discharge summary says the patient is on 25 milligrams of metoprolol. The physician said 50. Somewhere between the dictation and the document, a digit moved. That's the reason medical transcription is a discipline rather than a feature — the cost of an error isn't an awkward sentence, it's a clinical decision made on wrong information.

The good news is that the accuracy question now has a real answer. Not word error rate, which is the wrong instrument, but Missed Entity Rate: how often the drugs, conditions, and procedures spoken aloud fail to appear in the transcript. Universal-3.5 Pro with Medical Mode enabled posts a 3.2% MER — the lowest across the providers we've benchmarked.

This post covers what medical transcription is, the three ways it gets done today, how to measure whether a system is good enough, how the major providers actually compare, and what to build if you're putting transcription inside a healthcare product.

What medical transcription is

Medical transcription converts spoken clinical content into written documentation that becomes part of the patient record. The common outputs:

• Progress notes and SOAP notes from an office visit or encounter.
• History and physical (H&P) documents at admission.
• Operative reports dictated after a procedure.
• Radiology and pathology reports, typically dictated by a single specialist.
• Discharge summaries, which carry outsized risk because the next clinician often reads only this.
• Consultation letters and referral notes.

Two things make this harder than general transcription. The vocabulary is rare and unforgiving — drug names, dosages, ICD and CPT terminology, anatomical structures, abbreviations that mean different things by specialty. And the acoustics are frequently bad: a microphone across the desk, equipment noise, two or three people talking, a patient who speaks quietly.

The three ways it gets done today

The shift worth naming: legacy dictation asked the clinician to change how they speak. Modern ASR doesn't. That's why ambient documentation became viable — the model handles conversational speech, multiple speakers, and background noise well enough that nobody has to talk to the machine.

Why accuracy matters, and how to measure it properly

Word error rate is the metric everyone quotes and it's close to useless here.

Consider a ten-minute encounter with 1,500 words, twelve of which are clinical entities — three drug names, two dosages, four conditions, three procedures. A system that transcribes all 1,488 ordinary words perfectly and misses every single entity posts a 0.8% WER. That's a headline-grade number attached to a clinically worthless transcript.

Missed Entity Rate fixes the measurement by counting only the terms that carry clinical meaning. It's the number you should ask every vendor for, and the number you should compute yourself on your own audio.

What to require

Three checks, in order of how often they catch real problems:

1. Entity capture. List the drugs, conditions, procedures, and dosages actually spoken in a set of your own recordings. Count how many appear correctly in each candidate's transcript.
2. Speaker attribution. Verify that the patient's short answers were attributed to the patient. "Patient reports chest pain" and "clinician reports chest pain" are different records.
3. Failure behavior. When the model is uncertain, does it produce a gap or a confident guess? For clinical documentation, a gap is strictly safer.

How the providers compare

Methodology for the benchmarked figures is on the benchmarks page. Two honest caveats: our benchmark is ours, so run your own audio before deciding; and "not published in our benchmark" means we haven't measured that provider on medical entities, not that they're bad.

The structural difference worth more than a few points of accuracy: with Medical Mode, clinical support is a parameter on the model you're already using. With AWS and Deepgram it's a different model or service, which means a separate integration, separate feature parity questions, and a migration if you decide to change your mind.

Medical Mode: what the parameter does

Medical Mode is a clinical domain setting: domain: "medical-v1". Same model ID, same endpoint, same response shape, on both pre-recorded and streaming.

Against the same base model with it switched off:

• 3.2% Missed Entity Rate in absolute terms.
• ~20% fewer missed medical entities.
• 87% fewer entity errors — the terms it got wrong rather than skipped, which is the more dangerous category.

It adds $0.15/hr to the base rate: $0.36/hr for pre-recorded, $0.60/hr for streaming. Coverage is English, Spanish, German, and French.

Context does more than a vocabulary list

The most useful finding we have on clinical accuracy has nothing to do with the model. In an internal healthcare test, feeding the model a patient's prior-visit note as context cut missed medical terms by 31%.

The reason is simple: the hardest words in any encounter are the ones specific to this patient. The biologic they started last month, the surgeon they're seeing, the abbreviation this clinic uses. A general clinical model can't know those. The record already does. If your system can fetch the last note, that's a bigger accuracy win than most model upgrades.

Multi-speaker clinical audio

Dictation is one voice. An encounter is two to five, unevenly balanced, frequently overlapping. Getting attribution right is not optional — a transcript with the right words on the wrong speaker produces a note that's actively misleading.

Most diarization gets measured with Diarization Error Rate, which scores how much audio time landed on the right speaker. DER rewards handling long monologues and barely penalizes dropping a two-word patient answer. Concatenated minimum-permutation WER (cpWER) scores the words in each speaker's transcript instead — which is what your documentation pipeline consumes.

Universal-3.5 Pro's diarization is the most accurate we've shipped, and it's optimized for cpWER. It captures short turns and overlapped speech — the "since Tuesday," the "the left one," the interruptions — which is where clinical content actually lives. In streaming it supports revision, correcting early attributions as more audio arrives, and handles up to 10 speakers.

Multilingual encounters

Two separate capabilities, and mixing them up leads to bad scoping.

The base model code-switches natively across 18 languages with no configuration. An encounter that moves between English and Spanish mid-sentence transcribes correctly — no language detection, no request routing, no second pipeline.

Medical Mode's clinical entity accuracy covers English, Spanish, German, and French. A Vietnamese-English encounter still transcribes well on the base model; it just doesn't get the entity boost.

‍

Real-time or batch: pick by workflow

Both flagships share the model ID universal-3-5-pro. What differs is the transport, the price, and the parameter name.

Use batch (async) for dictation files, post-visit notes, chart review, and transcribing an archive. It's cheaper at $0.21/hr base, and the model sees the whole file before committing to anything. Async uses speech_models, plural:

import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro"],
    domain="medical-v1",
    speaker_labels=True,
    speakers_expected=2,
    punctuate=True,
    format_text=True,
)

transcript = aai.Transcriber(config=config).transcribe(
    "./discharge-dictation.wav"
)

if transcript.status == "error":
    raise RuntimeError(transcript.error)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

Use streaming when a clinician or patient sees words appear live — telehealth captions, live scribing, voice agents. Connect to wss://streaming.assemblyai.com/v3/ws with the singular speech_model:

from assemblyai.streaming.v3 import (
    StreamingClient,
    StreamingClientOptions,
    StreamingParameters,
    TurnEvent,
)

def on_turn(client, event: TurnEvent):
    if event.end_of_turn:
        print(event.transcript)

client = StreamingClient(StreamingClientOptions(api_key="YOUR_API_KEY"))
client.on(TurnEvent, on_turn)

client.connect(
    StreamingParameters(
        speech_model="universal-3-5-pro",
        domain="medical-v1",
        sample_rate=16000,
        speaker_labels=True,
        voice_focus="far-field",
        mode="max_accuracy",
    )
)

Set voice_focus to near-field for headsets and phones, far-field for a room microphone. mode takes min_latency, balanced, or max_accuracy — documentation belongs at the accurate end; turn detection defaults to min_turn_silence 128ms and max_turn_silence 1280ms on the balanced preset when conversational timing matters. Streaming runs $0.45/hr base, $0.60/hr with Medical Mode, and the Realtime post covers the rest. For genuinely conversational products — intake, triage, follow-up calls — the Voice Agent API replaces STT, LLM, and TTS with one WebSocket at a flat $4.50/hr. Rates on the pricing page.

Security, PHI, and compliance

Every healthcare transcription project ends up in a security review, and the questions are predictable enough to prepare for.

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI, which makes us a business associate under HIPAA. Terms and the request path are at can you sign a BAA and legal/business-associate-agreement. Send your compliance reviewer the links rather than a summary.

The supporting facts: PHI redaction across both audio and transcripts, which lets you keep a de-identified transcript alongside a redacted recording instead of choosing. SOC 2 Type 2. Self-hosted deployment and EU data residency via api.eu.assemblyai.com for teams with isolation or residency requirements.

What no vendor supplies: consent, retention policy, and access audit inside your own product. Recording clinical encounters carries state-level requirements. Decide your retention window before you store anything, build explicit deletion into the pipeline, and be able to say in one sentence who can access a transcript and for how long.

Where medical transcription is going

For thirty years the constraint on medical transcription was throughput — how fast words could become documents. That constraint is gone. A ten-minute encounter transcribes for pennies in seconds, with better clinical entity accuracy than the workflows it replaced.

The next constraint is memory. Every transcription request today arrives with no knowledge of the patient it concerns, which is why one prior-visit note as context cuts missed medical terms by 31% — the model was missing information the record already held. Follow that line and transcription stops being a step between speech and the chart and becomes something closer to a reader of the chart: a system that knows this patient's medications and this clinic's shorthand before anyone speaks, and gets more accurate with every visit.

That's an integration problem, not a model problem, which means the teams who win it will be the ones who wire their record system into the transcription request. Most haven't yet.

More depth: AI medical transcription, medical voice recognition, how to build an AI medical scribe, the best ambient AI scribes, the medical transcription use case, the healthcare solutions overview, and the docs. Healthcare teams on this stack include Sully AI, Heidi Health, Deepscribe, Knowtex, Magentus Healthcare, Chapter, NMDP, and Commure.

Frequently asked questions

How accurate does medical transcription need to be?

Accurate enough that clinical entities survive, which is a different bar than a low word error rate. A transcript can post under 1% WER while missing every drug name, because entities are a tiny fraction of total words. Ask for Missed Entity Rate instead: Universal-3.5 Pro with Medical Mode posts 3.2% MER. Then compute it yourself on your own audio.

AssemblyAI vs Amazon Transcribe Medical — which is better for clinical transcription?

On medical entity capture, Universal-3.5 Pro with Medical Mode measures 3.2% MER, the lowest in our benchmark set, with methodology on the benchmarks page. Architecturally, Transcribe Medical is a separate AWS service with its own endpoint and feature set, while Medical Mode is one parameter on the model you're already calling — so you're not maintaining two integrations or migrating if requirements change.

AssemblyAI vs OpenAI for medical transcription?

Whisper is a strong general-purpose model with no clinical domain option, no built-in speaker diarization tuned for clinical encounters, and no PHI redaction — you'd build those layers yourself. For clinical documentation, the gap isn't general transcription quality, it's medical entity accuracy and the compliance surface around it. If you're evaluating both, score entity capture and speaker attribution separately rather than comparing overall WER.

How does AssemblyAI handle HIPAA and PHI?

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI, which makes us a business associate under HIPAA. Alongside the BAA: PHI redaction across audio and transcripts, SOC 2 Type 2, and self-hosted or EU data residency options. Details and the request path are at can you sign a BAA.

Should I use real-time or batch medical transcription?

Batch unless someone is watching the words appear. Async is cheaper at $0.21/hr base ($0.36/hr with Medical Mode), the model sees the whole file before committing, and you avoid a class of connection-handling bugs. Use streaming — $0.45/hr base, $0.60/hr with Medical Mode — for live captions, in-encounter scribing, and voice agents.

How does AssemblyAI handle medical jargon and abbreviations?

Medical Mode covers the general clinical vocabulary by training on clinical language rather than boosting a keyword list, which avoids both the maintenance burden and the hallucination risk of aggressive term forcing. Patient-specific and clinic-specific terms come from contextual prompting: passing a patient's prior-visit note cut missed medical terms by 31% in an internal healthcare test. See the docs for how to pass context.