Medical Mode: now with fewer medical-term errors in pre-recorded and real-time transcription

The hard part is the vocabulary

Drug names, dosage phrasing, disease and anatomical terms, abbreviations, and eponyms are rare in the web and conversational audio that speech models learn from. A model that transcribes everyday speech almost perfectly can still turn Celebrex into “Celebrate,” turn lisinopril into “listen a pill,” or drop a dose altogether.

In our evaluations, drug names are the hardest category. On English clinical datasets, a strong general model misses about 6% of drug mentions but under 2% of symptom mentions. On the Spanish set and the French prescription dictations, the drug-name miss rate is more than three times the overall medical-term miss rate.

While word error rate (WER) is a good measure of overall accuracy, it treats every word the same. In a clinical conversation, a transcript can be 99% right and still be wrong where it matters.

Fewer missed medical terms in English, especially drug names

On four English medical datasets, Medical Mode cuts the drug-name miss rate by 30% on pre-recorded audio and by 16% on streaming. It lowers the overall medical-term miss rate on every pre-recorded set and leaves WER unchanged. The sets are doctor-patient conversations, primary-care consultations, respiratory-clinic consultations, and scripted medical dialogues: a mix of public and licensed data, 84 hours in total.

For each dataset, we report three metrics for both pre-recorded and real-time transcription. Lower is better for all three.

• Drug-name miss rate is the share of medication names in the reference transcript that are missing or wrong in the output.
• Medical-term miss rate is the same measure across all medical terms. It covers drugs, dosages, diseases, symptoms, anatomy, procedures, tests, and more. Spelling variants and formatting differences are canonicalized, so only real recognition errors count.
• Word error rate (WER) is the standard measure. We normalize with the Whisper text normalizer plus per-language number and unit rules, so writing a dose as digits or as words isn’t an error.

The charts below show the two miss rates. WER doesn’t change with Medical Mode, staying within 0.1 points on every set, so it appears only in the table under each chart.

Medical Mode helps the most with drug names

Medical Mode uses the surrounding conversation as context to correct medical terms, and the biggest gains come on drug names. On pre-recorded audio, it reduces drug-name misses by 30%.

Across the four datasets evaluated, drug mentions account for 3,065 of 45,000 scored medical-term occurrences. On pre-recorded audio, Medical Mode has a 30% reduction in drug-name misses. Disease names improve as well.

On streaming, the correction works turn by turn within a bounded time budget, so the net gain on drug names is smaller: 32 of 194 misses, or 16%.

The same pattern in Spanish, French, and German

The same parameter works for Spanish, French, and German on pre-recorded and streaming audio. Where a set has enough drug mentions to measure, the pattern repeats: drug names improve most and overall term misses fall.

The sets we evaluated are Spanish clinical conversations (licensed, 4 hours), German medical lecture clips (3.8 hours), and three French sets: read prescription dictations (0.8 hours), simulated emergency calls (3.2 hours), and medical lecture clips (1.1 hours).

Spanish shows the largest effect. Drug-name misses fall by 62% on pre-recorded audio and 49% on streaming, and overall term misses fall by about 40% on pre-recorded audio, on a dataset where drug names are a fifth of all medical terms. French prescription dictations, the densest drug-name set we have, improve by 31% on pre-recorded audio.

The two lecture-clip sets contain few drug mentions: 40 in German and 11 in French. Their overall term miss rates improve in German and are unchanged in French.

WER changes by at most 0.9 points. That largest change is on the French dictation set, where some of the corrected drug names also count as word corrections.

How Medical Mode compares with other providers

We ran five other speech-to-text services on the same nine datasets with the same metrics. Each ran in its medical-specific model or mode wherever the vendor offers one for the language.

Language was pinned wherever the API allows it. Punctuation and formatting were on, and filler removal was on where a toggle exists. No system received custom vocabulary or keyterms. Where a vendor returned an empty transcript for a short clip, it counts as a full deletion.

Pre-recorded audio

Averaged over the nine sets, AssemblyAI with Medical Mode has the lowest error rate on all three measures: WER, medical-term miss rate, and drug-name miss rate. Speechmatics is the closest on WER and drug names, and ElevenLabs Scribe v2 Medical is the closest on medical-term misses.

AssemblyAI also has the best mean rank across the nine sets on all three measures, but it doesn’t lead every row. ElevenLabs Scribe v2 Medical has the lowest error on the Spanish set on all three measures, and Speechmatics has the lowest drug-name miss rate on the French dictation and emergency-call sets. The per-set values are below.

Deepgram’s medical model accepts only English as a pinned language. For Spanish, French, and German it ran in its unpinned multi-language medical mode.

Streaming: accuracy against latency

For streaming, error rate is half the picture. A system also has to finalize its words quickly. The latency that matters in live use is time-to-final on long audio, where a vendor’s own finalization behavior shows. On a short clip, every vendor flushes its pending words when the stream ends, which hides that behavior. So we measured time-to-final on clips of 10 seconds and longer and report the 90th percentile, the value a product has to design around.

Speechmatics and Soniox have the lowest or near-lowest error rates, but at default settings they finalize words about 5 seconds behind the audio. Without Medical Mode, AssemblyAI is the fastest system at 0.63 seconds, with a WER within a point of those two. With Medical Mode, time-to-final rises by about 0.65 seconds to 1.28 seconds, and medical-term and drug-name misses drop to the lowest of any system that finalizes within about a second.

Both AssemblyAI configurations sit on the Pareto frontier: no other system beats either one on both error rate and latency. ElevenLabs and Deepgram finalize in under a second but trail on every error measure. Gemini Live, run at the six concurrent sessions its API allows, has the highest average WER and medical-term miss rate, and it returned short transcripts on two sets.

Whether 0.65 seconds is worth it depends on what you’re building. Medical Mode is one connection parameter, so you can measure the trade-off on your own audio.

Deepgram’s streaming medical model accepts English and German as pinned languages. For Spanish and French it ran in its unpinned multi-language medical mode. ElevenLabs Scribe v2 Realtime finalizes only part of each clip on short multi-segment German clips, so its German row stitches partial results and is less reliable than the others. Gemini Live returned transcripts well short of the reference on the French emergency calls and the primary-care consultations (about 0.64 and 0.88 of the reference length), so part of its error there is dropped content.

Turn it on with one parameter

For pre-recorded audio, add "domain": "medical-v1" to the transcript request. For streaming, pass domain=medical-v1 as a connection parameter when you open the WebSocket, alongside speech_model=universal-3-6-pro.

import requests

response = requests.post(
    "https://api.assemblyai.com/v2/transcript",
    headers={"authorization": "YOUR_API_KEY"},
    json={
        "audio_url": "https://example.com/clinical-visit.mp3",
        "speech_models": ["universal-3-5-pro"],
        "domain": "medical-v1",
    },
)
print(response.json()["id"])

from urllib.parse import urlencode

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-6-pro",
    "domain": "medical-v1",
}

API_ENDPOINT = (
    "wss://streaming.assemblyai.com/v3/ws?" + urlencode(CONNECTION_PARAMS)
)

Read the docs: Pre-recorded Medical Mode, Real-time Medical Mode.

Our latest models already handle most medical vocabulary well, so the lift from Medical Mode depends on your audio and your vocabulary. Run your files with and without it, score both on the terms that matter to your users, and keep Medical Mode on wherever it performs best. You can try it without code in the pre-recorded playground or the real-time playground.

How we measured

All numbers come from one evaluation run in November 2026 on 12 medical speech datasets in six languages, about 107 hours of audio, with human reference transcripts. These results come from a different evaluation than the figures on our benchmarks page, so the two shouldn’t be compared directly.

Datasets

The datasets are a mix of public and licensed sets:

• English: doctor-patient conversations, primary-care consultations, respiratory consultations, and scripted dialogues
• Spanish: scripted clinical conversations
• French: prescription dictations, emergency calls, and medical lecture clips
• German and Mandarin: medical lecture clips
• Vietnamese: medical speech and code-switched medical speech

Where a public set’s released alignments were shifted, we repaired the reference boundaries before scoring.

Medical-term scoring

A language model identified the medical terms in the reference transcripts using a fixed taxonomy of 14 categories. We calibrated it against two sets with human term annotations, and it reached 98% recall on the French one. A term counts as recognized when all its words appear correctly, in order, in the transcript. The miss rate is the share of reference terms not recognized.

Before scoring, the reference and the transcript go through the same normalization for every system: the Whisper text normalizer, per-language number and unit rules, and a canonicalization of spelling and formatting variants. Writing 500 mg as “five hundred milligrams,” or Diclofenac as “diclofénac,” doesn’t count as a miss. We also measured false-alarm rates. They stay below 3% for every system and are omitted here for brevity.

Industry comparison

Every system used its medical-specific model or mode wherever one exists for the language. Language was pinned where the API allows it, formatting was on, filler removal was on where available, and everything else stayed at vendor defaults. No system received custom vocabulary. Streaming runs sent 16 kHz audio in real time and scored the joined final transcripts. Where a vendor returned no transcript for a clip, it was scored as all words missing. The AssemblyAI Medical Mode numbers for pre-recorded audio are the mean of two runs at the production temperature setting.

Streaming latency

We measured latency with one client for every vendor. It streams 16 kHz audio over each vendor’s WebSocket API in 100 ms chunks at real-time pace, records when each chunk is sent and when each message arrives, and then aligns the words in the final transcripts to the reference words. Only words the system got right are timed.

A word’s time-to-final is the arrival time of the final message that contains it, minus the send time of the chunk in which the word ends in the reference. We take the last word of each final message, so a system isn’t penalized for finalizing several words at once.

For the latency measurement only, the audio was 500 clips of read English speech from LibriSpeech test-clean and test-other, about an hour in total. LibriSpeech played no part in the accuracy results, which all come from the medical sets. Vendors ran with default endpointing settings, and AssemblyAI ran with formatted turns. Because every vendor flushes its pending words when a stream ends, short clips flatter systems that finalize slowly. The latency figures here use only clips of 10 seconds and longer and report the 90th percentile.