Medical voice recognition: How AI solves terminology problems

Say "hydralazine" and "hydroxyzine" out loud. One's a vasodilator for hypertension, the other's an antihistamine used for anxiety and itching. They differ by a couple of phonemes, and a general-purpose speech model will happily pick whichever one appeared more often in its training data. That's medical voice recognition in a nutshell: the vocabulary that carries the clinical risk is the vocabulary the model has heard least.

The measurement that captures this is Missed Entity Rate — how often the drugs, conditions, and procedures spoken in the audio simply fail to appear in the transcript. Universal-3.5 Pro with Medical Mode lands at a 3.2% MER, the lowest across the providers we've benchmarked.

This post covers what medical voice recognition actually is, why generic models break on it, how Medical Mode changes the numbers, what it costs, and how to evaluate any vendor in this space — including us — without taking a marketing page at its word.

What medical voice recognition means

Medical voice recognition is speech-to-text tuned for clinical language and clinical acoustics. Two halves, and vendors tend to be good at one.

The language half is vocabulary and context: drug names, dosages, ICD and CPT terminology, anatomical structures, abbreviations that mean different things in cardiology than in dermatology. The acoustics half is the room: a microphone across the desk, a ventilator running, two people talking at once, a patient who speaks quietly and switches languages when their daughter walks in.

The applications split roughly three ways:

Dictation

A clinician speaking directly to the system, deliberately, often with a headset. Clean audio, single speaker, dense terminology. Historically the easiest case, and where legacy dictation products earned their reputation.

Ambient documentation

The system listens to the whole encounter and produces a note afterward. Harder on every axis — far-field audio, multiple speakers, conversational speech, and the clinician never repeats anything for the microphone's benefit.

Conversational voice

Intake bots, appointment lines, symptom triage, medication reminders. Real-time, latency-sensitive, and increasingly where healthcare voice budgets are going.

Why general-purpose models break on clinical audio

A general model's whole job is to be broadly right, which means resolving ambiguous audio toward the likelier word. In everyday speech that's correct behavior. In a clinic it's a systematic bias against exactly the terms you need.

The usual workaround is a custom vocabulary list. It helps, and it doesn't scale. You end up maintaining a formulary by hand, per specialty, and still missing the terms nobody thought to add — the new biologic, the local abbreviation, the device brand this hospital happens to use. Worse, aggressive keyword boosting starts hallucinating those terms into audio where nobody said them, which is a more dangerous failure than a miss.

There's also a quieter failure: speaker attribution. Get the words right but the speaker wrong and you've produced a note that says the clinician reported the symptom. That's not a transcription error you can catch by reading for typos.

How Medical Mode changes the numbers

Medical Mode is a domain setting on Universal-3.5 Pro and Universal-3.5 Pro Realtime. You set domain: "medical-v1". Same model ID, same endpoint, same response shape — no migration, no separate medical service to integrate against.

Against the same base model with Medical Mode off:

• 3.2% Missed Entity Rate in absolute terms.
• ~20% fewer missed medical entities.
• 87% fewer entity errors.

The 87% figure is the one worth internalizing. It's not about words the model skipped — it's about entities it got actively wrong, which is the "hydralazine becomes hydroxyzine" class of error.

Full methodology and the underlying files are on the benchmarks page. If a vendor won't show you a medical-specific evaluation, that itself is information.

Turning it on

For pre-recorded audio — dictation files, recorded encounters, chart review — note that speech_models is plural here and takes an array:

import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro"],
    domain="medical-v1",
    speaker_labels=True,
    speakers_expected=2,
)

transcript = aai.Transcriber(config=config).transcribe(
    "./clinic-encounter.wav"
)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

For live recognition, connect to wss://streaming.assemblyai.com/v3/ws. Streaming uses the singular speech_model:

from assemblyai.streaming.v3 import (
    StreamingClient,
    StreamingClientOptions,
    StreamingParameters,
    TurnEvent,
)

def on_turn(client, event: TurnEvent):
    if event.end_of_turn:
        print(event.transcript)

client = StreamingClient(StreamingClientOptions(api_key="YOUR_API_KEY"))
client.on(TurnEvent, on_turn)

client.connect(
    StreamingParameters(
        speech_model="universal-3-5-pro",
        domain="medical-v1",
        sample_rate=16000,
        speaker_labels=True,
        voice_focus="far-field",
        mode="max_accuracy",
    )
)

Two streaming settings matter more in healthcare than anywhere else. voice_focus takes near-field for headsets and phones or far-field for a room microphone — ambient use is almost always far-field, and leaving it on the wrong setting costs you accuracy for free. And mode takes min_latency, balanced, or max_accuracy; documentation workflows should sit at the accurate end, since nobody is waiting on a 200ms response to read their own note. Turn detection defaults to min_turn_silence 128ms and max_turn_silence 1280ms on the balanced preset when you do need conversational timing.

The doctor-patient separation problem

Diarization deserves more attention than it usually gets in vendor comparisons, because a scribe pipeline consumes speaker-attributed text, not raw text.

Most diarization is measured with Diarization Error Rate, which scores how much audio time landed on the wrong speaker. DER flatters systems that handle long monologues well and barely penalizes dropping a two-word patient answer. Concatenated minimum-permutation WER — cpWER — scores the words in each speaker's transcript instead. That's the number that predicts whether your note is right.

Universal-3.5 Pro's diarization is optimized for cpWER, and it's the most accurate diarization we've shipped. It captures short turns and overlapped speech, which is most of what a patient contributes to an encounter. In streaming it supports revision — early speaker guesses get corrected as more audio arrives — and handles up to 10 speakers, enough for a care team plus family in the room.

Far-field is the case worth testing hardest

Ambient clinical audio means the microphone is across the room, not at anyone's mouth. Set voice_focus to far-field, run mode: "max_accuracy" since documentation isn't latency-critical, and build your evaluation set from actual room recordings rather than headset audio — headset files flatter every vendor equally and tell you nothing.

Multilingual clinical audio

Two facts, and they get conflated constantly, so keep them apart.

The base model code-switches natively across 18 languages with no configuration. A conversation that moves between English and Spanish mid-sentence transcribes correctly — you don't detect language, you don't route requests, you don't run two pipelines.

Medical Mode's clinical entity accuracy covers English, Spanish, German, and French, for both pre-recorded and streaming. A Tagalog-English encounter gets the code-switching but not the entity boost.

Know which of those you're in before you promise a health system anything. Teams that need both usually find their patient population is well inside EN/ES/DE/FR anyway.

Where realtime accuracy shows up in practice

The combination of accuracy, latency, and language switching is what teams building live products actually evaluate on. As Foysal Osmany, Software Engineer at Fireflies, put it:

"We were searching for the best realtime ASR model for our voice agent pipeline in Fireflies. The new Universal 3.5 Pro speech model from Assembly is best so far in terms of accuracy, latency and language switching."

Healthcare teams building on the same stack include Sully AI, Heidi Health, Deepscribe, Knowtex, Magentus Healthcare, and Commure.

Context beats vocabulary lists

If you take one practical thing from this post, take this. In an internal healthcare test, feeding the model a patient's prior-visit note as context cut missed medical terms by 31%.

No new model. No fine-tune. No vocabulary maintenance. Just handing the model the document that already lists this patient's medications, conditions, and specialists before transcribing their next visit.

For conversational products, the streaming equivalent is prompt, which cut word error rate 10.2% across 20,000 voice agent files, and detailed context cut medical-term entity errors 43%.

The strategic point: accuracy in medical voice recognition is shifting from "which model" to "what context did you give it." Teams still maintaining 5,000-term specialty dictionaries are solving a problem that context handles better.

What it costs

Pricing here is simple enough to reason about on a whiteboard:

Run the math against clinician time and the decision usually makes itself. A ten-minute encounter transcribed with Medical Mode costs six cents. Details on the pricing page.

How to evaluate any vendor, including us

Test on your own audio, not a demo file

Vendor demo audio is clean. Yours isn't. Pull 20 real recordings that represent your worst case — the noisiest room, the fastest talker, the bilingual encounter — and run all candidates on the same set.

Score entities, not words

Build a list of the drugs, conditions, and procedures actually spoken in your test set and count how many appear correctly in each transcript. Word error rate will mislead you.

Score speaker attribution separately

Check whether the patient's short answers landed on the patient. This is where most pipelines quietly fail.

Ask the compliance questions early

Will the vendor sign a Business Associate Addendum? Can they redact PHI from audio as well as transcripts? Do they hold SOC 2 Type 2? Is there a self-hosted or regional deployment option if your customers demand it?

PHI handling and compliance

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI, which makes us a business associate under HIPAA. The terms and request path are documented at can you sign a BAA and legal/business-associate-agreement — send those to your compliance reviewer directly.

Alongside that: PHI redaction runs across both audio and transcripts, so you can keep a de-identified transcript and a redacted recording rather than choosing. We hold SOC 2 Type 2. Self-hosted deployment and EU data residency via api.eu.assemblyai.com are available for teams with residency or isolation requirements.

Consent is the piece no vendor supplies. Recording clinical encounters carries state-level requirements, and your product needs a documented consent step with an audit trail. Design it in, don't bolt it on.

Where medical voice recognition is heading

The interesting shift isn't another point of accuracy. It's that the accuracy ceiling is moving from the model to the context you feed it. A 31% reduction in missed medical terms from one prior-visit note is a bigger jump than most model generations deliver, and it costs a database query.

Follow that line and medical voice recognition stops being a transcription problem. The system that knows a patient's medication list, their specialists, and their clinic's local shorthand before the visit begins doesn't just hear better — it starts to be the place the record lives rather than a step on the way to it. The teams wiring their EHR into the transcription request today are building toward that. The ones maintaining vocabulary spreadsheets are not.

Adjacent reading: AI medical transcription, how to build an AI medical scribe, the best ambient AI scribes, and the medical transcription use case.

Frequently asked questions

Which speech-to-text API is most accurate for medical terminology?

On medical entity capture, Universal-3.5 Pro with Medical Mode posts a 3.2% Missed Entity Rate — the lowest across the providers we've benchmarked. See the benchmarks page for methodology. That said, run your own audio: the ranking on a curated benchmark and the ranking on your specialty's recordings aren't guaranteed to match, and any vendor worth using will let you test for free.

How accurate is AssemblyAI Medical Mode compared to other providers?

3.2% MER, the lowest across the providers benchmarked. Against our own base model without Medical Mode, it delivers roughly 20% fewer missed medical entities and 87% fewer entity errors. The 87% figure covers entities the model got wrong rather than skipped, which is the more clinically dangerous category.

How do I turn on Medical Mode, and what does it cost?

Add domain: "medical-v1" to your request. No model switch, no separate endpoint, works on both async and streaming. It adds $0.15/hr on top of the base rate: $0.36/hr combined for pre-recorded audio, $0.60/hr for streaming. It supports English, Spanish, German, and French.

How does AssemblyAI handle HIPAA and PHI?

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI, making us a business associate under HIPAA. Alongside the BAA you get PHI redaction across audio and transcripts, SOC 2 Type 2, and self-hosted or EU-residency deployment options. Start at can you sign a BAA.

Can medical voice recognition separate the doctor from the patient?

Yes, and it's worth testing carefully. Universal-3.5 Pro's diarization is optimized for cpWER rather than DER, which means it's scored on getting the right words attributed to the right speaker rather than on total audio time. It captures short turns and overlapped speech — the patient's three-word answers, the interruptions — and in streaming it revises early attributions as more audio arrives, up to 10 speakers.

How does AssemblyAI capture medical jargon accurately?

Medical Mode handles the general clinical vocabulary by training on clinical language rather than boosting a keyword list. Contextual prompting handles the patient-specific and clinic-specific terms — in an internal healthcare test, passing a patient's prior-visit note cut missed medical terms by 31%. For streaming voice agents, prompt cut WER 10.2% across 20,000 files. See the docs for both.

‍