Speech-to-text for healthcare developer guide

Hydralazine and hydroxyzine are one phoneme apart. One is a vasodilator for hypertension. The other is an antihistamine. A general-purpose speech model, hearing a tired resident say either one at speed over exam-room HVAC noise, will pick whichever appeared more often in its training data — and it will render it confidently, in fluent English, with no flag that anything went wrong.

That is the whole case for medical speech-to-text as a separate category. It isn't that clinical audio is noisier, though it is. It's that the words carrying almost all of the clinical meaning are statistically rare, acoustically confusable, and expensive to get wrong. A model optimized to be plausible is exactly the wrong tool.

The number worth anchoring on: Universal-3.5 Pro with Medical Mode enabled records a 3.2% Missed Entity Rate on medical entities — the lowest across benchmarked providers. This post covers what medical speech-to-text has to do, how it differs from general transcription, how the providers stack up, and how to build with it.

What medical speech-to-text actually means

Medical speech-to-text is automatic speech recognition tuned for clinical language and clinical settings. In practice that covers four distinct workflows that people often lump together:

• Dictation — a clinician speaking a note directly, one voice, close mic, usually structured.
• Ambient documentation — a recorded encounter between clinician and patient, transcribed and then structured into a note. This is what AI medical scribes do.
• Real-time conversational — a healthcare voice agent handling intake, scheduling or triage, where the transcript feeds a live decision.
• Retrospective analysis — transcribing archived calls or encounters for quality review, coding, or research.

They share a vocabulary problem and diverge on everything else — latency tolerance, speaker count, microphone distance, and how much post-processing you need. Pick your requirements from the workflow, not from a generic feature list.

Medical versus general-purpose speech-to-text

Notice what's missing from that table: a separate model. The older assumption was that medical transcription meant migrating to a different, usually worse, specialty engine and giving up whatever general capability you had. That trade-off is gone.

How medical models handle specialized vocabulary

There are three mechanisms in play, and they stack.

Domain-tuned decoding

Medical Mode is a single parameter on the request — domain: "medical-v1" — not a model swap. You keep the same model ID, the same endpoint and the same response shape. Against the base model running without it, Medical Mode delivers 87% fewer entity errors and roughly 20% fewer missed medical entities. That's the cleanest way to read what the add-on buys you: it isn't a general accuracy bump, it's a targeted collapse in the error class that matters clinically. Methodology sits on the benchmarks page.

Contextual prompting

Domain tuning handles the terminology every clinician uses. It can't know that this patient is on a specific investigational protocol, or that your practice abbreviates three procedures in a way nobody else does. Contextual prompting closes that gap: feeding a patient's prior-visit note cut missed medical terms by 31% in an internal healthcare test. If you have structured patient context available at request time — and in an EHR-integrated build you almost always do — this is the highest-return change you can make.

Formatting that matches clinical convention

"Twenty five milligrams b.i.d." should not come back as prose. Getting dosages, units, times and standard abbreviations rendered correctly is unglamorous and it's the difference between a transcript a clinician skims and one they rewrite.

The four hard problems in clinical audio

Terminology under acoustic stress

Rare words spoken quickly, often at the end of a sentence where energy drops. This is what Missed Entity Rate measures, and it's the first thing to evaluate.

Clinician versus patient attribution

Universal-3.5 Pro ships the most accurate diarization we've released, and crucially it's optimized for cpWER rather than DER. The distinction matters: cpWER scores whether the right words landed with the right speaker, not just whether speech boundaries were drawn neatly. Clinical encounters are full of the cases where those diverge — a patient answering in two words, a caregiver interjecting, both parties talking at once. Short turns and overlapped speech survive instead of being absorbed. Streaming diarization supports revision across up to 10 speakers.

Microphone distance

Ambient documentation puts the mic on a desk, not on a lapel. For streaming, voice_focus takes near-field or far-field; setting it correctly is a two-second change with a real effect on far-field exam-room audio.

Language that shifts mid-sentence

A Spanish-speaking patient with an English-speaking clinician produces one conversation containing both, frequently inside a single sentence. Universal-3.5 Pro code-switches natively across 18 languages with no configuration — you don't declare a language and you don't lose the switch. Medical Mode itself covers English, Spanish, German and French, across both pre-recorded and streaming. Keep those two facts distinct when you scope a deployment: broad code-switching on the base model, medical entity boost on those four languages.

What to require from a provider

A short list, in the order I'd weight them:

1. A published entity-level accuracy figure, not just a WER claim. If a vendor only quotes WER on clean read speech, they're answering a different question.
2. Async and streaming on the same model. Otherwise your live intake flow and your post-visit note disagree with each other, and you're debugging two accuracy profiles.
3. Diarization you can defend in a clinical record.
4. Language coverage that matches your patient population, including mid-sentence switching.
5. A signed BAA, PHI redaction, and SOC 2 Type 2 — plus residency or self-hosting if your program needs it.
6. One parameter to turn medical handling on, so you can A/B it rather than committing to a migration to find out.

Provider comparison for medical speech-to-text

On the AssemblyAI versus Deepgram question specifically: both offer a medical tier, and Medical Mode records the lowest Missed Entity Rate in the benchmark set. Against Amazon Transcribe Medical the gap is much wider, and there's an architectural difference too — AWS routes you to a separate service, while Medical Mode is a flag on the model you're already calling. Against OpenAI's models, there's no medical domain to enable at all; you're relying on a general model's priors for terms it has rarely seen.

Building it: pre-recorded audio

Async is the right default for ambient documentation and retrospective work. There's no reason to pay streaming rates for a file that already exists.

import requests, time

BASE = "https://api.assemblyai.com/v2"
HEADERS = {"authorization": "YOUR_API_KEY"}

payload = {
    "audio_url": "https://example.com/encounter-8812.wav",
    "speech_models": ["universal-3-5-pro"],
    "domain": "medical-v1",
    "speaker_labels": True,
    "redact_pii": True,
    "redact_pii_policies": ["person_name", "date_of_birth", "phone_number",
"email_address"],
    "redact_pii_audio": True
}

job = requests.post(f"{BASE}/transcript", json=payload, headers=HEADERS).json()

while (result := requests.get(f"{BASE}/transcript/{job['id']}", headers=HEADERS).json()
)["status"] != "completed": time.sleep(3)
for utt in result["utterances"]:
    print(f"[{utt['speaker']}] {utt['text']}")

Async takes speech_models as a plural array. Streaming takes speech_model, singular. Everyone gets caught by that once.

Building it: live audio

Universal-3.5 Pro Realtime connects at wss://streaming.assemblyai.com/v3/ws.

import websockets; from urllib.parse import urlencode

CONFIG = {
    "speech_model": "universal-3-5-pro",
    "domain": "medical-v1",
    "mode": "balanced",
    "voice_focus": "far-field",
    "speaker_labels": True,
    "prompt": "Primary care follow-up. Patient on lisinopril and metformin."
}

async def run(mic):
    url = "wss://streaming.assemblyai.com/v3/ws?" + urlencode({k: str(v).lower()
if isinstance(v, bool) else v for k, v in CONFIG.items()})
    async with websockets.connect(
        url, additional_headers={"Authorization": "YOUR_API_KEY"}
    ) as ws:
        
        async for chunk in mic:
            await ws.send(chunk)
mode takes min_latency, balanced or max_accu

mode takes min_latency, balanced or max_accuracy. Use min_latency for a conversational agent that has to reply, max_accuracy for an ambient recorder nobody is waiting on, balanced when you're not sure. prompt carries scenario context into the decode — across 20,000 voice agent calls, scenario context cut medical-term entity errors 24% and detailed context cut them 43%. agent_context is a separate parameter that carries the reply your voice agent just spoke. Turn detection defaults to min_turn_silence 128ms and max_turn_silence 1280ms on the balanced preset.

PHI, security and where the audio goes

The facts your security reviewer will ask for:

• AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and operates as a business associate under HIPAA. See the BAA FAQ and the BAA page.
• PHI redaction runs across audio and transcripts both, so the stored recording can be redacted at the source rather than only the text derived from it.
• SOC 2 Type 2 covers platform controls.
• EU data residency is available at api.eu.assemblyai.com. A self-hosted deployment exists for programs that can't send audio outside their own infrastructure.

What it costs

Async on Universal-3.5 Pro is $0.21/hr, and $0.36/hr with Medical Mode. Streaming on Universal-3.5 Pro Realtime is $0.45/hr base, $0.60/hr with Medical Mode. The Medical Mode add-on is $0.15/hr either way. If you're building a conversational agent rather than a transcription pipeline, the Voice Agent API is a flat $4.50/hr and replaces the whole STT plus LLM plus TTS chain with one WebSocket. Current rates live on the pricing page.

Worth comparing that against per-minute medical transcription services, which is where most of these budgets historically sat. The gap is large enough that the constraint stops being cost and starts being what you do with the transcripts.

Where medical speech-to-text is heading

The interesting frontier isn't accuracy on the transcript — that problem is close enough to solved that entity errors are now a rounding difference for most workflows. It's what happens when the model has real context. Contextual prompting cutting missed terms by 31% is an early signal of something bigger: speech recognition that knows which patient it's listening to, which medications are on the chart, and which specialty vocabulary applies, resolving ambiguity the way a human scribe familiar with the practice would. Vendors like Sully AI, Heidi Health, Deepscribe, Knowtex and Magentus Healthcare are already building in that direction. The teams that treat patient context as a first-class input rather than an afterthought are the ones whose transcripts will stop needing review.

Frequently asked questions

Which speech-to-text API is most accurate for medical terminology?

On Missed Entity Rate — the metric that scores drugs, conditions, procedures and dosages rather than every word equally — Universal-3.5 Pro with Medical Mode measures 3.2%, the lowest across benchmarked providers. Methodology is on the benchmarks page.

How does AssemblyAI Medical Mode compare to Deepgram Nova-3 Medical and Amazon Transcribe Medical?

Two differences. Accuracy: 3.2% MER, the lowest across benchmarked providers. Architecture: Medical Mode is one parameter on the model you already call, while both competitors route you to a separate medical model or service. That means you can A/B it on your own audio in an afternoon instead of planning a migration.

How does AssemblyAI handle HIPAA and PHI?

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and acts as a business associate under HIPAA. Supporting that: PHI redaction across audio and transcripts, SOC 2 Type 2, EU data residency, and a self-hosted deployment option. Details are in the BAA FAQ.

Does medical speech-to-text work in real time?

Yes. Universal-3.5 Pro Realtime streams over wss://streaming.assemblyai.com/v3/ws with turn detection defaulting to a 128ms min_turn_silence and a 1280ms max_turn_silence on the balanced preset, and Medical Mode applies to streaming as well as pre-recorded audio. That's fast enough for a conversational healthcare agent, and the same model handles your batch pipeline, so accuracy doesn't fork between the two.

Can it tell the doctor apart from the patient?

Diarization is a request parameter, and Universal-3.5 Pro's is the most accurate we've shipped — optimized for cpWER, which scores whether the right words landed with the right speaker rather than just whether boundaries look tidy. It holds up on short patient answers and overlapping speech, and streaming supports revision across up to 10 speakers.

What languages does medical speech-to-text support?

Medical Mode covers English, Spanish, German and French for both pre-recorded and streaming audio. Separately, the base Universal-3.5 Pro model code-switches natively across 18 languages with no configuration — so a bilingual encounter transcribes correctly even where the Medical Mode entity boost doesn't apply. See the medical transcription use case and the docs for specifics.

‍