Build a real-time medical transcription analysis app with AssemblyAI and LLM Gateway

A cardiologist says "we'll start her on metoprolol succinate, 25 milligrams daily, and titrate up." A general-purpose speech model hears "metoprolol succinate" maybe. Or it hears "metoprolol sucrose." Or it drops the second word entirely and hands your downstream LLM a drug name that doesn't exist. That's the whole problem with AI medical transcription in one sentence: the words that matter most are the words the model has seen least.

We built Medical Mode to close that gap, and the number we hold ourselves to is a 3.2% Missed Entity Rate — the lowest MER across the providers we've benchmarked. You can see the methodology on our benchmarks page.

This post is the build guide. What clinical audio actually demands, how to turn Medical Mode on, when to reach for async versus streaming, how to separate the clinician from the patient, and what your security reviewer is going to ask you. Code included, Python throughout.

What AI medical transcription has to get right

Medical transcription used to mean a human typist working from a dictaphone. AI medical transcription means a model doing that work in seconds, either from a recorded file or live during the encounter. The pipeline is short: capture audio, transcribe it, structure it, write it somewhere a clinician will read it.

The hard part isn't the transcription step in the abstract. It's that clinical audio stacks four difficulties at once:

Rare, high-stakes vocabulary

Drug names, conditions, procedures, dosages, anatomical terms. These are statistically rare in general training data and catastrophic to get wrong. "Hydralazine" and "hydroxyzine" are one phoneme apart and treat completely different things.

Multiple speakers, unevenly balanced

A clinician who talks in complete paragraphs and a patient who answers in three words. Caregivers interjecting. A resident in the corner. Getting attribution right matters because "the patient reports chest pain" and "the doctor reports chest pain" are not the same clinical record.

Real rooms

Exam rooms have HVAC noise, hallway traffic, rolling equipment, and microphones that sit six feet from the speaker rather than an inch from their mouth.

Language that shifts mid-sentence

A Spanish-speaking patient and an English-speaking clinician often produce a single conversation containing both, sometimes inside one sentence.

Any model can transcribe a clean dictation. The evaluation that matters is what survives a real encounter.

Why general-purpose ASR falls down here

Generic models are trained to be broadly right. That objective works against you in medicine, because the model's prior pushes ambiguous audio toward the common word. "Sitagliptin" becomes "sit again." Teams usually respond with a keyword list, then discover they're maintaining a formulary by hand and still missing the terms nobody thought to add.

Two things fix this properly. First, a model that has been trained on clinical language rather than patched with a word list. Second, the ability to give the model context about this specific encounter at request time. Universal-3.5 Pro does both.

Medical Mode: one parameter, measurable difference

Medical Mode isn't a separate model you have to migrate to. It's a domain flag on the request: domain: "medical-v1". Same model ID, same endpoint, same response shape.

Here's what turning it on buys you against the same base model with it off:

• 3.2% Missed Entity Rate, absolute — our headline accuracy figure for clinical entities.
• ~20% fewer missed medical entities than the base model without Medical Mode.
• 87% fewer entity errors than the base model without Medical Mode.

And against other providers on medical entity capture:

Medical Mode covers English, Spanish, German, and French across both pre-recorded and streaming. That's a separate thing from the base model's language range — more on that below, because people conflate them constantly.

Async or streaming: pick by workflow, not by preference

Both flagships use the same model ID, universal-3-5-pro. What differs is the transport and the price.

My blunt advice: if a clinician doesn't need to see words appear on screen during the encounter, use async. It's cheaper, it gets the full-file context, and you avoid an entire class of connection-management bugs. Reach for streaming when the interface is live. Full pricing sits on the pricing page.

Build it: the async pipeline

Recorded encounter in, structured note out. This is the version most teams should start with.

import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro"],
    domain="medical-v1",
    speaker_labels=True,
    speakers_expected=2,
    punctuate=True,
    format_text=True,
)

transcriber = aai.Transcriber(config=config)
transcript = transcriber.transcribe("./encounter-2026-08-27.wav")

if transcript.status == "error":
    raise RuntimeError(transcript.error)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

Four things worth noticing. speech_models is plural and takes an array — this trips people up moving from older code. domain is the only Medical Mode switch. speaker_labels gives you the clinician/patient split. And you check status before touching the transcript, because a failed job with an unread error is how bad notes reach a chart.

Structuring the note

Transcription gets you words. A SOAP note needs structure, which is an LLM job. Pass the diarized transcript through the LLM Gateway with an explicit schema and require the model to cite the transcript span for anything clinical. Do not let it infer. If a medication dose isn't in the transcript, the correct output is a flag for human review, not a plausible number.

Contextual prompting is the cheapest accuracy win available

Here's the finding that surprised us most. In an internal healthcare test, feeding the model a patient's prior-visit note as context cut missed medical terms by 31%. Not a new model. Not a fine-tune. Just handing the model the document that already lists this patient's medications, conditions, and specialists.

If your app has access to the last note, use it. It's the single highest-value line of code in the whole pipeline.

Build it: the streaming pipeline

For live scribing, connect to wss://streaming.assemblyai.com/v3/ws. Note the singular speech_model here.

import assemblyai as aai
from assemblyai.streaming.v3 import (
    StreamingClient,
    StreamingClientOptions,
    StreamingParameters,
    TurnEvent,
)

def on_turn(client, event: TurnEvent):
    if event.end_of_turn:
        print(f"[final] {event.transcript}")

client = StreamingClient(
    StreamingClientOptions(api_key="YOUR_API_KEY")
)
client.on(TurnEvent, on_turn)

client.connect(
    StreamingParameters(
        speech_model="universal-3-5-pro",
        domain="medical-v1",
        sample_rate=16000,
        
        speaker_labels=True,
        voice_focus="far-field",
        mode="balanced",
    )
)

client.stream(mic_stream())  # mic_stream(): your own pyaudio generator yielding 16 kHz PCM chunks

Three streaming parameters earn their place in clinical settings:

voice_focus

Set near-field for a headset or a phone held to the face, far-field for a microphone on the desk or wall. Ambient scribing is almost always far-field, and it's the single setting most teams forget to change.

mode

Choose min_latency, balanced, or max_accuracy. Ambient documentation should run max_accuracy or balanced — nobody is waiting on a 200ms round trip to read their own note. Save min_latency for conversational agents. For clinical audio, raise min_turn_silence to 800ms and max_turn_silence to 3600ms so a clinician pausing mid-sentence does not fragment the turn.

agent_context

If you're building something conversational — an intake bot, a symptom triage line — agent_context cut word error rate 10.2% across 20,000 voice agent files in our testing, with medical entity errors down 9.4%. Details in the Realtime launch post.

If you're building a full conversational agent rather than a transcription feed, the Voice Agent API collapses STT, the LLM, and TTS into one WebSocket at a flat $4.50/hr. Fewer moving parts, one bill.

Getting the clinician and the patient apart

Diarization is where a lot of medical transcription projects quietly fail. A transcript with the right words attached to the wrong speaker produces a chart note that's actively wrong.

Universal-3.5 Pro ships the most accurate diarization we've built. Two design choices matter for clinical audio. It's optimized for cpWER rather than DER, which means it's scored on whether the right words end up attributed to the right speaker — the thing you actually care about. And it captures short turns and overlapped speech, which is most of what a patient contributes to an encounter. "Mm-hmm," "since Tuesday," "the left one" — these get dropped by diarizers tuned for long, clean, alternating turns.

Streaming diarization supports revision, so an early attribution can be corrected as more audio arrives, and handles up to 10 speakers. That's enough for a family present in the room plus a care team.

Multilingual clinical audio: two facts, kept separate

People blur these together and then build the wrong thing, so let me be precise.

The base model code-switches natively across 18 languages with no configuration. A conversation that moves between English and Spanish mid-sentence transcribes correctly without you detecting language or routing requests.

Medical Mode itself covers English, Spanish, German, and French. So a Portuguese consult still transcribes well on the base model — it just doesn't get the clinical entity boost.

Practical read: if your patient population is largely EN/ES/DE/FR, you get both benefits. If it's broader, run Medical Mode where it applies and rely on native code-switching elsewhere. Don't build a language router; you don't need one.

PHI, BAAs, and the security review you're going to face

Compliance won't win you the deal, but the wrong answer here loses it. Get the facts straight before your first hospital call.

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI, which makes us a business associate under HIPAA. The request path and the terms are documented at can you sign a BAA and legal/business-associate-agreement. Send those links to your compliance team directly — it's faster than paraphrasing.

Beyond the BAA: PHI redaction runs across both audio and transcripts, so you can store a de-identified transcript alongside a redacted audio file rather than choosing between them. We hold SOC 2 Type 2. For teams with data residency or isolation requirements, self-hosted deployment and EU residency via api.eu.assemblyai.com are both available.

One thing no vendor can hand you: consent. Recording an encounter has state-level requirements, and your product needs a documented consent step and an audit trail. Build it early.

Where this goes next

The accuracy conversation in medical transcription is close to settled — a 3.2% MER on clinical entities is good enough that word errors stop being the bottleneck. What replaces it is context. The 31% reduction in missed medical terms from a single prior-visit note points at a model of documentation where the system already knows the patient before the visit starts: their medications, their specialists, the abbreviation their clinic uses for a procedure.

That's a different product than transcription. It's a documentation layer that gets more accurate the longer a patient stays with a practice, and the teams wiring their record system into the transcription request today are the ones who'll have it first. The API surface for it already exists. Most people just haven't plugged it in.

If you want the adjacent reads: building an AI medical scribe end to end, medical voice recognition, and the healthcare solutions overview. Teams shipping on this stack today include Sully AI, Heidi Health, Deepscribe, Knowtex, and Magentus Healthcare.

Frequently asked questions

What is the best speech-to-text API for medical transcription?

On medical entity accuracy, Universal-3.5 Pro with Medical Mode posts a 3.2% Missed Entity Rate — the lowest MER across the providers we've benchmarked. Methodology is on the benchmarks page. Beyond raw accuracy, the practical criteria are whether medical support is a parameter or a separate model to migrate to, whether the vendor will sign a BAA, and whether streaming and async share one model ID.

What is AssemblyAI's Medical Mode?

It's a clinical domain setting on Universal-3.5 Pro and Universal-3.5 Pro Realtime, activated with domain: "medical-v1". No model switch, no separate endpoint. It costs an extra $0.15/hr on top of the base rate — $0.36/hr combined for async, $0.60/hr for streaming — and delivers roughly 20% fewer missed medical entities and 87% fewer entity errors than the same model without it. It covers English, Spanish, German, and French.

How does AssemblyAI handle HIPAA and PHI?

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI, which makes us a business associate under HIPAA. On top of the BAA you get PHI redaction across audio and transcripts, SOC 2 Type 2, and options for self-hosted deployment or EU data residency. The details are at can you sign a BAA.

Does AssemblyAI automatically redact patient PII from medical transcripts?

Yes — PHI and PII redaction runs across both the transcript and the audio, so you can retain a de-identified transcript and a redacted recording rather than picking one. Configure it per request in the docs, and set your redaction policy before you start storing anything.

How does AssemblyAI capture medical jargon and terminology accurately?

Two mechanisms. Medical Mode trains the model on clinical language rather than bolting a keyword list onto a general model. Then contextual prompting lets you pass encounter-specific context at request time — in an internal healthcare test, feeding a patient's prior-visit note cut missed medical terms by 31%. Together they handle both the general clinical vocabulary and the terms specific to one patient.

Can AI medical transcription handle multiple languages in one conversation?

Yes. The base model code-switches natively across 18 languages with no configuration, so an encounter that moves between English and Spanish mid-sentence transcribes correctly. Medical Mode's clinical entity boost covers English, Spanish, German, and French — those are two separate capabilities, and it's worth knowing which one your patient population needs.

‍