AI medical scribe: how to build one that clinicians actually trust

A physician finishes a 14-minute follow-up with a patient who's on metformin, lisinopril, and a recently adjusted dose of levothyroxine. The ambient scribe running on the exam room mic transcribes "levothyroxine" as "levofloxacin." One is a thyroid hormone. The other is a fluoroquinolone antibiotic. The draft note goes into the EHR, the physician is on visit nine of twenty-two, and the error survives the review click.

That single substitution is the whole problem with AI medical scribes in miniature. The hard part isn't generating a SOAP note — a competent LLM does that. The hard part is the speech layer underneath it, because every downstream summary inherits whatever the transcript got wrong. If the drug name is wrong in the transcript, it's wrong in the assessment, wrong in the plan, and wrong in the chart.

So the accuracy metric that matters for a scribe isn't word error rate. It's whether the clinically meaningful tokens survive. AssemblyAI's Medical Mode is built against exactly that: it posts a 3.2% Missed Entity Rate (MER) on medical entities, the lowest MER across benchmarked providers. You can see the full methodology on the benchmarks page.

This post covers what an AI medical scribe is, how the pieces fit together, whether to build or buy, and — if you build — which speech models and parameters to wire up. Code samples included.

What an AI medical scribe actually is

An AI medical scribe listens to a clinical encounter and produces a structured clinical note without a human transcriptionist in the loop. Four traits separate a real scribe from generic dictation software:

Ambient capture, not dictation

Dictation software waits for a clinician to narrate a note. A scribe listens to the natural conversation between doctor and patient — overlapping speech, interruptions, a family member chiming in from the corner of the room, background noise from a hallway. That's a much harder acoustic problem, and it's why far-field performance and speaker separation matter more here than raw single-speaker accuracy.

Medical understanding, not just words

A general transcription model has seen "metformin" in its training data. It has probably not seen "empagliflozin 10 milligrams daily, hold if eGFR drops below thirty." Domain-specialized recognition is what keeps drug names, dosages, conditions, and procedure terms intact.

Structured output

Clinicians don't want a transcript. They want a note in the shape their specialty and their EHR expect — SOAP, H&P, or a specialty template. That's an LLM step layered on top of the transcript, and its ceiling is set by transcript quality.

Chart integration

A note that lives in a separate web app is a second system of record, which is to say a burden. Real scribes write into the EHR through FHIR, HL7, or a vendor API.

How the pipeline works, stage by stage

Strip away the branding and every ambient scribe is the same four stages.

1. Capture. Room mic, phone, or headset picks up the encounter. For ambient use, this is usually a far-field mic several feet from both speakers.

2. Transcribe. Audio goes to a speech-to-text model. This is the stage that determines everything downstream. You can run it after the visit (async) or live during the visit (streaming).

3. Structure. An LLM turns the diarized transcript into a draft note, pulling assessment and plan from the conversation and dropping the small talk.

4. Review and sync. The clinician edits and signs. The note posts to the chart.

Stage 2 is where teams underinvest and then spend a year trying to patch the consequences with prompt engineering. You cannot prompt your way out of a transcript that lost the drug name.

Async or streaming?

Most scribes are async. The visit ends, the audio uploads, the note is drafted in the background, and the clinician reviews it between patients. Universal-3.5 Pro handles this at $0.21/hr, and async gives the model the full recording to work with, which is a genuine accuracy advantage.

Streaming matters when the clinician wants to see the note forming live, or when the scribe is part of a two-way voice experience. Universal-3.5 Pro Realtime runs at $0.45/hr base with turn detection defaulting to min_turn_silence 128ms and max_turn_silence 1280ms on the balanced preset, which is what makes a live experience feel responsive rather than laggy.

My opinion: build async first. It's cheaper, more accurate, and covers the documentation use case completely. Add streaming when you have a product reason, not because live feels more impressive in a demo.

Before you build

If you haven't settled the buy-versus-build question yet, our companion post on how ambient AI scribes work lays out the tradeoffs against commercial platforms like Abridge and Nuance DAX. The rest of this post assumes you're building.

The rule of thumb that matters here: if documentation is your product, the workflow is the thing you're selling, and you can't differentiate on someone else's interface.

The speech layer: what to configure

Here's the async setup for an ambient scribe. Note that async takes speech_models as a list, and Medical Mode is one parameter — there's no separate medical model to switch to.

import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro"],
    domain="medical-v1",
    speaker_labels=True,
    speakers_expected=2,
)

transcript = aai.Transcriber().transcribe(
    "encounter-2026-08-27.wav",
    config=config,
)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

Medical entity accuracy

domain: "medical-v1" is the whole activation. It costs +$0.15/hr on top of the base model, so $0.36/hr combined for async. What it buys you: roughly 20% fewer missed medical entities than the base model without Medical Mode, and 87% fewer entity errors against that same baseline. See the before-and-after transcript tour for what that looks like on real audio.

Doctor/patient separation

A scribe that can't tell who said what will attribute the patient's symptom report to the physician's assessment. Universal-3.5 Pro ships the most accurate diarization AssemblyAI has built, optimized for cpWER rather than DER — meaning it's scored on whether the right words land with the right speaker, not just on segment boundaries. It holds up on short turns ("Mm-hm." "Any chest pain?" "No.") and on overlapping speech, which is most of what an exam room sounds like.

Contextual prompting from the chart

This one is underused and it's the single biggest win on the list. If you have the patient's prior-visit note, feed it to the model as context. In an internal healthcare test, doing exactly that cut missed medical terms by 31%. The patient's medication list, their active problems, their surgeon's name — the model stops guessing at terms it has already been told to expect.

Multilingual encounters

Two separate facts, and they're worth keeping straight. Medical Mode covers English, Spanish, German, and French. The base model code-switches natively across 18 languages with no configuration — useful when a family member interprets mid-visit and the audio flips languages in the same sentence.

Live capture, when you need it

For streaming, the parameter is speech_model, singular:

import websockets

URL = (
    "wss://streaming.assemblyai.com/v3/ws"
    "?sample_rate=16000"
    "&speech_model=universal-3-5-pro"
    "&domain=medical-v1"
    "&speaker_labels=true"
    "&voice_focus=far-field"
)

async def stream(audio_chunks):
    async with websockets.connect(
        URL, additional_headers={"Authorization": "YOUR_API_KEY"}
    ) as ws:
        async for chunk in audio_chunks:
            await ws.send(chunk)
            # Do not await ws.recv() inside the send loop; it stalls the audio stream.
            # Run a separate task that iterates: async for message in ws
            # and prints the transcript when the message type is Turn.

Two streaming settings earn their place in an exam room. voice_focus takes near-field or far-field — set it to far-field for a room mic. And streaming diarization supports revision across up to 10 speakers, so an early speaker guess gets corrected as more audio arrives instead of being locked in. Streaming with Medical Mode runs $0.60/hr combined.

PHI, BAA, and the security work you can't skip

Every recording of a clinical encounter is PHI. That shapes the architecture, not just the contract.

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI, acting as a business associate under HIPAA. The details are on the BAA FAQ and the BAA page. AssemblyAI is SOC 2 Type 2. PHI redaction runs across both audio and transcripts, so you can strip identifiers from the artifact and the source recording. For teams with data residency requirements, there's EU residency via api.eu.assemblyai.com and a self-hosted deployment option.

On your side of the line: get consent on the record before the mic goes live, encrypt at rest and in transit, scope access by role, set a retention policy and actually enforce it, and log every read of a recording. None of this is exotic. All of it gets skipped under launch pressure.

What the good scribes get right

Having looked at a lot of these pipelines, the ones clinicians keep using share three habits.

They treat transcript quality as the product metric, not note quality. Note quality is downstream and it lies to you — a fluent LLM will write a beautiful paragraph around a hallucinated drug name.

They evaluate on their own audio. Benchmarks are directional. Your mics, your rooms, your specialty's vocabulary, your patients' accents. Run 50 real encounters through the playground before you commit to anything.

They keep the clinician in the loop and make review fast rather than optional. The scribe drafts; the physician signs. Design for a 30-second review, not a zero-second one.

Who's building on this

The ambient documentation space has consolidated around a handful of serious teams. Sully AI, Heidi Health, Deepscribe, Knowtex, and Magentus Healthcare are all working in this territory, and Commure runs an ambient product built on AssemblyAI models.

"We've integrated the newest models from AssemblyAI for pre-recorded audio ASR in our ambient product, and it's been excellent. We're now exploring Universal-3.5 Pro for async and realtime speech-to-text capabilities for new use cases. What's been just as important is the reliability of the platform itself—both technically and in terms of partnership." — Gautam Pradeep, Tech Lead, Commure

The pattern in that comment is worth noting: the model matters, and so does whether the thing stays up during clinic hours. A scribe that fails at 10 a.m. on a Tuesday costs a physician their entire afternoon.

Where scribes go next

The interesting frontier isn't better notes. It's what happens when the transcript stops being a document and starts being a live input to the visit itself.

Once transcription is accurate enough and fast enough — 128ms min_turn_silence turn detection, entity-level reliability, speaker attribution you can trust — the scribe can do things during the encounter rather than after it. Flag a drug interaction the moment both medications are mentioned. Surface the guideline that applies to the symptom the patient just described. Notice that a screening question went unasked. Pre-populate the order set from the plan as the physician says it out loud.

That shifts the scribe from a documentation tool to a clinical safety layer, and it raises the bar on the speech stack considerably, because a system that intervenes has to be right. Teams building now should be asking whether their transcript is good enough to act on, not just good enough to summarize — because that's the product everyone will be shipping in eighteen months, and the accuracy floor is being set today.

Frequently asked questions

What is the best speech-to-text API for building an AI medical scribe?

For medical entity accuracy, AssemblyAI's Universal-3.5 Pro with Medical Mode is the strongest option on published benchmarks — 3.2% Missed Entity Rate, the lowest across benchmarked providers. It also brings cpWER-optimized diarization for doctor/patient separation and contextual prompting so you can feed the model prior-visit notes. Full numbers are on the benchmarks page.

How do I build an AI medical scribe like Nuance DAX or Abridge?

Four components: audio capture, speech-to-text with a medical domain setting, an LLM to structure the note, and an EHR write path. The speech layer is the one that determines whether the rest works, so start there — async transcription with speech_models: ["universal-3-5-pro"] and domain: "medical-v1", speaker labels on. The note-structuring prompt is comparatively easy work. See the medical transcription use case for reference architecture.

How does AssemblyAI handle HIPAA and PHI?

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI, and operates as a business associate under HIPAA. Alongside the BAA, AssemblyAI is SOC 2 Type 2 and supports PHI redaction across both audio and transcripts, EU data residency, and self-hosted deployment. Details are on the BAA FAQ.

How much does it cost to transcribe clinical audio?

Universal-3.5 Pro is $0.21/hr for pre-recorded audio. Medical Mode adds $0.15/hr, so $0.36/hr combined. Streaming is $0.45/hr base, or $0.60/hr with Medical Mode. Per-hour pricing means a scribe's cost scales with audio volume rather than clinician headcount, which usually works out favorably against per-seat scribe platforms. Current rates are on pricing.

Can an AI medical scribe handle a bilingual visit?

Yes, with a distinction worth understanding. Medical Mode covers English, Spanish, German, and French. Separately, the base model code-switches natively across 18 languages with no configuration, so audio that shifts language mid-sentence — common when a family member interprets — is transcribed correctly even outside the four Medical Mode languages.

What's the best speech-to-text for far-field ambient clinical environments?

Room mics several feet from the speakers are the hardest case in clinical audio. For streaming, set voice_focus=far-field to tune for distance, and rely on diarization with revision (up to 10 speakers) so speaker assignments get corrected as context accumulates rather than locked in from the first second. For async, the model has the whole recording available, which is why async generally outperforms streaming on the same far-field audio. Try both in the playground and read the docs for parameter details.

‍