What is an Ambient AI Scribe and how do they work?

Every ambient scribe demo runs on the same audio: two clear speakers, a cooperative script, a quiet room, and a finished note in about nine seconds. Then you point the same product at a recording from your own clinic — a microphone eight feet away on a counter, an HVAC unit cycling, a patient's daughter interjecting from the corner — and the note that comes back describes a slightly different visit than the one that happened.

The whole gap between those two runs sits in one place. Generating a SOAP note is the easy part — a competent language model does that reliably. The hard part is the speech layer underneath, because every summary inherits whatever the transcript got wrong. If the drug name is wrong in the transcript, it's wrong in the assessment, wrong in the plan, and wrong in the chart, and no amount of prompt engineering recovers it.

Which means the accuracy metric that matters for a scribe isn't word error rate. It's whether the clinically meaningful tokens survive. Medical Mode on Universal-3.5 Pro posts a 3.2% Missed Entity Rate — the lowest across the providers on our benchmark, where Deepgram Nova-3 Medical sits near 8.7% and AWS Transcribe Medical near 24.4%.

This post covers what an ambient scribe actually is, how the pipeline fits together, where the acoustic problems hide, and how to judge a vendor's speech layer before you sign anything. If you have already decided to build rather than buy, our companion post on building an AI medical scribe walks through the parameters and the code.

What separates a scribe from dictation software

An ambient AI scribe listens to a clinical encounter as it naturally happens and produces a structured note without a human transcriptionist. Four questions separate a real one from dictation software with better marketing, and they are the first four to put to any vendor.

Does it listen, or does it wait to be dictated to?

Dictation waits for a clinician to narrate a note in a controlled voice. A scribe listens to a real conversation: overlapping speech, interruptions, a family member chiming in from the corner, a hallway PA system, an exam table creaking. That's a substantially harder acoustic problem, and it's why far-field performance and speaker separation matter more here than single-speaker accuracy ever did.

Is the vocabulary clinical, or just fluent?

A general model has seen "metformin" in training. It has almost certainly not seen "empagliflozin ten milligrams daily, hold if eGFR drops below thirty." Domain-adapted recognition is what keeps drug names, doses, conditions, and procedure terms intact.

Does the output match your note format?

Clinicians don't want a transcript. They want a note in the shape their specialty and EHR expect — SOAP, H&P, or a specialty template. That's a language model step on top of the transcript, and its ceiling is set by transcript quality.

Does the note reach the chart?

A note that lives in a separate web app is a second system of record, which is to say a new burden. Real scribes write into the EHR through FHIR, HL7, or a vendor API, and they carry the encounter metadata needed to file the note against the right visit.

How the pipeline works, stage by stage

Strip the branding off any ambient scribe and you find the same four stages.

1. Capture

A room microphone, phone, or headset picks up the encounter. For genuinely ambient use this is a far-field microphone several feet from both speakers, which is where most quality problems originate. Capture at 16 kHz or better and avoid aggressive lossy compression before upload — the compression artifacts you can't hear are exactly the ones that eat consonants in drug names.

2. Transcribe

Audio goes to a speech-to-text model. This stage determines everything downstream. You can run it after the visit (async) or live during it (streaming). Most scribes should run async: the model sees the whole recording, accuracy is higher, and it costs less — $0.21/hr for Universal-3.5 Pro plus $0.15/hr for Medical Mode, so $0.36/hr combined.

3. Structure

A language model turns the diarized transcript into a draft note, pulling assessment and plan out of the conversation and dropping the small talk about the patient's weekend. This is also where you decide what the model is allowed to infer versus only report — a question with real safety weight.

4. Review and sync

The clinician edits and signs, and the note posts to the chart. Design this step assuming it gets three seconds of attention, because on visit nine of twenty-two it will.

Stage 2 is where teams underinvest and then spend a year patching the consequences with prompts. You can't prompt your way out of a transcript that lost the medication.

Why the speech layer is the ceiling

Word error rate counts every token equally, so "the" and "amiodarone" carry the same weight. In a clinical transcript, function words dominate — which means a model can post an impressive WER while missing the two words that change patient care.

Missed Entity Rate scores only what matters: drugs, dosages, conditions, procedures, anatomy, negations. Enabling Medical Mode on Universal-3.5 Pro delivers roughly 20% fewer missed medical entities than the base model and 87% fewer entity errors, at a 3.2% absolute MER. Full methodology is on our benchmarks page, and we go deeper on how these numbers are built in our post on AI medical transcription accuracy.

The configuration choices that decide scribe quality

Far-field handling

An ambient microphone on a desk is three to eight feet from the people talking, picking up room reflections and equipment noise along the way. Universal-3.5 Pro Realtime exposes voice_focus with near_field and far_field profiles. Getting this wrong costs measurable accuracy on the quieter speaker in the room, which in a clinical encounter is usually the patient — the person supplying the history.

Diarization that survives real conversation

"I stopped taking the lisinopril" is a different clinical fact depending on who said it. Most diarization systems are tuned on diarization error rate, which rewards getting the long monologues right and quietly tolerates dropping short interjections. Universal-3.5 Pro's diarization is optimized for cpWER instead — it only improves if the words land under the correct speaker. It's the most accurate diarization we've shipped, and the gains concentrate on short turns and overlapped speech, which is exactly what an encounter is made of. Streaming supports up to 10 speakers with revision as more audio arrives, which matters for family-present visits and multi-clinician rounds.

Contextual prompting from the chart

This is the single biggest accuracy win most scribe teams haven't taken. Medical Mode gives you general clinical language; contextual prompting gives you this patient. Passing the patient's prior-visit note along with the audio cut missed medical terms by 31% in an internal healthcare test, because the model now has the actual medication list, the actual diagnoses, and the actual provider names before it starts decoding.

Your scribe already has the chart open when recording starts. That context is free, it never goes stale the way a custom vocabulary file does, and it scales across specialties without per-specialty engineering.

Multilingual encounters

Universal-3.5 Pro code-switches natively across 18 languages with no configuration, so a patient moving between English and Spanish mid-sentence doesn't need a language hint or a second request. Medical Mode's clinical adaptation covers English, Spanish, German, and French. Keep those two facts separate when you plan coverage — the base model's language range is wider than Medical Mode's.

Buy a platform, or build on an API?

Commercial scribe platforms — Abridge, Nuance DAX, and others — sell a finished product priced per clinician per month. Building means you own the pipeline and pay per hour of audio instead.

The honest split: a clinic buying documentation relief for twelve physicians should buy. A health tech company whose customers are clinicians should build, because the workflow is what you're selling and you can't differentiate on somebody else's interface.

How much integration work this actually is

For the async path — the right default for a scribe — this is the whole transcription call:

import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro"],
    domain="medical-v1",
    speaker_labels=True,
    redact_pii=True,
    redact_pii_audio=True, redact_pii_policies=[aai.PIIRedactionPolicy.person_name, aai.PIIRedactionPolicy.date_of_birth, aai.PIIRedactionPolicy.phone_number],
)

transcript = aai.Transcriber(config=config).transcribe("encounter.wav")

encounter = [
    {"speaker": u.speaker, "text": u.text, "start": u.start}
    for u in transcript.utterances
]
# hand `encounter` to your note-generation step

Note the plural speech_models — async takes a list. If you're building a live-forming note instead, streaming uses the singular speech_model against the v3 WebSocket endpoint:

import json
import websockets

WS_URL = "wss://streaming.assemblyai.com/v3/ws"

config = {
    "speech_model": "universal-3-5-pro",
    "domain": "medical-v1",
    "sample_rate": 16000,
    "voice_focus": "far_field",
    "mode": "max_accuracy",
    "speaker_labels": True,
}

async def stream(chunks):
    async with websockets.connect(
        WS_URL, additional_headers={"Authorization": "YOUR_API_KEY"}
    ) as ws:
        await ws.send(json.dumps({"type": "configure", **config}))
        async for chunk in chunks:
            await ws.send(chunk)
            msg = json.loads(await ws.recv())
            if msg.get("type") == "Turn":
                print(msg["transcript"])

Streaming runs $0.45/hr base and $0.60/hr with Medical Mode. The mode field trades latency for accuracy across min_latency, balanced, and max_accuracy — for a scribe, nobody's waiting on a response, so choose accuracy. Full reference is in the docs.

Consent, PHI, and the parts legal will ask about

Ambient recording in a clinical setting needs patient notification and, depending on jurisdiction, explicit consent. Build the consent state into the recording flow rather than bolting it on — the scribe should refuse to start if consent isn't recorded, and the audit trail should show who consented and when.

On the vendor side: AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and operates as a business associate under HIPAA. PHI redaction is available across both audio and transcripts, so you can strip identifiers from the artifacts you retain. The platform is SOC 2 Type 2 audited, and self-hosted deployment plus EU data residency are available where residency rules apply. Details are in the BAA FAQ and the BAA terms.

Decide your audio retention policy before launch, not after your first security review. Most teams keep audio only long enough to support clinician review of the draft note, then delete it.

What teams building this actually report

Several ambient documentation products run on AssemblyAI's speech-to-text, including Commure, Sully AI, Heidi Health, Deepscribe, Knowtex, and Magentus Healthcare.

"We've integrated the newest models from AssemblyAI for pre-recorded audio ASR in our ambient product, and it's been excellent. We're now exploring Universal-3.5 Pro for async and realtime speech-to-text capabilities for new use cases. What's been just as important is the reliability of the platform itself—both technically and in terms of partnership." — Gautam Pradeep, Tech Lead, Commure

That second point is underrated. Scribe accuracy gets all the attention in evaluations, and then production reality turns out to be about uptime, latency consistency, and whether your provider ships improvements you get for free.

Where ambient scribing goes next

The current generation of scribes treats the encounter as an isolated audio file. The next generation won't. Contextual prompting already shows what happens when the model reads the chart before it listens — a 31% reduction in missed medical terms from information the health system already had in a database. Extend that and the scribe stops being a transcription-plus-summarization pipeline and becomes a system that knows what this patient is on, what was planned last visit, and what the referral said, before the first word is spoken. The scribes that clinicians end up trusting won't be the ones with the best note templates. They'll be the ones wired most deeply into the record.

Frequently asked questions

What is the best API for building an ambient AI scribe?

You want the lowest entity-level error rate you can get, robust far-field diarization, and the ability to pass chart context with the request. Universal-3.5 Pro with Medical Mode covers all three: 3.2% Missed Entity Rate, cpWER-optimized diarization built for short turns and overlap, and contextual prompting that cut missed medical terms 31% in an internal test. See medical transcription use cases for how it fits together.

How do I evaluate an ambient AI scribe vendor?

Ask for Missed Entity Rate on drugs, conditions and procedures rather than word error rate, and ask what audio it was measured on. Then run your own encounters through it before you commit, because your microphones, your rooms and your specialty's vocabulary are the only benchmark that binds. Two further questions separate serious vendors quickly: which diarization metric do you optimize, and can I pass the patient's prior-visit note as context?

Can an ambient scribe tell the doctor and patient apart?

Yes, and it's essential — "I stopped taking the lisinopril" means different things from different speakers. Universal-3.5 Pro's diarization is optimized for cpWER rather than diarization error rate, so it's measured on whether words land under the right speaker. Streaming supports up to 10 speakers with revision, which handles family-present visits and rounds.

Do patients have to consent to an ambient AI scribe?

Notification is standard practice and explicit consent is required in many jurisdictions, so treat consent as a gate on starting the recording rather than a disclosure you make afterward. Keep an audit trail of who consented and when, and define your audio retention window up front — PHI redaction across audio and transcripts helps limit what you're holding.

What's the best speech-to-text for far-field ambient clinical environments?

Look for an explicit far-field mode rather than a general-purpose model you hope handles distance. Universal-3.5 Pro Realtime's voice_focus setting has a far_field profile for desk and ceiling microphones, and the accuracy difference between the right and wrong setting is real on the quieter speaker in the room. For post-visit processing, async with max_accuracy is stronger still because the model sees the full recording.

How does AssemblyAI handle HIPAA and PHI?

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and acts as a business associate under HIPAA. Beyond the BAA: PHI redaction across audio and transcripts, SOC 2 Type 2 audited infrastructure, and self-hosted or EU-resident deployment where required. See the BAA FAQ to request one.