What is an ambient AI scribe? How it works, from audio to SOAP note

An ambient AI scribe listens to a clinical encounter, transcribes it, and drafts a structured note, usually a SOAP note, for the clinician to review and sign. Teams building one on AssemblyAI run Universal-3.5 Pro with Medical Mode for the transcript and an LLM through LLM Gateway for the note.

Every ambient scribe demo runs on the same audio: two clear speakers, a cooperative script, a quiet room, and a finished note in about nine seconds. Then you point the same product at a recording from your own clinic. There's a microphone eight feet away on a counter, an HVAC unit cycling, a patient's daughter interjecting from the corner. The note that comes back describes a slightly different visit than the one that happened.

The whole gap between those two runs sits in one place. Generating a SOAP note is the easy part, and a competent language model does that reliably. The hard part is the speech layer underneath, because every summary inherits whatever the transcript got wrong. If the drug name is wrong in the transcript, it's wrong in the assessment, wrong in the plan, and wrong in the chart, and no amount of prompt engineering recovers it.

That means the accuracy metric that matters for a scribe isn't word error rate. It's whether the clinically meaningful tokens survive. Medical Mode on Universal-3.5 Pro posts a 3.2% Missed Entity Rate (MER), with about 20% fewer missed medical entities than the same model without it.

This post covers what an ambient scribe actually is, how the pipeline fits together, how the transcript becomes a SOAP note, where the acoustic problems hide, and how to judge a vendor's speech layer before you sign anything. If you've already decided to build rather than buy, our companion post on how to build an AI medical scribe walks through the live architecture, the parameters and the code.

What separates a scribe from dictation software

An ambient AI scribe listens to a clinical encounter as it naturally happens and produces a structured note without a human transcriptionist. Four questions separate a real one from dictation software with better marketing, and they're the first four to put to any vendor.

Does it listen, or does it wait to be dictated to?

Dictation waits for a clinician to narrate a note in a controlled voice. A scribe listens to a real conversation: overlapping speech, interruptions, a family member chiming in from the corner, a hallway PA system, an exam table creaking. That's a much harder acoustic problem. It's why far-field performance and speaker separation matter more here than single-speaker accuracy ever did.

Is the vocabulary clinical, or just fluent?

A general model has seen "metformin" in training. It has almost certainly not seen "empagliflozin ten milligrams daily, hold if eGFR drops below thirty." Domain-adapted recognition is what keeps drug names, doses, conditions, and procedure terms intact.

Does the output match your note format?

Clinicians don't want a transcript. They want a note in the shape their specialty and EHR expect: SOAP, H&P, or a specialty template. That's a language model step on top of the transcript, and transcript quality sets its ceiling.

Does the note reach the chart?

A note that lives in a separate web app is a second system of record, which means a new burden. Real scribes write into the EHR through FHIR, HL7, or a vendor API, and they carry the encounter metadata needed to file the note against the right visit.

How the pipeline works, stage by stage

Strip the branding off any ambient scribe and you find the same four stages.

1. Capture

A room microphone, phone, or headset picks up the encounter. For genuinely ambient use, this is a far-field microphone several feet from both speakers, which is where most quality problems start. Capture at 16 kHz or better and avoid aggressive lossy compression before upload. The compression artifacts you can't hear are exactly the ones that eat consonants in drug names.

2. Transcribe

Audio goes to a speech-to-text model. This stage determines everything downstream. You can run it after the visit (async) or live during it (streaming). Most scribes should run async: the model sees the whole recording, accuracy is higher, and it costs less. That's $0.21/hr for Universal-3.5 Pro plus $0.15/hr for Medical Mode, so $0.36/hr combined.

3. Structure

A language model turns the diarized transcript into a draft note. It pulls the assessment and plan out of the conversation and drops the small talk about the patient's weekend. This is also where you decide what the model is allowed to infer versus only report, which is a question with real safety weight. The next section covers it in detail.

4. Review and sync

The clinician edits and signs, and the note posts to the chart. Design this step assuming it gets three seconds of attention, because on visit nine of twenty-two it will.

Stage 2 is where teams underinvest, then spend a year patching the consequences with prompts. You can't prompt your way out of a transcript that lost the medication.

From transcript to SOAP note: what the LLM step does

A SOAP note has four sections, and each one draws on a different part of the conversation.

• Subjective: what the patient reports. Symptoms, history, current medications, and how they've been taking them, in the patient's own account.
• Objective: what was observed or measured. Vitals read aloud, exam findings the clinician narrates, results discussed during the visit.
• Assessment: the clinician's diagnosis or clinical impression, as the clinician stated it.
• Plan: orders, prescriptions with dose and frequency, referrals, and follow-up.

The LLM's job is to sort the transcript into those buckets, not to practice medicine. Three rules make that safe.

Ground every line in a transcript span. Each sentence in the draft should point back to a timestamped, speaker-attributed piece of the transcript. Speaker labels decide whether "I stopped taking the lisinopril" lands in Subjective as the patient's report or reads as a clinician instruction. Timestamps let a reviewer jump to the audio behind any line in one click.

Report, don't infer. The Assessment and Plan should hold only what the clinician actually said. If no diagnosis was stated, the section should say "Not documented in this encounter", not offer a plausible guess. An empty section is correct output. An invented one is a patient-safety problem.

Treat entities as load-bearing. A drug name, a dose, or a laterality the transcript got wrong doesn't stay in one place. It flows from Subjective into Assessment and then into the Plan and the prescription, which is how a misheard drug name propagates into the chart. The note model can't detect it, because "hydralazine" and "hydroxyzine" are both valid drugs. This is why the speech layer, covered next, sets the ceiling for the whole system.

For a working reference, the AssemblyAI docs include an end-to-end medical scribe example. It transcribes with Medical Mode and speaker labels, labels speakers as Provider and Patient, then generates the SOAP note through LLM Gateway with the same API key. For a live version that drafts the note during the visit and finalizes it afterwards, see the build post linked above.

Why the speech layer is the ceiling

Word error rate counts every token equally, so "the" and "amiodarone" carry the same weight. Function words dominate a clinical transcript, so a model can post an impressive WER while missing the two words that change patient care.

Missed Entity Rate scores only what matters: drugs, dosages, conditions, procedures, anatomy, negations. Here's what turning on Medical Mode does on Universal-3.5 Pro.

Both relative figures are correct because they measure different things. Our before-and-after tour shows what 20% fewer missed medical entities looks like on real transcripts. Activation is one parameter, domain: "medical-v1", at $0.15/hr on top of the base model.

When you evaluate any vendor, ask for entity-level numbers on audio that looks like yours, then run your own encounters through it. Our guide to choosing a speech-to-text API for ambient scribes goes through that evaluation step by step.

The configuration choices that decide scribe quality

Far-field handling

An ambient microphone on a desk is three to eight feet from the people talking, and it picks up room reflections and equipment noise along the way. Universal-3.6 Pro Realtime exposes voice_focus with near-field and far-field profiles (+$0.10/hr), plus an optional voice_focus_threshold that defaults to 0.7. Getting this wrong costs measurable accuracy on the quieter speaker in the room. In a clinical encounter that's usually the patient, the person supplying the history.

Diarization that survives real conversation

"I stopped taking the lisinopril" is a different clinical fact depending on who said it. Most speaker diarization systems are tuned on diarization error rate, which rewards getting the long monologues right and quietly tolerates dropping short interjections. Universal-3.5 Pro's diarization is optimized for cpWER instead, so it only improves if the words land under the correct speaker. It's the most accurate diarization we've shipped, and the gains concentrate on short turns and overlapped speech, which is what an encounter is made of. Streaming supports up to 10 speakers and revises labels as more audio arrives. That matters for family-present visits and multi-clinician rounds.

Contextual prompting from the chart

This is the single biggest accuracy win most scribe teams haven't taken. Medical Mode tunes the model for clinical vocabulary; contextual prompting tells it about this specific encounter. Feeding the patient's prior-visit note along with the audio cut missed medical terms by 31% in internal healthcare testing. The model now has the actual medication list, diagnoses, and provider names before it starts decoding.

There's a public benchmark too. On 20,000 real voice-agent calls, detailed context cut medical-term entity errors by 43% and scenario-level context by 24%, according to the prompting and keyterms docs. Your scribe already has the chart open when recording starts. That context is free, it doesn't go stale the way a custom vocabulary file does, and it scales across specialties without per-specialty engineering. Medical Mode and contextual prompting together beat either one alone.

Multilingual encounters

Universal-3.5 Pro code-switches natively across 18 languages with no configuration, so a patient moving between English and Spanish mid-sentence doesn't need a language hint or a second request. Medical Mode's clinical adaptation covers English, Spanish, German, and French, on both pre-recorded and streaming audio. Keep those two facts separate when you plan coverage, because the base model's language range is wider than Medical Mode's. If you send Medical Mode with an unsupported language, the API skips Medical Mode, returns a warning, and doesn't charge for it. Our post on multilingual clinical accuracy covers the Spanish, German, and French results.

Ambient scribes like Nuance DAX and Abridge: buy one, or build on an API?

Commercial scribe platforms such as Nuance DAX and Abridge sell a finished product, priced per clinician per month. Under the interface, they run the same pipeline described above: audio capture, a speech-to-text layer, a language model that drafts the note, and an EHR write-back. Building means you own that pipeline and pay per hour of audio instead.

The honest split: a clinic buying documentation relief for twelve physicians should buy. A health tech company whose customers are clinicians should build, because the workflow is what you're selling and you can't differentiate on somebody else's interface. We go deeper on that decision in our build vs buy against Nuance DAX and Abridge breakdown.

How much integration work this actually is

For the async path, the right default for a scribe, this is the whole transcription call:

from assemblyai import TranscriptStatus
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

config = TranscriptionConfig(
    speech_models=["universal-3-5-pro"],
    domain="medical-v1",
    speaker_labels=True,
)

transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(
    "./encounter.wav"
)

if transcript.status == TranscriptStatus.error:
    raise RuntimeError(f"Transcription failed: {transcript.error}")

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

Note the plural speech_models: async takes a list. If you're building a live-forming note instead, streaming uses the singular speech_model against the v3 WebSocket endpoint. Streaming Medical Mode pairs with universal-3-6-pro, the recommended streaming model in the docs. Connection settings go in the URL query string, and you send audio and read results at the same time.

import asyncio
import json
import wave
from urllib.parse import urlencode

import websockets

# Connection parameters go in the URL query string, not in a message.
PARAMS = {
    "speech_model": "universal-3-6-pro",
    "domain": "medical-v1",
    "sample_rate": 16000,
    "encoding": "pcm_s16le",
    "voice_focus": "far-field",
    "mode": "max_accuracy",
    "speaker_labels": "true",
    "min_turn_silence": 800,
    "max_turn_silence": 3600,
}
WS_URL = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(PARAMS)}"


async def stream(chunks):
    """chunks: async iterator of 50-1000 ms PCM frames, sent at real time."""
    async with websockets.connect(
        WS_URL, additional_headers={"Authorization": "<YOUR_API_KEY>"}
    ) as ws:

        async def receive():
            async for raw in ws:
                msg = json.loads(raw)
                if msg["type"] == "Turn" and msg["end_of_turn"]:
                    print(f'{msg.get("speaker_label")}: {msg["transcript"]}')
                elif msg["type"] == "Termination":
                    return

        receiver = asyncio.create_task(receive())
        async for chunk in chunks:
            await ws.send(chunk)
        # Terminate flushes the last turn; wait for Termination before closing.
        await ws.send(json.dumps({"type": "Terminate"}))
        await receiver


async def wav_chunks(path, frame_ms=100):
    """Yield 16 kHz mono 16-bit PCM frames from a WAV file at real-time pace."""
    with wave.open(path, "rb") as wav:
        frames_per_chunk = int(wav.getframerate() * frame_ms / 1000)
        while True:
            data = wav.readframes(frames_per_chunk)
            if not data:
                break
            yield data
            await asyncio.sleep(frame_ms / 1000)


if __name__ == "__main__":
    asyncio.run(stream(wav_chunks("./encounter_16k_mono.wav")))

Streaming runs $0.45/hr base and $0.60/hr with Medical Mode. Speaker labels add $0.12/hr and Voice Focus $0.10/hr. The mode field trades latency for accuracy across min_latency, balanced, and max_accuracy. Nobody's waiting on a scribe's response, so choose accuracy. The turn-silence values follow the streaming Medical Mode docs, which recommend longer pauses for clinical audio so a clinician reading the chart mid-sentence doesn't split a turn. The 3.2% MER figure above was measured on Universal-3.5 Pro async. Run your own encounters through the streaming path before you quote numbers for it.

Consent, PHI and the BAA: what legal will ask about

Ambient recording in a clinical setting needs patient notification and, depending on jurisdiction, explicit consent. Build the consent state into the recording flow rather than bolting it on. The scribe should refuse to start if consent isn't recorded, and the audit trail should show who consented and when.

On the vendor side, start with what a business associate is. Under HIPAA, it's a company a covered entity hires to do work that involves protected health information (see the HHS guidance on business associates). Here's how AssemblyAI fits:

AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI.

In practice:

• Paid accounts can review and sign the standard BAA self-serve from Data Controls in the dashboard, with no sales call. Enterprise teams can also work through the BAA with our sales team. Details are in the BAA FAQ and the BAA terms.
• Files submitted under a signed BAA aren't used for model training.
• PHI redaction works across both audio and transcripts, so you can strip identifiers from the artifacts you retain.
• AssemblyAI maintains SOC 2 Type 2, ISO 27001:2022, and PCI DSS v4.0. Self-hosted deployment and EU data residency are available where residency rules apply.

Decide your audio retention policy before launch, not after your first security review. Most teams keep audio only long enough to support clinician review of the draft note, then delete it.

What teams building this actually report

Commure runs AssemblyAI speech-to-text in its ambient documentation product:

"We've integrated the newest models from AssemblyAI for pre-recorded audio ASR in our ambient product, and it's been excellent. We're now exploring Universal-3.5 Pro for async and realtime speech-to-text capabilities for new use cases. What's been just as important is the reliability of the platform itself—both technically and in terms of partnership." — Gautam Pradeep, Tech Lead, Commure

That second point is underrated. Scribe accuracy gets all the attention in evaluations. Then production reality turns out to be about uptime, latency consistency, and whether your provider ships improvements you get for free. A scribe that's accurate but unavailable during Monday-morning clinic is not accurate in any way a clinician will notice.

Where ambient scribing goes next

Clinicians have adopted ambient scribes faster than almost any other generative AI tool in healthcare. The open questions are about value, patient trust, and how these tools are evaluated over time, as a JAMA Network Open editorial on early results lays out.

The current generation of scribes treats the encounter as an isolated audio file. The next generation won't. Contextual prompting already shows what happens when the model reads the chart before it listens: a 31% reduction in missed medical terms in internal testing, from information the health system already had in a database. Extend that and the scribe stops being a transcription-plus-summarization pipeline. It becomes a system that knows what this patient is on, what was planned last visit, and what the referral said before the first word is spoken, and that writes a SOAP note grounded in all of it. The scribes clinicians end up trusting won't be the ones with the best note templates. They'll be the ones wired most deeply into the record.

Frequently asked questions

What does ambient AI scribing mean?

Ambient AI scribing means software listens to a clinical encounter as it naturally happens, transcribes it, and drafts a structured note (usually a SOAP note) for the clinician to review and sign. The clinician doesn't dictate. The scribe works from the real conversation between clinician and patient. Teams building scribes on AssemblyAI use Universal-3.5 Pro with Medical Mode as the speech layer and generate the note through LLM Gateway.

How much does an ambient AI scribe cost?

Commercial ambient scribe platforms are typically priced per clinician per month. Building your own on an API is priced per hour of audio. On AssemblyAI that's $0.36 per hour for Universal-3.5 Pro with Medical Mode, plus LLM tokens for note generation. A health tech company scribing thousands of visits pays for the audio it processes rather than for seats.

How does an ambient AI scribe create a SOAP note?

An ambient AI scribe creates a SOAP note by passing a speaker-labeled transcript to a language model that sorts the conversation into Subjective, Objective, Assessment, and Plan. Good implementations tie each line of the note to a timestamped transcript span and only report what the clinician actually said, writing "not documented" rather than inferring a diagnosis. Note quality is capped by transcript quality, because a misheard drug name carries straight into the Plan.

Can I build an ambient AI scribe like Nuance DAX or Abridge?

Yes. Commercial scribes like Nuance DAX and Abridge run a four-stage pipeline you can build yourself: capture, speech-to-text, LLM note generation, and EHR write-back. On AssemblyAI, the speech layer is one API call with Universal-3.5 Pro and Medical Mode, and the note is a second call through LLM Gateway. Building makes sense when the clinical workflow is your product; buying makes sense when you only need documentation relief for your own clinicians.

Is an ambient AI scribe HIPAA-compliant?

AssemblyAI is considered a business associate under HIPAA and offers a standard Business Associate Addendum (BAA) for customers processing PHI. Whether a scribe product meets HIPAA requirements depends on the whole system: every vendor that processes PHI needs a BAA, and your application has to handle access controls, consent, and retention correctly. Paid AssemblyAI accounts can sign the BAA self-serve, and PHI redaction is available across audio and transcripts.

Is an AI scribe worth it?

An AI scribe is worth it when its notes are accurate enough that clinicians review rather than rewrite them, and that comes down to the speech layer. Evaluate vendors on Missed Entity Rate for drugs, conditions, and procedures rather than word error rate, and test on your own encounter audio, because your rooms and specialties are the only benchmark that binds. Medical Mode on Universal-3.5 Pro posts a 3.2% Missed Entity Rate, which is the kind of entity-level number worth asking every vendor for.