Create an ambient AI scribe that works during telehealth video calls

It's 7pm. A clinician has finished fourteen telehealth visits and has fourteen unfinished notes. Every one of those conversations was already digital — recorded, or at least streamable, sitting inside a video platform. The transcript could have written itself. Instead the clinician is typing from memory, four hours after the visit that memory refers to.

That gap is the entire opportunity for an AI medical scribe. And telehealth is the easiest place to close it, because you don't have to solve the hardest part of ambient documentation — getting clean audio out of a room. In a video call the audio is already separated, already digital, and already flowing through a system you control.

What you do have to solve is clinical vocabulary. A general speech model transcribing "we'll switch her to sacubitril-valsartan" will produce something confident and wrong. Universal-3.5 Pro with Medical Mode measures a 3.2% Missed Entity Rate on medical entities — the lowest across benchmarked providers — and it's one parameter on a request you already know how to make. This post is the build.

What an AI medical scribe is, and what it isn't

An AI medical scribe listens to a clinical encounter and produces a structured note without anyone dictating to it. Three things get confused with it:

Dictation is a clinician speaking a note directly to a machine. One voice, deliberate delivery, structured input. Useful, and a completely different product — the clinician is still doing the composition work.

A human scribe sits in on the visit and writes the note. Accurate, expensive, and doesn't scale. The whole point of ambient AI is to hit acceptable quality at a cost that lets every visit have one.

A meeting summarizer pointed at healthcare audio is what most first attempts actually are. It produces something readable that no clinician will sign, because it doesn't know clinical structure and it gets the drug names wrong.

A real medical speech-to-text pipeline is the difference. Ambient means the participants talk to each other, not to the tool, and the tool figures it out.

Why telehealth audio is easier — and where it bites

Telehealth gives you three advantages over in-person ambient recording:

• Per-participant audio tracks. Most platforms can give you separate streams per speaker, which is a diarization shortcut nobody in an exam room gets.
• Near-field microphones. Everyone is a foot from a laptop mic or wearing a headset, not six feet from a desk unit.
• A system boundary you control. The audio is already flowing through software, so you don't need hardware in the room.

Three things bite back. Network artifacts — dropouts, jitter and codec compression chew up exactly the acoustic detail that distinguishes "hydralazine" from "hydroxyzine." Home environments are noisy in unpredictable ways: kids, TVs, traffic, a caregiver in the next room. And participants often aren't alone, so even with per-participant tracks you can have two people on one feed.

The pipeline

Four steps: capture audio, transcribe it, attribute it, structure it. Let's build each one.

Step 1: get the audio out of the call

Every major platform offers a path — a recording API, a bot participant that joins the call, or a raw media stream. Zoom, Microsoft Teams and purpose-built telehealth platforms all support at least one of these. Two decisions to make up front:

Per-participant or mixed? Take per-participant tracks if the platform offers them. It makes role attribution trivial. If you only get a mixed track, diarization handles it.

Recording or live stream? If the note is written after the visit, take the recording and use the async API. If you want a draft note appearing as the visit ends, stream it. Async is cheaper and more accurate; streaming is what makes the clinician feel like the tool is keeping up.

Step 2: transcribe with Medical Mode

import requests, time

BASE = "https://api.assemblyai.com/v2"
HEADERS = {"authorization": "YOUR_API_KEY"}

payload = {
    "audio_url": "https://example.com/telehealth-visit-3319.wav",
    "speech_models": ["universal-3-5-pro"],
    "domain": "medical-v1",
    "speaker_labels": True,
    "redact_pii": True,
    "redact_pii_policies": ["person_name", "date_of_birth", "phone_number",
"email_address"],
    "redact_pii_audio": True
}

job = requests.post(f"{BASE}/transcript", json=payload, headers=HEADERS).json()

while (result := requests.get(f"{BASE}/transcript/{job['id']}", headers=HEADERS).json()
)["status"] != "completed": time.sleep(3)
transcript = "\n".join(
    f"[{u['speaker']}] {u['text']}" for u in result["utterances"]
)

Three parameters carry the weight. speech_models: ["universal-3-5-pro"] selects the async flagship — note it's plural on async and singular on streaming, which catches everyone once. domain: "medical-v1" is the entire Medical Mode activation; no model switch, no separate endpoint. And speaker_labels gives you utterance-level attribution, which the note generation step depends on.

Step 3: figure out who's the clinician

Diarization gives you speakers A, B and C. It doesn't tell you which one is the doctor, and the note depends entirely on knowing. "The patient reports chest pain" and "the clinician reports chest pain" are not the same record.

Universal-3.5 Pro's diarization is the most accurate we've shipped, and it's optimized for cpWER rather than DER. That distinction is worth understanding: DER scores whether speech boundaries were drawn in the right places, cpWER scores whether the right words ended up attributed to the right speaker. Clinical encounters are built out of the cases where those diverge — a patient answering "no, the left one," a caregiver interjecting, both people speaking at once. Short turns and overlapped speech survive rather than getting absorbed into the neighboring speaker. Streaming diarization supports revision across up to 10 speakers.

For role mapping, use the cheapest signal available in this order: per-participant track metadata from the platform if you have it; the authenticated clinician's identity from your own session data; and only then inference from the transcript, since the clinician is reliably the one asking structured questions.

Step 4: generate the note

With an attributed transcript, note generation is a structured extraction problem. The LLM Gateway lets you run it without standing up separate inference infrastructure.

SOAP_PROMPT = """You are drafting a clinical note from a telehealth visit transcript.
Speaker roles are labeled. Produce JSON with keys: subjective, objective,
assessment, plan.

Rules:
- Use only information present in the transcript.
- Attribute symptoms to the patient and findings to the clinician.
- Preserve medication names, doses and frequencies exactly as transcribed.
- If a field has no supporting content, return an empty string. Do not infer.
- Include a `flags` array naming anything ambiguous for clinician review.
"""

# Send SOAP_PROMPT plus the diarized transcript to the LLM Gateway.
# Return the JSON to your UI as a draft for clinician review and sign-off.

Two design rules I'd treat as non-negotiable. First, never let the model infer. An empty Objective section is a clinician's cue to fill it in; a plausible invented one is a liability. Second, always require the flags array, and put those flags in front of the clinician. The value of an AI scribe is not that it's always right — it's that it's honest about where it isn't.

Accuracy: what to measure, and how to move it

Evaluate on Missed Entity Rate, not Word Error Rate. WER weights "the" the same as "tirzepatide," so a model can post a great WER while dropping every clinically meaningful term in the visit.

Methodology is on the benchmarks page. Three things you can do to improve on the baseline:

Turn Medical Mode on. Against the same base model without it: 87% fewer entity errors and roughly 20% fewer missed medical entities. That's a one-line change.

Feed patient context. In an internal healthcare test, supplying a patient's prior-visit note cut missed medical terms by 31%. If your scribe is integrated with a chart, you already have this — use it. Nothing else available to you delivers that much for that little work.

Handle multilingual visits properly. Universal-3.5 Pro code-switches natively across 18 languages with no configuration, so a visit that moves between Spanish and English mid-sentence transcribes correctly without you declaring a language. Medical Mode itself covers English, Spanish, German and French. Those are two separate facts — plan coverage against both.

Real time or after the visit?

Async is the right default. It's more accurate, it's $0.21/hr base and $0.36/hr with Medical Mode, and nobody is waiting on it — the note appears while the clinician is walking to the next visit.

Stream when the product design calls for the note to exist the moment the call ends, or when you want live prompts during the visit. Universal-3.5 Pro Realtime connects at wss://streaming.assemblyai.com/v3/ws:

import websockets; from urllib.parse import urlencode

CONFIG = {
    "speech_model": "universal-3-5-pro",
    "domain": "medical-v1",
    "mode": "max_accuracy",
    "voice_focus": "near-field",
    "speaker_labels": True,
    "prompt": "Telehealth follow-up. Patient on lisinopril and metformin."
}

async def run(call_audio):
    url = "wss://streaming.assemblyai.com/v3/ws?" + urlencode({k: str(v).lower()
if isinstance(v, bool) else v for k, v in CONFIG.items()})
    async with websockets.connect(
        url, additional_headers={"Authorization": "YOUR_API_KEY"}
    ) as ws:
        
        async for chunk in call_audio:
            await ws.send(chunk)

Use max_accuracy for a scribe — nothing is waiting on a sub-second reply, unlike a healthcare voice agent where min_latency is the right choice. Set voice_focus to near-field for headset and laptop-mic audio. prompt carries scenario context into the decode: across 20,000 voice agent calls, scenario context cut medical-term entity errors 24% and detailed context cut them 43%. agent_context is a separate parameter that carries the reply your voice agent just spoke, so it does not apply to a scribe. Streaming is $0.45/hr base and $0.60/hr with Medical Mode.

PHI, consent and compliance

Telehealth adds a wrinkle in-person recording doesn't: the patient is at home, on their own device, and the consent conversation happens on camera. Build consent capture into the flow, log it, and make it easy to decline without losing the visit.

On the processing side:

• AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and acts as a business associate under HIPAA. See the BAA FAQ and the BAA page.
• PHI redaction runs across audio and transcripts both, so a retained recording can be redacted at the source rather than only the text derived from it.
• SOC 2 Type 2 covers platform controls.
• EU data residency at api.eu.assemblyai.com, plus a self-hosted deployment for programs that can't send audio outside their own infrastructure.

Decide your retention policy before launch, not after your first records request. The most common answer that works: keep the note, discard the audio on a short clock, and redact whatever you retain. Current rates for everything above are on the pricing page.

Who's building this

Ambient documentation is the most competitive corner of clinical AI right now, and the teams furthest along have converged on roughly this architecture. Commure is one of them:

"We've integrated the newest models from AssemblyAI for pre-recorded audio ASR in our ambient product, and it's been excellent. We're now exploring Universal-3.5 Pro for async and realtime speech-to-text capabilities for new use cases. What's been just as important is the reliability of the platform itself—both technically and in terms of partnership." — Gautam Pradeep, Tech Lead, Commure

Sully AI, Heidi Health, Deepscribe, Knowtex and Magentus Healthcare are working in adjacent parts of the same space. What's notable across all of them is how little of the differentiation sits in the transcription layer anymore — that's become infrastructure. The product is what you do with the transcript.

What changes next

The scribe that wins won't be the one with the best transcript. It'll be the one that stops producing a note for review and starts producing a note that's already right, because it knew the patient before the visit started. Contextual prompting cutting missed terms by 31% is the early version of that. The full version is a scribe that reads the chart, knows the last three visits, recognizes which medications this patient has struggled to pronounce, and drafts accordingly — at which point the clinician's job shifts from editing to confirming. That's a different product, and it's close. Start with the docs, the medical transcription use case, or the Medical Mode launch post.

Frequently asked questions

What's the best API for building an AI medical scribe?

Judge on three things: entity-level accuracy, diarization quality, and whether async and streaming share a model. Universal-3.5 Pro with Medical Mode measures 3.2% Missed Entity Rate — the lowest across benchmarked providers — ships the most accurate diarization we've released, and uses the same model ID for both pre-recorded and streaming audio. See the benchmarks page for methodology.

Can I build something like Nuance DAX or Abridge on top of a speech API?

The transcription layer, yes — that's what this post walks through, and it's a few hundred lines. What those products actually sell is everything downstream: EHR integration depth, specialty-specific note templates, clinician workflow, and the trust built by years of deployment. Plan for the transcription to be the easy part.

How is an AI medical scribe different from dictation?

With dictation the clinician composes the note out loud — the machine just types. With ambient scribing the clinician and patient talk to each other and the system figures out the note from the conversation. That requires diarization, clinical vocabulary handling, and structured extraction, none of which dictation needs.

How does AssemblyAI handle HIPAA and PHI?

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and operates as a business associate under HIPAA. Supporting that: PHI redaction across audio and transcripts, SOC 2 Type 2, EU data residency, and a self-hosted deployment option. Start with the BAA FAQ.

Does an AI scribe have to integrate with the EHR?

Not to be useful. A draft note the clinician copies in still removes most of the work. But EHR read access is what unlocks contextual prompting, which cut missed medical terms by 31% in an internal healthcare test — so integration pays for itself in accuracy, not just in convenience.

What languages does an AI medical scribe support?

Medical Mode covers English, Spanish, German and French for both pre-recorded and streaming audio. Separately, Universal-3.5 Pro code-switches natively across 18 languages with no configuration, so bilingual telehealth visits transcribe correctly even outside those four — you just don't get the medical entity boost on top. More detail on the healthcare solutions page.

‍