How to build clinical dictation on AssemblyAI

Clinical dictation and ambient scribing get lumped together constantly, and they're not the same engineering problem.

An ambient scribe listens to a whole encounter — two people, fifteen minutes, overlapping speech, a lot of silence. Clinical dictation is a clinician alone, talking deliberately into a device for twenty seconds, expecting the text before they've finished walking to the next room. One is a streaming problem. The other is a short-clip problem.

This post is about the second one. Since the Dictation API shipped, there are now two short-clip endpoints that fit clinical dictation, and they answer different questions. Picking the wrong one costs you either a formatting layer you didn't need to build or a rewrite pass you didn't want.

Before anything else, the scoping note that will save you a debugging session.

Read this first: where Medical Mode runs

Medical Mode is not available on the Sync API or the Dictation API.

Medical Mode is our clinical-accuracy add-on, activated with a single parameter — domain: "medical-v1" — and it runs on Universal-3.5 Pro (pre-recorded) and Universal-3.5 Pro Realtime (streaming). It's worth using where it's available: 3.2% Missed Entity Rate, roughly 20% fewer missed medical entities than the base model without it, and the lowest Missed Entity Rate across benchmarked providers including Deepgram, Speechmatics, AWS, and Google. You can see the comparison on our benchmarks page. It supports English, Spanish, German, and French, in both pre-recorded and streaming modes, and costs $0.15/hr on top of the base model rate — $0.36/hr combined with flagship async.

Neither short-clip endpoint's config accepts a domain parameter. Sending one won't enable Medical Mode.

So which do you reach for?

If your product is short dictation, the rest of this post covers both short-clip paths. There's a good answer either way, it's just a different one depending on whether you want finished text or a faithful transcript.

The shape of clinical dictation

Real clinician dictations look like this:

"Patient is a 62-year-old male, follow-up for hypertension. Blood pressure today 148 over 92. Continue lisinopril 20 milligrams daily, add hydrochlorothiazide 12.5 milligrams. Recheck in four weeks."

Twelve seconds. One speaker. Dense with exactly the vocabulary a general model is most likely to mangle — drug names, dosages, and the difference between "hydrochlorothiazide" and whatever a model guesses when it doesn't know the word.

That's the whole optimization target. Get the drugs and the numbers right, get the text back fast enough that the clinician doesn't context-switch.

Everything after that is a formatting decision, and it's the one that decides which endpoint you want.

Verbatim or finished text? The decision that picks your endpoint

Both short-clip endpoints run on Universal-3.5 Pro and both accept the same accuracy controls. The difference is what comes back.

The Sync API returns the transcript. Every word as spoken, including "um," including the false start where the clinician said "continue lisi— continue lisinopril." If your product needs the record to reflect what was actually said, or a downstream model reads the note and formatting carries clinical meaning, that's the right behavior. Physicians are trained to speak their punctuation, so "patient open parentheses Andrew close parentheses is tired" is a rendering instruction rather than a filler pattern — and a cleanup pass that decides to interpret it is a problem, not a feature.

The Dictation API returns both: the verbatim transcript in text, never altered, and a cleaned, steerable version in llm_response. One request, one round trip, both versions in the same response object, so your product never has to guess which one it's showing. The steering is llm.instruction — plain English, up to 2,048 characters, set per request — which is how you get a SOAP note, an SBAR handoff, or a JSON object shaped like your intake form out of the same clip.

Two boundaries to hold onto before you pick.

Cleanup polishes; it does not correct. A rewrite pass reformats the words it was given. Hand it a misheard drug name and you get a confidently formatted wrong drug name, which reads worse than a messy right one because it reads authoritative. A clinician skims it, it goes in the chart, nobody notices. Accuracy is fixed upstream — in the next section — never in the rewrite instruction.

Medical Mode isn't on either one. If medical entity accuracy is the whole value proposition of your product — coding, medication reconciliation, anything where a missed drug name is a safety event — the honest recommendation is to run that workload on Pre-recorded STT or Real-time STT with domain: "medical-v1" and accept the different interaction shape.

The rest of this post builds on the Sync API, because the verbatim path is the one that needs the most explaining. Everything in the next two sections — contextual prompting, keyterms, pre-warming, data residency — applies to the Dictation API too.

Step 1: Send the dictation

The base call is four lines.

import os

from assemblyai.sync.v1 import SyncTranscriber

transcriber = SyncTranscriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])
result = transcriber.transcribe("./dictation.wav")
print(result.text)

import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const result = await client.sync.transcribe("./dictation.wav");
console.log(result.text);

Constraints worth designing around: audio between 80 milliseconds and 120 seconds, up to 40MB, as WAV or raw PCM S16LE, 16-bit, mono or stereo. Two minutes is generous for a dictation — most clinical notes land under 45 seconds. If yours run longer, route them to Pre-recorded STT, where you also get Medical Mode.

Calling it over plain HTTP instead? Two things the SDKs handle for you: the Authorization header takes the raw key with no Bearer prefix, and X-AAI-Model: universal-3-5-pro is required on every request.

curl -X POST https://sync.assemblyai.com/transcribe \
  -H 'Authorization: <YOUR_API_KEY>' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F 'audio=@dictation.wav;type=audio/wav'

The response carries text, per-word confidence, an overall confidence score, audio_duration_ms, a session_id, and request_time_ms — the end-to-end server-side processing time. Log the session_id on every request, not just failures; it's the first thing support asks for.

Step 2: Give the model clinical context

Here's what replaces domain: "medical-v1" on the short-clip endpoints: contextual prompting and keyterms. They're not the same mechanism as Medical Mode's entity correction, but they're the documented, supported way to steer Universal-3.5 Pro toward clinical vocabulary, and for a known specialty they work well.

prompt is a description of the audio, not an instruction to the model. Three levels of specificity, and you should use the least specific one that fixes your errors:

That specificity ladder is worth taking seriously, and the medical numbers are better than the general ones. Benchmarked on 20,000 real calls, scenario-level context cut medical-term entity errors by about 24%, and detailed context by about 43%. In internal healthcare testing, feeding a patient's prior-visit note cut missed medical terms by 31% — even when the note came from an earlier visit.

That last result is the one worth building around. If your application already knows who the patient is and what happened last time, you're holding the most valuable prompt content available to you, and it costs nothing to send.

keyterms_prompt takes an explicit vocabulary list and biases the decoder toward those tokens. This is where your formulary goes:

from assemblyai.sync.v1 import SyncTranscriber, SyncTranscriptionConfig

config = SyncTranscriptionConfig(
    prompt="Cardiology follow-up about hypertension and medication adjustment.",
    keyterms_prompt=[
        "lisinopril",
        "hydrochlorothiazide",
        "metoprolol succinate",
        "Humalog",
        "atrial fibrillation",
    ],
)

transcriber = SyncTranscriber(api_key="<YOUR_API_KEY>", config=config)
result = transcriber.transcribe("./dictation.wav")
print(result.text)

Full detail on both fields is in the Sync prompting and keyterms guide. Practical guidance, and these bite teams:

Start with neither. Universal-3.5 Pro is tuned to perform without prompting, and a large keyterm list — especially one padded with common words — causes overcorrection, where the model starts hearing your terms in audio that doesn't contain them. In a clinical context that's not a cosmetic error. Add terms only for the words you've watched it get wrong in your own audio.

Scope the list per specialty, not per organization. A cardiology dictation doesn't need your dermatology formulary. You have 2,048 characters total across all terms, which goes further when the list is targeted. If your app already knows which clinic the user is in, build the list from that.

Spell terms exactly as you want them output, including capitalization. Humalog, not humalog.

language_code is ignored when you set a custom prompt. This one is easy to miss and hard to debug, because nothing errors — you just quietly get English steering on Spanish audio. If you need both, name the language inside the prompt text: "Spanish-language cardiology follow-up about hypertension." Non-English clinical dictation is exactly where this shows up, usually in a second-market rollout.

And the framing that keeps the two surfaces straight if you're on the Dictation API: prompt tells the speech model what it's about to hear. llm.instruction tells the rewrite what to do with the transcript after it's written down. Putting "hydrochlorothiazide" in the instruction does nothing — the word was never going to be heard in the first place. It goes in keyterms_prompt.

Step 3: Pre-warm so the clinician isn't waiting

Sync STT is one request/response, so connection setup sits inside your latency budget. If the HTTPS connection isn't open when the clinician stops talking, the request negotiates one first — DNS, then TCP, then TLS, a round trip each. Near the serving region that's a few tens of milliseconds; from a distant client it can add well over 100ms, which is a meaningful fraction of the transcription itself on a short clip.

Move that work to the moment recording starts, when you know audio is coming but don't have it yet:

import assemblyai as aai
from assemblyai.sync.v1 import SyncTranscriber

# Hold the idle connection long enough to cover the whole recording.
aai.settings.keepalive_expiry = 120

with SyncTranscriber(api_key="<YOUR_API_KEY>") as transcriber:
    transcriber.warm()        # clinician taps record
    # ... dictation happens ...
    result = transcriber.transcribe("dictation.wav")   # no handshake

GET /warm is an unauthenticated no-op — the value is entirely on the wire. The warm call and the transcribe call have to share a connection pool: same client object, same base URL, or the second request quietly opens a fresh connection and pays the handshake anyway (full details here). Pooled connections also expire — some HTTP clients drop idle connections after five seconds — so call /warm right before you need it rather than at app startup. It's idempotent and cheap.

That base-URL detail matters more than usual here, because of where your audio needs to live.

Step 4: PHI, data residency, and the BAA

If your dictations contain protected health information, three things need attention before you ship.

Pick your endpoint deliberately. Alongside the global endpoint, which routes to the nearest region, there are dedicated US and EU data-residency endpoints:

Choose based on where your PHI is allowed to be processed, then use that same base URL consistently — including for /warm. A connection warmed against one endpoint does nothing for a request sent to another. In the Python SDK the base URL is the sync_base_url setting; in JavaScript it's the syncBaseUrl client option.

Sign the BAA. AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI. Paid accounts can review and sign it self-serve, at no additional cost, from the Data Controls page in the dashboard — no sales call needed. The BAA FAQ and the BAA legal page have the details. AssemblyAI is SOC 2 Type 2 certified, ISO 27001:2022 certified, and PCI DSS v4.0 compliant.

Consider redaction downstream. PHI redaction across audio and transcripts is available on our pre-recorded pipeline. If your architecture stores dictations, decide where redaction happens before you accumulate a corpus you'd rather not have.

"We've integrated the newest models from AssemblyAI for pre-recorded audio ASR in our ambient product, and it's been excellent. We're now exploring Universal-3.5 Pro for async and realtime speech-to-text capabilities for new use cases. What's been just as important is the reliability of the platform itself—both technically and in terms of partnership."

— Gautam Pradeep, Tech Lead, Commure

The honest summary

Short clinical dictation is a good fit for both short-clip endpoints — one speaker, one utterance, a response fast enough to keep a clinician in flow. The choice between them is a formatting question, not an accuracy one: the Sync API hands you the words as spoken and you own everything after that; the Dictation API hands you both the verbatim transcript and a version shaped the way you asked for it, in the same response.

What you're trading away on either is Medical Mode's entity-level correction, and you're compensating with contextual prompting and a per-specialty keyterm list. For a lot of dictation products that trade is fine, because the vocabulary in any one specialty is narrower than people assume and a well-scoped keyterm list covers it — and because a prior-visit note in the prompt does more work than most teams expect.

For products where medical entity accuracy is the whole value proposition, run that workload on Pre-recorded STT or Real-time STT with domain: "medical-v1" and accept the different interaction shape.

Pick the API that matches the workload rather than the one that matches the demo.

Explore what fits your build: see the medical solutions overview or compare approaches in best medical speech-to-text.

Frequently asked questions

Can I use Medical Mode with the Sync API or the Dictation API?

No. Medical Mode is activated with domain: "medical-v1" and runs on Universal-3.5 Pro (pre-recorded) and Universal-3.5 Pro Realtime (streaming). Neither short-clip endpoint's config accepts a domain parameter. On both, use prompt and keyterms_prompt for clinical vocabulary — and if your product depends on entity-level medical accuracy, run that workload on the async or realtime endpoint instead.

Should I use the Dictation API or the Sync API for clinical dictation?

Use the Dictation API when you want finished text — a structured chart note, an SBAR handoff, filler removed — because it returns the verbatim transcript and a steerable rewrite in one response, shaped per request with llm.instruction. Use the Sync API when the record needs the words exactly as spoken, which is common in legal, compliance, and any workflow where a clinician speaks their own punctuation. Both run on Universal-3.5 Pro and take the same accuracy controls.

What's the difference between clinical dictation and an ambient scribe?

Dictation is a single clinician speaking deliberately into a device, usually under a minute, expecting text immediately — a short-clip workload. An ambient scribe passively captures a multi-speaker encounter over many minutes — a streaming workload. Different APIs, different accuracy requirements, different UX.

Does AssemblyAI sign a BAA?

Yes. AssemblyAI is considered a business associate under HIPAA and offers a standard Business Associate Addendum (BAA) for customers processing PHI. Paid accounts can sign it self-serve from the Data Controls page in the dashboard, at no additional cost and without a sales call.

Which languages does Medical Mode support?

English, Spanish, German, and French, in both pre-recorded and streaming modes. If you send an unsupported language, the API ignores the domain parameter, returns a warning, and doesn't charge for Medical Mode.

How much does Medical Mode cost?

$0.15/hr on top of the base model rate. Combined with flagship async transcription, that's $0.36/hr. It's an add-on rather than a separate product, so there's no model switch — you add one parameter to a request you're already making.

Can I keep clinical audio in a specific region?

Yes. Alongside the global endpoint, there are dedicated US (sync.us.assemblyai.com) and EU (sync.eu.assemblyai.com) data-residency endpoints. Use the same base URL for every call in a session, including connection pre-warming — a connection warmed against one endpoint doesn't help a request sent to another.

How many medical terms can I pass as keyterms?

Up to 2,048 characters total across all terms. Scope the list to the specialty rather than the whole organization — a shorter, targeted list outperforms a long one and avoids overcorrection, where the model starts hearing your terms in audio that doesn't contain them.