Best speech-to-text API for AI medical ambient scribes (2026)

For ambient clinical capture, the API you want is the one that gets medical entities right, attributes words to the right speaker when people talk over each other, holds up when the microphone is across the room, and comes with a Business Associate Addendum (BAA) you can actually sign. On those four criteria, Universal-3.5 Pro with Medical Mode posts a 3.2% Missed Entity Rate, ranks first of nine on pre-recorded diarization error rate, exposes voice_focus: "far-field" for room audio, and offers a BAA that paid accounts can sign self-serve.

The rest of this post explains why those are the four criteria, how the major APIs compare on them, and which AssemblyAI surface belongs in which part of a clinical workflow.

Building with a coding agent? Connect the docs MCP server with claude mcp add assemblyai-docs --transport http https://assemblyai.com/docs/mcp, or start from the end-to-end medical scribe example in the docs, which goes from a recorded visit to a clinical note.

What an ambient scribe demands that dictation doesn't

Dictation and ambient capture look like the same problem, and they aren't. Dictation is one known speaker, close to the microphone, speaking deliberately into a device. Ambient capture is two or more speakers at unpredictable distances in an untreated room, talking over each other, with clinical meaning spread across the whole conversation.

That difference creates four hard requirements that dictation never imposes.

1. Speaker attribution that survives crosstalk

A note built from an encounter is wrong if the patient's symptom report gets attributed to the clinician. The metric for this is cpWER (concatenated minimum-permutation word error rate). It scores transcription and speaker diarization together, so putting the right words on the wrong speaker costs you. That's exactly the failure that breaks a scribe.

On AssemblyAI's published benchmarks for pre-recorded audio (AMI, CallHome, DiPCo and NOTSOFAR), Universal-3.5 Pro ranks first of nine on diarization error rate at 12.2. On cpWER it ranks second at 26.7, 0.3 behind ElevenLabs Scribe v2 at 26.4, followed by Speechmatics at 33.7, Deepgram Nova-3 at 34.8 and Gladia at 36.5. The top two are close, so treat the published numbers as a shortlist rather than a verdict and run both on your own encounter audio.

On the streaming path, you can cap the session at up to 10 speakers with max_speakers. Labels arrive with each turn, and when the session ends the server runs a refinement pass over the whole conversation and sends a SpeakerRevision message correcting any turns it got wrong. That matters for the family-member-in-the-room case, where a third voice appears thirty seconds in and a system that locked its speaker map early never recovers.

2. Far-field audio

A clinician wearing a headset and a microphone on a desk two meters away are different acoustic problems. Close to the mouth, the voice dominates the signal. Across the room, the voice arrives weaker and mixed with reflections off hard surfaces, HVAC hum and whatever else is happening in the corridor, and the patient is usually farther from the mic than the clinician.

AssemblyAI exposes this as an explicit parameter rather than a generic noise setting. Voice Focus takes "near-field" for headsets and phones and "far-field" for rooms, kiosks and laptop mics. Note the hyphens; the underscored form is rejected. Strength is set with voice_focus_threshold from 0.0 to 1.0, defaulting to 0.7. Voice Focus is a streaming add-on on Universal-3.6 Pro Realtime.

A room-mic configuration for an exam room looks like this:

• voice_focus: "far-field" for a tablet, phone or puck sitting on the desk or counter.
• speaker_labels: true, with max_speakers set a little above the number of people you expect (3 or 4 covers clinician, patient and a family member).
• min_turn_silence: 800 and a max_turn_silence around 2000 ms. The streaming Medical Mode docs recommend longer silence thresholds for clinical audio, because clinicians pause mid-sentence to check a chart, and default voice-agent settings would split those pauses into separate turns.

One caution: Voice Focus works by suppressing audio that isn't the main speaker. That's what you want for corridor noise and a television, but in a multi-party visit the patient or a family member can sit farther from the mic than the clinician. A/B test it on your own exam-room recordings, and check that quieter participants still make it into the transcript, before you turn it on for every session.

3. Medical entities, not words

Word error rate is the wrong scoreboard for clinical audio, because it weights "the" and "metoprolol" identically. The metric that predicts whether a note is safe to sign is Missed Entity Rate: how often a clinically significant entity fails to appear at all. The MER vs WER explainer covers the metric in depth.

With Medical Mode, Universal-3.5 Pro posts a 3.2% MER, with roughly 20% fewer missed medical entities and about 87% fewer medical entity errors than the base model. The same table on the medical transcription use case page lists Speechmatics Enhanced Medical at 3.6%, Deepgram Nova-3 Medical at 4.7%, AWS Transcribe Medical at 8.7% and Google Medical Conversation at 24.4%.

4. Context about the encounter

An ambient scribe knows things the model doesn't: the specialty, the reason for the visit, the patient's active medication list. Passing that in is a second accuracy lever that stacks with Medical Mode rather than competing with it.

On a public benchmark of 20,000 real voice-agent calls, detailed context cut medical-term entity errors by 43% and scenario-level context by 24%, per the prompting and keyterms docs. In internal testing, feeding a patient's prior-visit note cut missed medical terms by 31%.

How Medical Mode works

Medical Mode is a post-processing pass. The speech model transcribes the audio first; then an LLM correction step reads the transcript, uses the surrounding clinical context, and fixes medical terms the model heard but spelled wrong: mainly medication names and, to a lesser extent, disease names. It doesn't measurably change anatomy terms, dosages, tests or procedures, which are mostly numbers, units or everyday words. You switch it on with one parameter, domain: "medical-v1", on both pre-recorded and streaming audio, in English, Spanish, German and French.

Medical Mode now runs on an updated correction model that improves medical-term recall: up to 13% fewer missed medical terms than the base model, with no rise in wrongly inserted terms. It's the same domain: "medical-v1" parameter, and it needs no code change. It runs on Universal-3.5 Pro and Universal-2 for pre-recorded audio, and on Universal-3.6 Pro Realtime, Universal-3.5 Pro Realtime and Universal-Streaming for streaming.

AssemblyAI's Medical Mode research puts numbers on that scope. Pooled across four English clinical sets (84 hours), Medical Mode removed 30% of drug-name misses in batch (55 of 182, on Universal-3.5 Pro) and 16% in streaming (32 of 194, on Universal-3.6 Pro Realtime). On the doctor-patient set, the batch drug-name miss rate fell from 7.9% to 5.1%. Word error rate stayed flat, moving by 0.1 point or less on every English set, so the correction fixes terms without disturbing the rest of the transcript. Medical Mode covers English, Spanish, French and German.

Ambient scribes often run in real time, so the streaming results matter too. These are averages over nine sets; latency is p90 time-to-final.

Medical Mode adds about 0.65 s at p90. Among systems that finalize words within about a second, AssemblyAI with Medical Mode has the lowest error rates. Speechmatics posts a lower streaming WER (9.6% vs 10.4%) and drug-name miss rate, and Speechmatics and Soniox finalize at about 4.7 to 5.2 s.

Two things to design around:

• It corrects misspellings. If the speech model writes a wrong but valid word, the correction pass may not catch it. Contextual prompting and keyterms work at the model input, so they cover a different part of the problem.
• Keep keyterm lists tight. Keyterms work alongside Medical Mode. On long recordings, keep the list to about 200 terms or fewer; very long lists start adding false insertions.

Medical Mode is a $0.15/hr add-on on top of the base model rate.

Which AssemblyAI surface for which clinical workflow

Because dictation and ambient capture are different problems, they route to different endpoints. This is the part most build decisions get wrong, so here it is explicitly.

Ambient capture: realtime, async, or both

Most production scribes run both paths. Streaming drives the live view the clinician watches during the encounter. Async re-processes the full recording afterward for the note that gets signed, because a model that sees the whole file makes better diarization and entity decisions than one working turn by turn.

The post-encounter pass on Universal-3.5 Pro takes speech_models (plural) and domain in the transcription config:

import os

from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

config = TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    domain="medical-v1",
    speaker_labels=True,
    keyterms_prompt=["metoprolol", "lisinopril", "atorvastatin"],
)

transcriber = Transcriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])
transcript = transcriber.transcribe("encounter.wav", config)

if transcript.status == "error":
    raise RuntimeError(transcript.error)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

The live pass runs on Universal-3.6 Pro Realtime. Streaming takes speech_model (singular) with the same domain, plus the far-field and turn-silence settings from earlier. This example replays a 16 kHz mono WAV in 100 ms chunks so you can test it without a microphone; in production you'd stream from the device instead.

import os
import time
import wave

from assemblyai.streaming.v3 import (
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    SpeakerRevisionEvent,
    TurnEvent,
)


def on_turn(client: RealTimeTranscriber, event: TurnEvent):
    if event.end_of_turn and event.transcript:
        print(f"[{event.speaker_label}] {event.transcript}")


def on_revision(client: RealTimeTranscriber, event: SpeakerRevisionEvent):
    # End-of-session pass: corrected speaker labels for earlier turns.
    for revision in event.revisions:
        print(f"turn {revision.turn_order} -> {revision.speaker_label}")


def on_error(client: RealTimeTranscriber, error: RealTimeError):
    print(f"Error: {error}")


client = RealTimeTranscriber(
    RealTimeTranscriberOptions(terminate_timeout=30.0),
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
)
client.on(RealTimeEvents.Turn, on_turn)
client.on(RealTimeEvents.SpeakerRevision, on_revision)
client.on(RealTimeEvents.Error, on_error)

client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-6-pro",
        domain="medical-v1",
        speaker_labels=True,
        voice_focus="far-field",  # room mic; A/B test on multi-party visits
        min_turn_silence=800,
        max_turn_silence=2000,
    )
)

# Replay a 16 kHz, mono, 16-bit WAV in 100 ms chunks, paced like live audio.
with wave.open("encounter.wav", "rb") as wav:
    frames_per_chunk = wav.getframerate() // 10
    while chunk := wav.readframes(frames_per_chunk):
        client.stream(chunk)
        time.sleep(0.1)

# Terminate flushes the last turn and waits for the Termination event.
client.disconnect(terminate=True)

Register handlers with the RealTimeEvents enum. Passing the TurnEvent class to client.on() registers nothing, and the script runs silently.

Front-end dictation: the Dictation API

When a clinician is dictating into a field, such as a chart note, an order or a message to a colleague, you don't want a verbatim transcript. Speakers restate and self-correct, and what was said isn't what anyone wants to send. The Dictation API runs speech-to-text and an LLM cleanup pass in one call and returns both: text is the verbatim transcript and is never altered, and llm_response is the cleaned version. It returns finished text in about 0.36 s and runs on Universal-3.5 Pro.

import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

config = aai.DictationConfig(
    stt_prompt="A doctor dictating a patient visit note.",
    keyterms_prompt=["amoxicillin", "lisinopril", "metoprolol"],
    llm_instruction="Remove filler words and rewrite as a concise clinical chart note.",
)

result = aai.DictationTranscriber().transcribe_live("clip.wav", config)

print(result.text)        # verbatim transcript, never altered
print(result.final_text)  # cleaned text, falling back to the transcript

Two constraints to design around:

• 120 seconds of audio per live call. A routine SOAP note is about 250–400 words, two to three minutes of speech, so the cap covers short dictations and sits right at the edge of a typical note. Route longer notes to pre-recorded transcription.
• No Medical Mode. domain: "medical-v1" runs on the pre-recorded and streaming APIs only. There's no domain parameter on the Dictation API, and none on the Sync API either. The Dictation API still takes keyterms_prompt for drug names, which covers a lot of clinical vocabulary in practice, but it's a different capability.

One more thing to know before you pick: clinicians are trained to speak punctuation ("Medications colon aspirin comma naproxen period"). AssemblyAI's models apply punctuation automatically and handle spoken command words inconsistently, and there's no spoken-form mode today. Deepgram has a dictation parameter that converts spoken commands into punctuation, English only. The documented approach on the AssemblyAI side is an LLM pass over each finalized turn through the LLM Gateway to interpret and strip command words, which is the pattern in the medical scribe best practices guide. It's a workaround, not a native mode.

Ambient scribes beyond human medicine

Veterinary practices have the same documentation problem with a different vocabulary, and a human-medicine model gets it wrong in a specific way. It doesn't just misspell veterinary drugs; it swaps them for real human drugs that sound similar. A base model can hear Zydax and write Xanax, and nothing in the sentence looks wrong.

Vet Mode, now in beta, applies the same post-processing approach to veterinary audio. You enable it with domain: "veterinary-v1" on pre-recorded audio (Universal-3.5 Pro and Universal-2) and on streaming (Universal-3.6 Pro Realtime, Universal-3.5 Pro Realtime and Universal-Streaming). It's built for terms like Cerenia, Heartgard, trilostane, carprofen and Antisedan, including swaps such as ibuprofen for carprofen. Setup is in the pre-recorded and streaming Vet Mode docs. For how vet and legal scribes differ from clinical ones, see ambient scribes for veterinary and legal.

How the major speech-to-text APIs compare for ambient capture

Below is what each vendor publishes or what a published benchmark shows. Medical MER figures come from one table on the medical transcription use case page; vendors not in that table are marked as such rather than estimated. Check each vendor's own documentation before you commit.

Three things to read out of that table.

First, the gaps are wider on entities than on words. On the English voice-agent benchmark from the Universal-3.6 Pro Realtime research, the word error rate gap to Deepgram Nova-3 is about three and a half points, while the entity error gap is nearly twelve. Those entity figures cover general conversational entities (names, phone numbers, emails, dates, codes), not clinical terms, but the pattern holds on the medical table too: Medical MER separates providers far more than WER does.

Second, diarization is the row where the field is closest. ElevenLabs Scribe v2 leads cpWER by 0.3, while Universal-3.5 Pro leads diarization error rate. If speaker attribution decides your build, run both on your own exam-room audio.

Third, a BAA is available from every vendor in the table, but the terms differ. Deepgram signs for Enterprise customers, the cloud providers cover speech services under their account-level BAAs, and ElevenLabs' terms are worth confirming with ElevenLabs directly. Who can sign, on which plan and how fast are the questions that move a pilot's start date.

What teams building scribes run into

Commure builds clinical workflow software, including an ambient product, on AssemblyAI:

"We've integrated the newest models from AssemblyAI for pre-recorded audio ASR in our ambient product, and it's been excellent. We're now exploring Universal-3.5 Pro for async and realtime speech-to-text capabilities for new use cases. What's been just as important is the reliability of the platform itself—both technically and in terms of partnership."

— Gautam Pradeep, Tech Lead, Commure

The patterns that recur across teams building in this space:

• The room, not the model, is the hard part. Exam rooms have hard surfaces, HVAC noise and a microphone nobody positions correctly. Mic placement and a far-field Voice Focus test are the first things to try when production accuracy trails your bench tests.
• Diarization errors surface as clinical errors. A misattributed symptom report is worse than a misspelled word, and a WER-based eval won't catch it. Evaluate with cpWER, and use the end-of-session speaker revision for the signed note.
• Context is free accuracy. If your product already knows the specialty and the medication list, passing them in is the cheapest improvement available: 43% fewer medical-term entity errors on the public benchmark.
• The signed note isn't the live note. Re-processing the recording asynchronously after the encounter produces a better document than the streaming transcript alone.

If you're still weighing whether to build at all, the roundup of the best ambient AI scribes covers the products you'd otherwise buy.

PHI, BAAs, and what you can and can't say

AssemblyAI is a business associate under HIPAA, not a covered entity. The legal-approved description:

AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI.

Paid accounts can review and sign the BAA self-serve from the Data Controls page in the dashboard, without a sales call. See can you sign a BAA and the Business Associate Addendum. Supporting controls include PHI redaction across audio and transcripts, SOC 2 Type 2, ISO 27001:2022 and PCI DSS v4.0. Medical Mode doesn't change existing data handling or retention policies. EU data residency is available at the same price.

The transcription layer is the cheap part

The speech layer decides whether a note can be trusted, but it's rarely where the budget goes. The expensive parts of an ambient scribe are EHR write-back, consent flows, specialty templates and a review UI clinicians will tolerate. So pick the API on the four criteria at the top of this post, test it on your hardest room, and spend the engineering time on the parts your clinicians actually see.

Frequently Asked Questions (FAQs)

Does AssemblyAI sign a Business Associate Addendum (BAA)?

AssemblyAI is considered a business associate under HIPAA and offers a standard Business Associate Addendum (BAA) for customers processing PHI. Paid accounts can review and sign the BAA self-serve from the dashboard, without a sales call. Supporting controls include PHI redaction across audio and transcripts, SOC 2 Type 2, ISO 27001:2022 and PCI DSS v4.0.

What is the best speech-to-text API for an AI medical scribe?

The best API is the one with the lowest missed entity rate on clinical terms and the strongest speaker attribution measured by cpWER. Universal-3.5 Pro with Medical Mode posts a 3.2% Missed Entity Rate and ranks first of nine on pre-recorded diarization error rate (12.2) and second on cpWER (26.7, behind ElevenLabs Scribe v2 at 26.4). Word error rate alone is a poor proxy, because it weights common words and drug names the same. Run your own encounter audio through the shortlist before deciding.

How do you handle multiple speakers in an exam room?

Turn on streaming diarization with speaker_labels: true and set max_speakers slightly above the number of people you expect, up to 10. Labels arrive with each turn, and an end-of-session refinement pass sends corrected labels for any turns it reassigns, which handles a family member who joins partway through. For the signed note, use the revised labels or re-process the recording with pre-recorded diarization.

Does the model handle a microphone across the room?

Yes. Set voice_focus: "far-field" for room microphones, tablets and laptop mics, and "near-field" for headsets and phones. Strength is controlled by voice_focus_threshold from 0.0 to 1.0, defaulting to 0.7. Test it on your own recordings first, and check that quieter participants, such as a patient farther from the mic, still come through.

Does Medical Mode work in real time?

Yes. Medical Mode runs on streaming audio with Universal-3.6 Pro Realtime, Universal-3.5 Pro Realtime and the Universal-Streaming models, using the same domain: "medical-v1" parameter as pre-recorded audio. The gain is smaller than in batch, because streaming corrects each turn with less context and under a time budget: on four English clinical sets it removed 16% of drug-name misses in streaming versus 30% in batch, and it adds about 0.65 s at p90 time-to-final. Many scribes use streaming for the live view and re-process the recording with pre-recorded Medical Mode for the signed note.

Can I use the same API for a veterinary scribe?

Yes. Vet Mode is in beta and is enabled with domain: "veterinary-v1" on pre-recorded and streaming audio. It corrects veterinary drug names that a general model misspells or swaps for similar-sounding human drugs, such as writing Xanax for Zydax or ibuprofen for carprofen. It runs on Universal-3.5 Pro and Universal-2 for pre-recorded audio, and on Universal-3.6 Pro Realtime, Universal-3.5 Pro Realtime and Universal-Streaming for streaming.

How much does the transcription layer of an ambient scribe cost?

Medical Mode is a $0.15/hr add-on on top of the base model rate: Universal-3.5 Pro at $0.21/hr for pre-recorded audio and Universal-3.6 Pro Realtime at $0.45/hr for streaming. Running both paths with Medical Mode comes to $0.96 per hour of audio. Optional add-ons such as diarization and Voice Focus are itemized on the pricing page.