AssemblyAI for Healthcare: the most accurate voice AI for clinical products

New today: an updated Medical Mode and Vet Mode in beta

Our latest speech models already handle most clinical vocabulary well. These two modes go further on the terms where a single wrong word matters most.

What’s new in Medical Mode

Medical Mode adds a correction pass tuned for clinical language on top of our speech-to-text models. It finds medical terms the model heard but spelled wrong, like drug names, procedures, and conditions, and fixes them using the context around them.

In our benchmarks, turning on Medical Mode, compared to a base AssemblyAI model, resulted in up to 13% fewer missed medical terms, with no rise in wrongly inserted terms. On Universal-2, Medical Mode cuts missed medical terms by 27%.

We ran five other speech-to-text services on the nine datasets in the four languages Medical Mode supports, with the same metrics, each in its medical-specific model or mode wherever the vendor offers one for the language: Soniox (stt-async-v6 and stt-rt-v6), ElevenLabs (Scribe v2 Medical for batch, Scribe v2 Realtime for streaming), Google Gemini 3.5 Transcribe (batch and Live), Deepgram (nova-3-medical where it accepts the language, nova-3 otherwise) and Speechmatics (Enhanced with the medical domain where supported).

Averaged over the nine pre-recorded sets, AssemblyAI with Medical Mode has the lowest error rate on all three measures: WER, overall medical-term misses and drug-name misses (see the chart above).

For streaming, error rate is only half the picture: a system also has to commit its words quickly. Speechmatics and Soniox have the lowest or near-lowest error rates, but at default settings they finalize words about 5 seconds behind the audio. Among the systems that finalize within about a second, AssemblyAI with Medical Mode has the lowest error on all three measures. Without Medical Mode, the same model is the fastest system in the benchmark at 0.63 seconds, with the same WER but more missed drug names, so you can trade drug-name accuracy for a faster time-to-final.

Our latest models already handle most medical vocabulary well, so Medical Mode’s lift depends on your specific audio and your vocabulary. Run your files with and without it, score both on the terms that matter to your users, and keep Medical Mode on wherever it performs best. Developers already using Medical Mode with domain: "medical-v1" have received the update automatically, no changes needed.

Read the full technical deep dive →

Where it runs:

• Pre-recorded: Universal-3.5 Pro and Universal-2
• Streaming: Universal-3.6 Pro Realtime, Universal-3.5 Pro Realtime, and Universal-Streaming English and Multilingual
• Languages: English, Spanish, German, and French
• Pricing: $0.15/hr add-on

import requests

response = requests.post(
    "https://api.assemblyai.com/v2/transcript",
    headers={"authorization": "YOUR_API_KEY"},
    json={
        "audio_url": "https://example.com/clinical-visit.mp3",
        "speech_models": ["universal-3-5-pro"],
        "domain": "medical-v1",
    },
)
print(response.json()["id"])

from urllib.parse import urlencode

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-6-pro",
    "domain": "medical-v1",
}

API_ENDPOINT = (
    "wss://streaming.assemblyai.com/v3/ws?" + urlencode(CONNECTION_PARAMS)
)

Read the docs: Pre-recorded Medical Mode, Real-time Medical Mode.

Vet Mode, in beta

Veterinary medicine has its own vocabulary problem. Many vet drugs sound like human ones, and a general model tends to transcribe the word it has seen more often. Vet Mode runs the same kind of correction pass, tuned for veterinary language. It fixes two kinds of error:

• Misspellings. The model heard the term but didn’t know how to spell it.
• Wrong but real words. The model wrote a correctly spelled, similar-sounding human drug or everyday word.

With early feedback from veterinary customers, we heard some common corrections such as:

As we continue to build Vet Mode, we are making it available today in beta for our pre-recorded speech-to-text and real-time API:

• Pre-recorded audio on Universal-3.5 Pro and Universal-2
• Streaming on Universal-3.6 Pro Realtime, Universal-3.5 Pro Realtime, and Universal-Streaming

Turn it on with domain: "veterinary-v1". Try it on your own calls and let us know what else you’d like to see. Learn more about AssemblyAI for veterinary.

Read the docs: Pre-recorded Vet Mode, Real-time Vet Mode.

Where it fits

Most healthcare voice products pass the transcript to something else: an LLM that drafts the note, a field in the EHR, or an agent that books the next appointment. If a drug name is wrong in the original transcript, it’s wrong everywhere after that. Medical Mode fixes those terms before they move on. See how teams use AssemblyAI in healthcare.

• Ambient scribes: transcribe the visit with Medical Mode and speaker labels so the note keeps clinician and patient apart, then draft a SOAP note, after-visit summary, or referral letter with LLM Gateway. Use streaming if clinicians review the note during the visit, or pre-recorded audio once it ends. Our medical scribe example shows you how to build a SOAP note end to end. Learn more about AI scribes.
• Healthcare voice agents: use Medical Mode on streaming so scheduling, refill, intake, and triage agents hear medication and condition names correctly the first time, without you maintaining a custom keyterm list. Build a telehealth triage agent, or learn more about voice agents.
• Clinical dictation: use the Dictation API for fast, cleaned-up dictation in your app, so clinicians can dictate orders, findings, and notes without stopping to fix drug names and dosages. Follow our clinical dictation tutorial, or learn more about dictation.
• Medical transcription: run Medical Mode on pre-recorded audio for records, research, and review. It suits batch work like recorded consults, telehealth visits, and research interviews, and pairs with PHI redaction when transcripts leave the care team. Learn more about medical transcription.

Built for patient data

Every use case above runs on the same infrastructure, so the security review you pass for one product covers the next one you build.

• BAA available: AssemblyAI is a business associate under HIPAA and signs a standard BAA for customers processing PHI
• SOC 2 Type 2 certified
• PHI redaction across audio and transcripts
• Zero data retention so sensitive audio is never stored after transcription
• Deployment options, including on-prem for teams that need it

For enterprise teams, that means one vendor for the speech-to-text, the redaction, and the LLM step behind a clinical product, under one BAA.

We’ve integrated the newest models from AssemblyAI for pre-recorded audio ASR in our ambient product, and it’s been excellent. We’re now exploring Universal-3.5 Pro for async and realtime speech-to-text capabilities for new use cases. What’s been just as important is the reliability of the platform itself—both technically and in terms of partnership.

Get started

The updated Medical Mode is live now, and Vet Mode is in beta.

• Playground: compare results with and without Medical Mode before you write any code. Upload your own clinical audio in the pre-recorded playground, or speak into the real-time playground.
• API: add domain: "medical-v1" (or domain: "veterinary-v1" for Vet Mode) to your existing request, with the API key you already have.
• Docs: supported models, languages, and code samples for Medical Mode (pre-recorded, real-time) and Vet Mode (pre-recorded, real-time).
• BAA and enterprise pricing: contact our sales team to sign a BAA or talk about volume pricing.

New to AssemblyAI? Get your API key to start.

At HLTH in Las Vegas, November 15–18? Stop by our booth to see Medical Mode and Vet Mode side by side on clinical audio, or book a meeting.