Voice AI Meetup recap: How Commure and Ona Health are building for healthcare

A physician in a Bay Area clinic starts an encounter in English, switches to Spanish when the patient's daughter joins the room, and switches back when she dictates the plan. The visit lasts eleven minutes. The note takes longer than the visit. That single pattern — multilingual, multi-speaker, ambient, and expensive in clinician time — came up in some form in nearly every conversation at our healthcare-focused Voice AI meetup, and it's the reason the evening turned into less of a demo showcase and more of an engineering post-mortem.

We hosted the meetup for teams building clinical documentation, telehealth, and behavioral health products. The room was mostly engineers and technical founders, which meant the questions were specific: how do you separate a doctor from a patient when they talk over each other, how do you catch a hallucinated drug name before it reaches a chart, and what does any of this actually cost per hour at scale.

This is a recap, but the logistics are the least interesting part. What follows is the set of durable takeaways — the engineering problems that haven't gone away, the ones that have, and where the answers now sit in our stack. Where product details have moved since the event, we've updated them to what's shipping today: Universal-3.5 Pro for pre-recorded audio and Universal-3.5 Pro Realtime for streaming.

Takeaway 1: medical entity accuracy is the only accuracy number that matters

Word error rate came up early and got dismissed fast. Nobody in the room cared about a half-point of WER on filler words. What they cared about was whether the model got "metoprolol" right, or turned it into "metropolol," or dropped it entirely.

That's a different measurement, and it has a name: Missed Entity Rate. It counts how often the drugs, conditions, and procedures in the audio fail to make it into the transcript. It's the metric that maps to clinical risk, because a missing medication is a missing medication regardless of what the overall WER says.

Medical Mode — one parameter, domain: "medical-v1" — lands at a 3.2% Missed Entity Rate, the lowest across the providers we've benchmarked. The full methodology and numbers live on our benchmarks page.

Against the base model without Medical Mode on, the add-on delivers roughly 20% fewer missed medical entities and 87% fewer entity errors. Those are two different measurements of the same improvement, and the second one is the one that made people in the room sit up.

Medical Mode is a parameter, not a different model

The most common misconception we cleared up at the event: enabling Medical Mode doesn't mean switching to a separate medical model with its own feature set and its own regressions. You keep Universal-3.5 Pro, you keep diarization, you keep timestamps, you add one field.

import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro"],
    domain="medical-v1",
    speaker_labels=True,
)

transcript = aai.Transcriber().transcribe(
    "https://example.com/clinic-encounter.wav",
    config=config,
)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

Pricing follows the same logic. Universal-3.5 Pro is $0.21/hr, Medical Mode adds $0.15/hr, so clinical async transcription runs $0.36/hr all in. Streaming starts at $0.45/hr base and reaches $0.60/hr with Medical Mode on. All of it is on the pricing page.

Takeaway 2: diarization is the hard part, and it's getting easier

Ask ten teams building ambient scribes what breaks most often and eight will say speaker attribution. A clinical encounter isn't a podcast. People interrupt. A patient answers halfway through the clinician's question. A family member says three words from across the room. Traditional diarization, tuned for clean turn-taking, falls apart on exactly those moments — and those moments are where the clinically important content lives.

Part of the problem is how diarization gets measured. Diarization Error Rate scores how much audio time got assigned to the wrong speaker, which flatters systems that handle long monologues and punishes nobody for dropping a two-word turn. Concatenated minimum-permutation WER — cpWER — scores the words in each speaker's transcript, which is what a scribe pipeline actually consumes downstream.

Universal-3.5 Pro's diarization is optimized for cpWER rather than DER, and it's the most accurate diarization we've shipped. It captures short turns and overlapped speech, which is the specific failure mode this room kept describing. In streaming, diarization comes with revision — early speaker assignments get corrected as more context arrives — and handles up to 10 speakers.

Why revision matters for live scribing

If you're rendering a live transcript to a clinician, you can't wait for the end of the session to decide who said what. But you also can't be wrong on screen. Revision splits the difference: you show your best guess immediately, then correct it as the conversation gives the model more to work with. By the end of the session, your speaker labels are close to what you'd get from a full pre-recorded pass.

Several teams at the meetup were running two pipelines for this reason — streaming for the live view, async for the note of record. That's a legitimate architecture, and it's cheaper than it sounds, but it's no longer the only option.

Takeaway 3: multilingual clinical audio isn't an edge case

The bilingual-encounter scenario at the top of this post wasn't hypothetical, and it wasn't rare. Teams serving community health systems described code-switching as the default, not the exception, and every one of them had a story about a pipeline that handled it by picking one language and mangling the other.

Two facts are worth keeping distinct here, because conflating them causes real problems in procurement conversations.

First: the base model code-switches natively across 18 languages, with no configuration and no per-utterance language hinting. A speaker moving between languages mid-sentence stays intelligible in the transcript.

Second: Medical Mode itself supports English, Spanish, German, and French. So a Spanish-English encounter gets both native code-switching and clinical entity accuracy. A Tagalog-English encounter gets the code-switching but not the Medical Mode entity boost. Knowing which of those you're in changes what you promise a health system.

Takeaway 4: context is the cheapest accuracy win nobody uses

This was the surprise of the night. Several teams were spending real engineering effort on custom vocabulary lists, maintaining thousands of terms per specialty and re-uploading them constantly. Meanwhile the single highest-impact input was sitting in their database already: the patient's prior-visit note.

In an internal healthcare test, feeding a patient's prior-visit note as context cut missed medical terms by 31%. Not a curated term list — just the note. The patient's medications, conditions, and specialist names are already in there, which is exactly what the model needs to disambiguate a mumbled drug name.

On the streaming side the equivalent lever is prompt, which cut WER by 10.2% across 20,000 voice agent files, with detailed context cutting medical-term entity errors 43%. Same idea: tell the model what the conversation is about, and it stops guessing.

import assemblyai as aai

# Streaming: single speech_model, plus session context
config = {
    "sample_rate": 16000, "speech_model": "universal-3-5-pro",
    "domain": "medical-v1",
    "prompt": (
        "Behavioral health follow-up. Patient on sertraline 100mg "
        "and lamotrigine. Prior visit noted sleep-onset insomnia."
    ),
    "voice_focus": "far-field",
    "mode": "balanced",
    "speaker_labels": True,
}

# wss://streaming.assemblyai.com/v3/ws

A note on voice_focus, which came up repeatedly from teams putting a device on a desk rather than a headset on a clinician: near-field and far-field are separate settings for a reason. An exam room with a mic across the room is a far-field problem, and telling the model that is free.

Takeaway 5: the compliance conversation is shorter than teams expect

Every healthcare team eventually hits the procurement question, and most of them arrive at it with more anxiety than it warrants. The short version: AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI, which makes us a business associate under HIPAA for that data. We're SOC 2 Type 2. PHI redaction runs across both audio and transcripts, so the redacted artifact you persist doesn't carry the identifiers the raw capture did.

The details are in the BAA FAQ and the Business Associate Addendum itself. For teams with data-residency requirements, EU processing and self-hosted deployment both exist.

What surprised people was how much of the hard work is on their side, not the vendor's: retention windows, clinician sign-off before a note enters the chart, and audit trails on who read what. Those are product decisions, and no API grants them to you.

Takeaway 6: measure time-to-edit, not accuracy

The most useful metric discussed all evening didn't come from a model card. Multiple teams had converged independently on the same production signal: how long a clinician spends editing a generated note before signing it.

It's a good metric because it collapses everything that matters into one number a clinician feels. Transcription errors show up in it. Bad speaker attribution shows up in it. A note template that doesn't match how this specialty documents shows up in it. And unlike a benchmark, it moves when your product gets better for your users specifically.

The teams tracking it were also the teams shipping fastest, which is probably not a coincidence.

What we're watching next

The interesting shift isn't accuracy — that curve is well understood and still bending. It's that the transcript is stopping being the product. Once entity accuracy is good enough that clinicians stop proofreading, the transcript becomes plumbing, and the competition moves to what sits on top: does the note match how this clinician writes, does it pull forward the right history, can the clinician talk to it instead of editing it.

That last one is closest. A Voice Agent API session at a flat $4.50/hr — one WebSocket replacing STT, LLM, and TTS — turns "fix the assessment section" into something a clinician says out loud while walking to the next room. The teams that get there first won't win on Missed Entity Rate. They'll win because a clinician never opened a keyboard.

Frequently asked questions

What is AssemblyAI's Medical Mode?

Medical Mode is a domain setting you add to a transcription request with domain: "medical-v1". It tunes recognition for medical entities — drug names, conditions, procedures — without switching models, so you keep Universal-3.5 Pro or Universal-3.5 Pro Realtime and everything else you had configured. It costs an extra $0.15/hr on top of the base model and supports English, Spanish, German, and French. Full details are on the medical solutions page.

How accurate is AssemblyAI Medical Mode compared to other providers?

Medical Mode records a 3.2% Missed Entity Rate, the lowest across the providers we've benchmarked. Against our own base model without Medical Mode enabled, it delivers roughly 20% fewer missed medical entities and 87% fewer entity errors. The methodology is published on our benchmarks page.

Which speech-to-text model should I use for an ambient AI scribe?

Use Universal-3.5 Pro Realtime with Medical Mode if the clinician needs a live view during the encounter, and Universal-3.5 Pro with Medical Mode for the note of record after it. Many teams run both. There's a longer breakdown in our guide to the best medical speech recognition software and APIs.

How does AssemblyAI handle HIPAA and PHI?

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI, which makes us a business associate under HIPAA for that data. We're also SOC 2 Type 2, and PHI redaction runs across both audio and transcripts. Our BAA FAQ covers how to get one in place.

Can AssemblyAI handle clinical conversations that switch languages?

Yes. The base model code-switches natively across 18 languages with no configuration, so a clinician and patient moving between languages mid-conversation stay intelligible in the transcript. Medical Mode's clinical entity tuning covers English, Spanish, German, and French, so a Spanish-English encounter gets both.

How well does diarization separate doctor from patient?

Universal-3.5 Pro's diarization is the most accurate we've shipped and is optimized for cpWER rather than DER, which means it's scored on getting each speaker's words right rather than on how much audio time landed in the right bucket. It captures short turns and overlapped speech — the interruptions and one-word answers that break most clinical pipelines. In streaming it supports up to 10 speakers with revision as the session progresses. There's more in our post on behavioral health documentation clinicians trust and in the docs.