Healthcare voice agents: Complete implementation guide

A patient calls to ask about a refill. She says "Xarelto." The agent hears "Zocor." Both are real drugs, both are common, and the two are treated by completely different specialists for completely different problems. The agent's next question is now wrong, the patient knows something is off, and whatever trust the deployment had built is gone in one turn.

That's the specific failure mode that separates a healthcare voice agent from a general one. Everything else in the stack — the telephony, the LLM, the voice synthesis, the escalation logic — is the same problem everyone building voice agents solves. The clinical vocabulary is not. It's the binding constraint, and it's the part most teams underestimate until they're in a pilot.

Two facts to anchor the build. Universal-3.5 Pro Realtime with Medical Mode hits a 3.2% Missed Entity Rate on medical entities — the lowest across benchmarked providers. And turn detection defaults to min_turn_silence 128ms and max_turn_silence 1280ms on the balanced preset, which is what makes the agent feel like it's listening rather than waiting.

What a healthcare voice agent is, and two ways to build one

A healthcare voice agent is a system that holds a spoken conversation with a patient or member and does something as a result — books an appointment, collects intake answers, triages a symptom, confirms a medication list, follows up after a discharge. It listens, understands, decides, and speaks back.

There are two architectures, and the choice mostly comes down to how much control you need.

The assembled pipeline

You wire streaming speech-to-text to an LLM to a text-to-speech service, and you own the orchestration: turn detection, interruption handling, backchannel, latency budget, error recovery. More work, more control. Worth it when you have unusual routing logic, an existing agent framework, or strict requirements about which model handles reasoning.

The Voice Agent API

One WebSocket replaces the whole STT plus LLM plus TTS chain at a flat $4.50/hr. You're not managing three vendors, three latency budgets and three failure modes. For most healthcare use cases — scheduling, intake, refill requests, reminders — this is the faster path to something you can put in front of patients.

Either way, the speech recognition layer is where clinical accuracy is won or lost, so that's where most of this post sits.

The latency budget

Conversation has a rhythm, and people notice when it breaks. The working rule of thumb for voice agents is that a response should land inside about a second from the moment the caller stops talking — beyond that, callers start repeating themselves or assume the line dropped.

That second gets spent across four things: detecting that the caller finished, finalizing the transcript, generating a response, and synthesizing speech. The first one is where a lot of agents quietly lose. If your turn detection waits on a fixed silence timeout, you've spent 700ms before the LLM has seen a token. Universal-3.5 Pro Realtime's turn detection defaults to min_turn_silence 128ms and max_turn_silence 1280ms on the balanced preset, and the streaming mode parameter lets you choose explicitly: min_latency, balanced, or max_accuracy. For a conversational agent, min_latency is usually the right call — you can run max_accuracy on a parallel async pass afterward if you need a clean record.

Why speech recognition quality is the binding constraint

An LLM cannot recover a drug name it never received. This sounds obvious and it's routinely ignored, because teams evaluate the agent end to end, see it behaving sensibly on scripted test calls, and never separate out how often the transcript was already wrong.

The metric to evaluate on is Missed Entity Rate, not Word Error Rate. WER weights "the" identically to "tirzepatide." An agent can post an excellent WER and still be losing every medication name in the call.

Methodology is on the benchmarks page. The thing I'd emphasize for agent builders isn't the headline number, it's that Medical Mode is the same parameter on streaming as on async. If your live agent and your post-call pipeline use different models, they will disagree about what the patient said, and reconciling that is a category of bug you don't want.

Getting the medication names right

Three mechanisms stack here, and using all three is the difference between a demo and a deployment.

Medical Mode

One parameter, domain: "medical-v1". Against the base model without it: 87% fewer entity errors and roughly 20% fewer missed medical entities. It's a $0.15/hr add-on on top of the $0.45/hr streaming base, so $0.60/hr combined.

Context carryover with agent_context

This is the feature I'd point at first for agent builders. agent_context carries what's already happened in the conversation into the decode, so the model isn't transcribing each turn cold. Across 20,000 voice agent files it cut WER by 10.2%, and detailed context cut medical-term entity errors 43%. Practically: if the agent just asked "which medication are you calling about," the model decodes the answer knowing a drug name is coming.

Contextual prompting

If you can identify the caller before the conversation starts — and in a patient portal or a callback flow you usually can — feed their context in. In an internal healthcare test, supplying a patient's prior-visit note cut missed medical terms by 31%. A caller asking about a refill is overwhelmingly likely to name a drug already on their list.

Real phones, real rooms

Phone audio is narrowband and compressed. Speakerphone in a kitchen is worse. Two controls matter.

voice_focus takes near-field or far-field. Handset calls are near-field; a patient on speakerphone across a room is far-field, and setting it correctly is a one-line change with a real effect.

Diarization matters more than people expect on healthcare calls, because a caller is often not alone — an adult child on the line for a parent, an interpreter, a caregiver answering on the patient's behalf. Universal-3.5 Pro's diarization is the most accurate we've shipped and is optimized for cpWER rather than DER, which means it's tuned to attribute the right words to the right speaker rather than just draw tidy boundaries. Streaming supports diarization with revision across up to 10 speakers.

And language: the base model code-switches natively across 18 languages with no configuration, so a caller switching between Spanish and English mid-sentence doesn't break the transcript. Medical Mode itself covers English, Spanish, German and French. Keep those two facts separate when you plan coverage.

Building the streaming layer

import asyncio, json, websockets; from urllib.parse import urlencode

CONFIG = {
    "speech_model": "universal-3-5-pro",
    "domain": "medical-v1",
    "mode": "min_latency",
    "voice_focus": "near-field",
    "speaker_labels": True,
    "prompt": "Inbound patient call to a cardiology clinic. Active medications: 
apixaban, atorvastatin, metoprolol.", "agent_context": "Which medication are you
calling about today?"
}

async def run_agent(phone_audio, on_transcript):
    url = "wss://streaming.assemblyai.com/v3/ws?" + urlencode({k: str(v).lower()
if isinstance(v, bool) else v for k, v in CONFIG.items()})
    async with websockets.connect(
        url, additional_headers={"Authorization": "YOUR_API_KEY"}
    ) as ws:
        

        async def send():
            async for chunk in phone_audio:
                await ws.send(chunk)

        async def receive():
            async for msg in ws:
                event = json.loads(msg)
                if event.get("end_of_turn"):
                    await on_transcript(event["transcript"])

        await asyncio.gather(send(), receive())

Note that streaming takes speech_model singular while the async API takes speech_models as a plural array. Same model ID, universal-3-5-pro, on both.

Where healthcare voice agents earn their keep

Scheduling and rescheduling

The highest-volume, lowest-risk starting point. Calls are short, intent is narrow, and a wrong answer is recoverable. Start here.

Intake and pre-visit data collection

Medication lists, allergies, symptom history, insurance details. This is where clinical vocabulary accuracy stops being a nice-to-have — a mis-transcribed allergy is a clinical safety issue, not a customer service annoyance. Turn Medical Mode on and turn confirmation prompts on for anything that lands in a chart.

Refills, triage and nurse-line front doors

Higher stakes and higher value. The design rule that matters: the agent's job is to collect and route accurately, then hand off. Escalation to a human should be cheap, fast and frequent early in a deployment.

Outbound follow-up

Post-discharge check-ins, medication adherence, appointment reminders. Outbound is where volume economics get interesting, because these calls mostly don't happen today at all.

On the infrastructure side, this is a space where the tooling has matured fast. As David Zhao, Co-founder at LiveKit, put it:

"We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms." — David Zhao, Co-founder at LiveKit

That last clause is the practical point. The alternative to context carryover is maintaining keyword lists per specialty, per formulary, per clinic — which is a job nobody wants and nobody keeps current.

Integration and escalation

Three integration surfaces decide whether the agent is useful or just impressive.

Telephony. Whatever carries the call has to hand you raw audio frames you can stream, and accept synthesized audio back with low added latency.

The system of record. An agent that can't read the schedule or the medication list is doing guesswork. Read access is what makes contextual prompting possible in the first place; write access is what makes the agent actually save anyone time.

Human escalation. Design the handoff before the happy path. The agent should escalate on low ASR confidence, on any clinical-urgency signal, and whenever the caller asks — with the transcript so far handed to the human so the patient doesn't repeat themselves.

PHI, BAA and security

What your security review will ask for, stated plainly:

• AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and acts as a business associate under HIPAA. See the BAA FAQ and the BAA page.
• PHI redaction runs across audio and transcripts both — call recordings get redacted at the source, not just the derived text.
• SOC 2 Type 2 covers platform controls.
• EU data residency at api.eu.assemblyai.com; self-hosted deployment for programs that can't send audio out.

Get the BAA executed before the pilot, not during it.

What it costs

The assembled pipeline looks cheaper on the speech line and usually isn't cheaper overall once you add LLM tokens, TTS minutes and the engineering time to keep three latency budgets in sync. Current rates are on the pricing page, and the async model post covers the batch side.

Where this is going

The next constraint on healthcare voice agents won't be accuracy or latency — both are already good enough for the workflows above. It'll be memory. An agent that remembers the last three calls, knows which medication the patient struggled to pronounce, and carries that forward will feel qualitatively different from one that starts cold every time. Context carryover across a turn is the first step; context carryover across a care relationship is the interesting one. Teams building on clinical speech infrastructure today — Sully AI, Heidi Health, Chapter, Commure — are the ones positioned to get there. Worth reading alongside this: medical speech-to-text and the Medical Mode launch post.

Frequently asked questions

How accurate does a healthcare voice agent need to be on medical terminology?

Accurate enough that a caller never has to correct a medication name, because that's where trust breaks. Evaluate on Missed Entity Rate rather than Word Error Rate. Universal-3.5 Pro Realtime with Medical Mode measures 3.2% MER — the lowest across benchmarked providers. See the benchmarks page.

How fast does a voice agent have to respond to feel natural?

Aim to have audio coming back within about a second of the caller finishing. The largest avoidable chunk of that is turn detection — Universal-3.5 Pro Realtime's turn detection defaults to min_turn_silence 128ms and max_turn_silence 1280ms on the balanced preset, and the mode parameter (min_latency, balanced, max_accuracy) lets you trade explicitly rather than discovering the trade in production.

How does AssemblyAI handle HIPAA and PHI?

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and operates as a business associate under HIPAA. Supporting that: PHI redaction across audio and transcripts, SOC 2 Type 2, EU data residency, and a self-hosted deployment option. Start with the BAA FAQ.

Should I build the pipeline myself or use the Voice Agent API?

Use the Voice Agent API — one WebSocket, flat $4.50/hr — unless you have a specific reason to control orchestration, such as an existing agent framework or a requirement about which model does the reasoning. Assembling STT plus LLM plus TTS yourself gives you more control and three latency budgets to keep in sync.

Does the agent redact patient PHI automatically?

PHI and PII redaction is a request parameter, and it applies to the audio as well as the transcript — so stored call recordings can be redacted at the source. You choose which policies apply based on what your workflow actually needs downstream.

What languages can a healthcare voice agent handle?

Medical Mode covers English, Spanish, German and French for streaming. Separately, the base Universal-3.5 Pro model code-switches natively across 18 languages with no configuration, so a caller moving between languages mid-sentence still transcribes correctly — a common pattern on patient lines. Details in the docs.

‍