Voice agents in healthcare: Automating phone interactions for scheduling, billing, and more

A patient calls a cardiology clinic at 7:40 p.m. to move an appointment and ask whether she should keep taking her eliquis before a dental procedure. Nobody picks up, because the front desk closed at five. She leaves a voicemail. Someone calls her back Thursday.

That call is the entire business case for healthcare voice agents, and it has almost nothing to do with replacing staff. It's about the inbound calls that arrive when no human is available and that a competent automated system could close in ninety seconds — reschedules, refill status, insurance eligibility, directions, pre-op instructions.

What makes healthcare harder than every other voice agent vertical is that the words are load-bearing. A retail agent that mishears "blue" as "black" ships the wrong sweater. A clinical intake agent that mishears "eliquis" as "Elavil" has confused an anticoagulant with a tricyclic antidepressant. That's why the accuracy figure worth caring about here is entity-level: Medical Mode posts a 3.2% Missed Entity Rate, the lowest across benchmarked providers.

Below: how the stack fits together, what it costs per hour, which use cases are actually ready, and where these systems break.

What a healthcare voice agent is made of

Every voice agent is three functions in a loop — hear, think, speak — and you can buy them separately or together.

The three-part stack

Speech-to-text converts the caller's audio into text, live, with turn detection so the agent knows when the caller has finished. Universal-3.5 Pro Realtime handles this at $0.45/hr base, with turn detection defaulting to a min_turn_silence of 128ms.

An LLM decides what to do with the text — answer, ask a clarifying question, look something up in the scheduling system, or hand off to a human.

Text-to-speech renders the reply as audio.

Wire those three together yourself and you own the orchestration: buffering, interruption handling, barge-in, latency budgeting across three vendors, and three sets of failure modes. It's the flexible path and it's real engineering.

Or one WebSocket

The Voice Agent API collapses all three into a single WebSocket connection at a flat $4.50/hr. One connection, one bill, no orchestration layer to maintain. You give up some control over model selection in exchange for not owning the plumbing.

My read: if your agent's value is in the conversation design and the backend integrations — which for healthcare scheduling and triage it usually is — take the single WebSocket and spend your engineering time on the workflow. Build the three-part stack when you have a specific reason to control one of the components.

What the timing actually looks like

Callers don't measure latency in milliseconds. They measure it in whether the pause felt wrong. The threshold where a conversation stops feeling like a phone tree is roughly the length of a natural human pause, and end-of-turn detection is the piece that governs it.

Universal-3.5 Pro Realtime detects end of turn in about 128ms. That number is doing more work than it looks like. Detect too early and you cut the caller off mid-sentence, which is the single most infuriating thing a voice agent can do. Detect too late and every exchange has a dead beat in it. Healthcare callers are also more likely to pause mid-utterance — elderly patients, patients reading a medication list off a bottle, patients who are unwell — so a system tuned for a fast-talking sales caller will interrupt them constantly.

You also get three latency modes: min_latency, balanced, and max_accuracy. For a scheduling agent, balanced is right. For an agent capturing a medication list that will be written to a chart, max_accuracy is worth the extra milliseconds, because the caller will forgive a slightly slower agent long before they forgive a wrong drug name.

Setting it up

Streaming takes speech_model as a singular string, and Medical Mode is the same one-parameter switch it is on async:

import json, websockets; from urllib.parse import quote

URL = (
    "wss://streaming.assemblyai.com/v3/ws"
    "?sample_rate=16000"
    "&speech_model=universal-3-5-pro"
    "&domain=medical-v1"
    "&mode=balanced"
    "&voice_focus=near-field"
)

PROMPT = """
Clinic: Northside Cardiology. Caller intents: reschedule, refill status,
pre-procedure instructions, insurance eligibility.
Common medications: apixaban, eliquis, metoprolol, atorvastatin,
furosemide, amiodarone, warfarin.
Providers: Dr. Adeyemi, Dr. Okonjo, Dr. Vasquez.
"""

async def run(audio_chunks):
    async with websockets.connect(
        URL + "&prompt=" + quote(PROMPT), additional_headers={"Authorization": "YOUR_API_KEY"}
    ) as ws:
        # agent_context is a different parameter: after each agent reply, send
        # an UpdateConfiguration message carrying the words the agent just spoke.
        # The static briefing above belongs in prompt, set once at connect time.
        # Audio send loop follows.
        async for chunk in audio_chunks:
            await ws.send(chunk)
            # Do not await ws.recv() inside the send loop; it stalls the audio stream.
            # Run a separate task that iterates: async for message in ws
            # and prints the caller transcript on each end-of-turn Turn message.

prompt is the highest-value parameter here

If you take one configuration detail from this post, take this one. prompt lets you tell the model what this conversation is about before it hears a word — the clinic's name, the provider names, the medications your patient population is actually on, the intents you handle.

Across 20,000 voice agent files, context cut WER by 10.2%, and detailed context cut medical-term entity errors by 43%. That's a free accuracy gain from a string you already have sitting in your application config. Most teams don't set it.

Medical Mode on top

domain: "medical-v1" costs +$0.15/hr, putting streaming with Medical Mode at $0.60/hr combined. Against the base model without Medical Mode it delivers roughly 20% fewer missed medical entities and 87% fewer entity errors. For any agent that touches medications, conditions, or procedures, that's not optional.

Medical Mode covers English, Spanish, German, and French. Separately — and this is a different capability, worth not conflating — the base model code-switches natively across 18 languages with no configuration, which matters on a patient line where callers switch mid-sentence.

Which use cases are ready now

Not all of them. Here's my honest sort.

Ready today

Scheduling and rescheduling. Bounded intent, verifiable outcome, low harm if the agent has to escalate. This is where nearly everyone should start.

Refill status and prescription routing. The agent confirms which medication and which pharmacy and routes the request. It doesn't make clinical decisions about the refill.

Insurance eligibility and benefits questions. Tedious, high-volume, entirely rules-based, and callers hate waiting on hold for it.

Appointment reminders and pre-procedure instructions. Outbound, scripted, with confirmation capture. Reliable and immediately measurable in no-show rates.

Post-discharge check-ins. Structured questions, escalation on any concerning answer. The agent's job is to ask consistently and hand off fast, not to assess.

Not ready

Anything that renders clinical judgment. Symptom triage where the agent decides urgency. Medication counseling. Anything where the wrong answer harms someone and there's no human in the loop to catch it.

The design rule I'd hold to: a healthcare voice agent should route, confirm, and capture. It should not decide. Build the escalation path before you build the happy path, and make escalation cheap — a caller who gets to a human in twelve seconds is a satisfied caller, even if the agent failed. For the wider set of patient-facing workflows this sits inside, see conversational AI in healthcare.

Where these systems break

Transcription errors on the terms that matter

Already covered, but it bears repeating in a different frame: your agent's WER is not the number to watch. Instrument entity-level accuracy on the specific vocabulary your callers use — your formulary, your provider names, your procedure names — and treat a missed entity as a hard failure rather than a rounding error. See the benchmarks page for how providers separate on this.

Phone audio is bad audio

Narrowband codecs, packet loss, hold music bleed, callers on speakerphone in a car. Set voice_focus to near-field for handset audio and far-field for speakerphone and room scenarios. Don't evaluate a voice agent on clean studio recordings; evaluate it on your own call recordings, which sound considerably worse than you remember.

Multi-party calls

An adult child calls on a parent's behalf and both speak. Streaming diarization with revision handles up to 10 speakers and corrects earlier speaker assignments as more audio arrives, which matters because the first two seconds of a call rarely contain enough signal to assign speakers confidently.

PHI handling

Every one of these calls contains PHI. AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and acts as a business associate under HIPAA — see the BAA FAQ and the BAA page. AssemblyAI is SOC 2 Type 2, supports PHI redaction across both audio and transcripts, offers EU data residency at api.eu.assemblyai.com, and a self-hosted deployment for teams that can't send audio out at all.

Your obligations don't end there. Disclose the recording at the top of the call. Set retention deliberately. Redact before transcripts land in any analytics system. Log access.

What the cost math looks like

One thing teams forget: budget for async too. You want every call re-transcribed after the fact with Universal-3.5 Pro at $0.21/hr for QA, because async sees the whole recording and will catch the errors your live pipeline made. That's how you find out your agent is mishearing one provider's name on 8% of calls. Full rates on pricing.

Who's shipping this

The voice infrastructure layer has matured fast, and Universal-3.5 Pro is available on LiveKit Inference.

"We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms." — David Zhao, Co-founder at LiveKit

On the healthcare side, Sully AI, Heidi Health, Chapter, and NMDP are all building voice and documentation workflows on this stack.

The shift worth planning for

Right now the interesting constraint on healthcare voice agents isn't accuracy or latency. Both are good enough for the ready-today use cases. The constraint is trust architecture — who's accountable when the agent gets it wrong, and how a health system proves to itself that the agent is safe to leave running overnight.

That's a measurement problem, and it's the one I'd be building toward. The teams that win the next two years won't be the ones with the fastest agent. They'll be the ones who can show, from their own call logs, exactly what their entity accuracy is on their own formulary, what their escalation rate is by intent, and which calls the agent should never have taken. That evidence is what unlocks the use cases currently sitting in the "not ready" column — and it's built from post-call transcripts you could start collecting this week.

Frequently asked questions

What accuracy does a healthcare voice agent need?

Word error rate is the wrong target. What matters is whether medications, conditions, and procedure names survive, because those are the tokens that carry consequence. Universal-3.5 Pro Realtime with Medical Mode posts a 3.2% Missed Entity Rate — the lowest across benchmarked providers. Setting prompt improves it further: it cut WER 10.2% across 20,000 voice agent files, with detailed context cutting medical-term entity errors 43%.

How much does a healthcare voice agent cost per hour?

Streaming speech-to-text with Universal-3.5 Pro Realtime is $0.45/hr, or $0.60/hr with Medical Mode enabled. The Voice Agent API, which replaces the separate speech-to-text, LLM, and text-to-speech components with one WebSocket, is a flat $4.50/hr. Async transcription for post-call review is $0.21/hr, or $0.36/hr with Medical Mode. See pricing.

How does AssemblyAI handle HIPAA and PHI?

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and operates as a business associate under HIPAA. Beyond the BAA: SOC 2 Type 2, PHI redaction across both audio and transcripts, EU data residency, and self-hosted deployment. The BAA FAQ covers how to get one in place.

What tasks should a voice agent handle, and what should go to a human?

Agents are ready for scheduling, rescheduling, refill status, insurance eligibility, appointment reminders, pre-procedure instructions, and structured post-discharge check-ins. They should not render clinical judgment — no symptom triage that decides urgency, no medication counseling. Route, confirm, and capture; escalate anything else, and make the escalation fast.

Which languages do healthcare voice agents support?

Medical Mode covers English, Spanish, German, and French for both streaming and pre-recorded audio. The base model separately code-switches natively across 18 languages with no configuration, so a caller who shifts language mid-sentence is still transcribed correctly. Details in the docs.

How do you handle calls where more than one person is speaking?

Streaming diarization supports up to 10 speakers with revision, meaning speaker assignments get corrected as more audio arrives rather than being locked in from the opening seconds. That's important on family calls, where an adult child and a parent both speak. For post-call analysis, async diarization on Universal-3.5 Pro is more accurate still, since it has the entire recording available.

‍