Build a voice agent for telehealth triage

It's 11:40pm. A patient calls the after-hours line describing "pressure in my chest, kind of like heartburn, and my left arm feels heavy." The nurse queue is eleven callers deep. What happens next — how fast that description is captured, whether "left arm" and "pressure" survive transcription, and whether the system flags them — decides if this call escalates in seconds or waits behind ten sore throats.

That's the case for a triage voice agent, and also the reason it's harder to build than a scheduling bot. A scheduling agent that mishears a date annoys someone. A triage agent that mishears a symptom, or that fails to escalate, is a patient safety event.

Two things make it tractable. First, entity-level speech accuracy: Universal-3.5 Pro Realtime with Medical Mode posts a 3.2% Missed Entity Rate on clinical entities, the lowest across benchmarked providers. Second, architecture that keeps the agent narrow — it collects and routes, and a licensed human makes every clinical decision.

This post covers what the agent should and shouldn't do, the call flow, the two ways to build it, and the configuration that decides whether it's safe to put in front of patients.

What a triage voice agent does — and what it must never do

Start with the boundary, because everything else follows from it.

It does not diagnose

A voice agent doesn't tell a patient what's wrong with them, doesn't rule anything out, and doesn't recommend a treatment. It gathers structured information against a clinical protocol and routes the call. The clinical judgment stays with a licensed clinician, every time. This isn't just a legal position — it's what makes the system's failure modes bounded. An agent that only collects and routes can be wrong in ways a nurse catches. An agent that offers conclusions can be wrong in ways nobody catches.

What it does do

Verify identity. Ask protocol-driven questions in the right order. Capture the chief complaint, symptom onset, severity, and current medications in the patient's own words. Score against your triage protocol. Escalate to a nurse when the score or a red flag says to. Book, message, or transfer as the protocol directs. And write a structured record of the whole interaction into the chart.

Where the real value sits

Not in replacing nurses — in triaging the queue so nurses spend time on the callers who need them, arriving with a completed structured summary instead of a cold call. That's a throughput change, and it's the honest business case.

The call flow

1. Greeting and scope. Identify the service, state plainly that this is an automated intake line, and give an immediate path to a human and to emergency services.
2. Identity verification. Match the caller to a patient record before any clinical content is discussed.
3. Red-flag screen. Before the structured protocol, ask the small set of questions that trigger immediate escalation — chest pain with radiation, difficulty breathing, stroke symptoms, uncontrolled bleeding, suicidal ideation. This runs first, not last.
4. Chief complaint capture. Open-ended, in the patient's words, transcribed with clinical entity accuracy.
5. Protocol questions. Branching questions from your triage protocol, adapted to the complaint.
6. Severity scoring and disposition. Score the responses, pick the disposition your protocol specifies, and say what happens next.
7. Handoff and documentation. Transfer or schedule, then write the structured encounter — transcript, extracted entities, score, disposition, and timestamps — to the chart.

Step 3 deserves emphasis. Red-flag screening belongs at the front of the call, before the protocol tree, because the whole point is to catch the emergency in the first twenty seconds rather than at question fourteen.

Why the speech layer decides whether this is safe

A triage agent is a chain: transcribe, understand, decide, act. Errors at the transcription step propagate through everything after, and they don't announce themselves.

Clinical entity accuracy

Patients name their medications, and they name them imperfectly. A general speech model handles conversational English beautifully and turns "hydralazine" into "hydroxyzine," or drops the word "left" from "left arm." Medical Mode — one parameter, domain: "medical-v1" — adapts the model to clinical vocabulary and delivers 87% fewer entity errors than the same model without it, at a 3.2% Missed Entity Rate, the lowest across benchmarked providers. Methodology is on our benchmarks page; we go deeper on measurement in our post on AI medical transcription accuracy.

Conversation context, carried forward

A patient mentions "the metoprolol" at minute two and refers to "that beta blocker" at minute six. Without conversation-level context, the model treats each turn independently. With agent_context, Universal-3.5 Pro Realtime applies what's already been said to the words still coming: word error rate dropped 10.2% across 20,000 voice agent files, and detailed context cut medical-term entity errors 43%. On a triage call, that's the difference between a clean medication list and a nurse re-asking everything.

Turn-taking that doesn't interrupt a patient describing symptoms

Turn detection defaults to min_turn_silence 128ms and max_turn_silence 1280ms on the balanced preset. Too eager and the agent cuts off a patient mid-sentence — which on a triage call means losing the clause that mattered. Too slow and the conversation feels broken and callers hang up. The mode setting trades latency against accuracy across min_latency, balanced, and max_accuracy; for triage, balanced is usually right, because you need both responsiveness and entity accuracy.

Phone audio, and callers in bad places

Telehealth triage calls arrive over PSTN at 8 kHz, from cars, hospital lobbies, and speakerphones. Set voice_focus to match — near-field for a handset, far-field for a speakerphone. And plan for language: Universal-3.5 Pro code-switches natively across 18 languages with no configuration, so a caller moving between English and Spanish is handled without a language selection step. Medical Mode's clinical adaptation covers English, Spanish, German, and French, which is a shorter list than the base model's range — worth knowing before you promise coverage.

Two ways to build it

You can assemble the stack yourself or use a single interface that handles the whole conversation loop.

For a first triage deployment, the Voice Agent API's flat $4.50/hr is easier to defend to a finance team than a three-vendor bill that varies with call length. If you already run an orchestration framework, keep it and swap in Universal-3.5 Pro Realtime as the transcription layer.

"We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms." — David Zhao, Co-founder at LiveKit

That last clause matters specifically for triage. You cannot enumerate in advance every medication a patient population might name. Context that builds during the call beats a keyword list you wrote in advance.

Building the transcription layer

If you're assembling the stack, this is the streaming configuration for a triage call. Streaming uses the singular speech_model field and the v3 WebSocket endpoint:

import json
import websockets; from urllib.parse import urlencode

WS_URL = "wss://streaming.assemblyai.com/v3/ws"

config = {
    "speech_model": "universal-3-5-pro",
    "domain": "medical-v1",
    "sample_rate": 8000,          # PSTN telephony
    "voice_focus": "near-field",  # handset; far-field for speakerphone
    "mode": "balanced",
    "agent_context": "Thanks for calling the nurse line. What is going on today?", 
# the reply your TTS just spoke; resend after each agent turn
}

async def triage_call(audio_chunks, on_turn):
    async with websockets.connect(
        WS_URL + "?" + urlencode({k: str(v).lower() if isinstance(v, bool) else v for k, v in config.items()}), additional_headers={"Authorization": "YOUR_API_KEY"}
    ) as ws:
        
        async for chunk in audio_chunks:
            await ws.send(chunk)
            # Read Turn messages in a separate asyncio task - interleaving recv
with send stalls the stream

Post-call, run the recording through async transcription for the record of truth. Async sees the full audio and is more accurate than any live pass, and at $0.36/hr with Medical Mode it's cheap insurance on a document that may be reviewed later:

import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro"],
    domain="medical-v1",
    speaker_labels=True,
    redact_pii=True,
    redact_pii_audio=True, redact_pii_policies=[aai.PIIRedactionPolicy.person_name, aai.PIIRedactionPolicy.date_of_birth, aai.PIIRedactionPolicy.phone_number],
)

transcript = aai.Transcriber(config=config).transcribe("triage-call.wav")

Note the plural speech_models on async versus singular speech_model on streaming. Full reference in the docs.

The parts that make it safe

A system prompt with hard refusals

Write the boundary into the prompt explicitly: never state or imply a diagnosis, never advise for or against a medication, never estimate how serious something is, never tell a caller not to seek care. Include the exact escalation script for each red flag and the exact wording for offering a human. Then test that the agent holds those lines under pressure — patients will ask "do you think it's a heart attack?" and the answer has to be a transfer, not an opinion.

Tool calling with a narrow surface

Give the agent a small set of tools: look up patient, record symptom, compute triage score, escalate to nurse, schedule appointment, write encounter note. Every tool that touches the chart should be idempotent and logged. Escalation should be one call that works even if everything else in the flow has failed.

Deterministic severity scoring

Don't ask a language model to decide urgency. Have it extract structured fields, then score those fields with the same deterministic logic your nurse line already uses. Scoring in code means it's reviewable, testable, and identical on every call — three properties a model's judgment doesn't have.

Escalation that fails toward the human

Every ambiguous case goes to a person. If the transcript confidence is low, if the patient is distressed, if the protocol doesn't match, if a tool call fails — transfer. Design the default path so that a broken agent escalates rather than proceeding. An agent that hangs up on an edge case is far worse than one that transfers too often.

Audit logging you'd be comfortable handing to a regulator

Log the audio reference, transcript, extracted entities with timestamps, every tool call and result, the computed score, the disposition, and who took the handoff. Log the model and configuration version too, so a case reviewed six months later reproduces.

Testing before patients

Build a test suite of recorded scenarios covering every red flag, every protocol branch, heavy accents, background noise, interruptions, off-script callers, and a second language. Include cases designed to make the agent give medical advice. Run it on every prompt change — prompt edits regress safety behavior invisibly. Then pilot with a nurse listening live before the agent takes a call alone. For the documentation side of the same stack, see our guide to ambient AI scribes.

PHI, consent, and contracting

A triage call is PHI from the first sentence. Tell callers at the top that the call is recorded and handled by an automated system, and give them a path to a human immediately rather than after the disclosure.

On the vendor side: AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and operates as a business associate under HIPAA. PHI redaction is available across audio and transcripts, the platform is SOC 2 Type 2 audited, and self-hosted deployment and EU data residency are available where residency requirements apply. See the BAA FAQ and the BAA terms, and our healthcare solutions page for how the pieces fit.

Get the BAA in place before the pilot, not before launch — pilot calls are real PHI.

Where triage agents go next

Today's triage agents are protocol executors with a voice interface — they ask the questions a paper algorithm would ask, in order. The change coming is context. The same mechanism that cut missed medical terms 31% when a prior-visit note was passed with the audio applies to a phone call: an agent that reads the chart before the patient starts talking already knows the medication list, the recent procedure, and the standing care plan. It can skip questions it has answers to and ask sharper ones about what changed. That's a better patient experience and a shorter call, and it moves the agent from executing a generic protocol toward executing this patient's protocol. The technical pieces exist now. The work is plumbing the record into the call.

Frequently asked questions

How do I build a telehealth triage voice agent?

Put a red-flag screen at the front of the call, run your existing triage protocol as branching questions, extract structured fields with a clinically adapted speech model, score those fields deterministically in code rather than in the model, and escalate to a licensed clinician on every ambiguous case. Either use the Voice Agent API at a flat $4.50/hr for the whole conversation loop, or assemble streaming transcription with your own LLM and TTS. See medical transcription use cases for the surrounding pieces.

Why can't the voice agent diagnose the patient?

Because diagnosis is clinical practice and it needs a licensed clinician, and because an agent that only collects and routes has bounded failure modes that a nurse can catch. Write hard refusals into the system prompt — no diagnosis, no medication advice, no severity opinions — and test them with adversarial cases, since callers will ask directly.

How does the agent handle medical terminology patients use?

Medical Mode (domain: "medical-v1") adapts the model to clinical vocabulary and cuts entity errors 87% versus the base model, at a 3.2% Missed Entity Rate. On top of that, agent_context applies earlier conversation to later turns — it cut word error rate 10.2% across 20,000 voice agent files, and detailed context cut medical-term entity errors 43% — which handles callers who name a drug once and refer to it loosely afterward.

What happens when a caller reports a red-flag symptom?

Screen for red flags before the protocol tree, not after, so an emergency is caught in the first twenty seconds. On a hit, the agent reads a fixed escalation script and transfers immediately — no further questions, no scoring. Escalation should be a single tool call that works even if the rest of the flow has failed.

How is this different from a scheduling voice agent?

Stakes and accuracy requirements. A scheduling agent that mishears a date creates an inconvenience; a triage agent that mishears "left arm" or fails to escalate creates a safety event. That drives everything: clinical entity accuracy, deterministic scoring, mandatory human escalation paths, and audit logging detailed enough for a retrospective review.

How does AssemblyAI handle HIPAA and PHI?

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and acts as a business associate under HIPAA. Supporting that: PHI redaction across audio and transcripts, SOC 2 Type 2 audited infrastructure, and self-hosted or EU-resident deployment where required. Get the BAA in place before your pilot, since pilot calls carry real PHI — start at the BAA FAQ.

‍