How to build an AI scribe for therapy sessions

A therapist finishes her last session at 4:40pm. The note is drafted before she stands up, and the two corrections she wants — move the medication change into the plan, soften the risk language — she makes by talking to it while walking to the kitchen. No keyboard. That's the target, and every piece of it is buildable today with three APIs.

The architecture has three phases: ambient capture during the session, speaker attribution that survives real conversation, and note generation the clinician can revise by speaking. This walkthrough covers all three with working code, then the hardening work that separates a demo from something you can put in front of clinicians.

One design note before the code. Ambient means the clinician isn't operating the tool during the session — no dictation, no commands, no glancing at a screen. That constraint drives every decision downstream, and it's the reason turn detection and diarization get more attention here than the LLM prompt does.

Architecture

Two phases, split by when they run.

During the session: audio streams to Universal-3.5 Pro Realtime with Medical Mode on. You get speaker-labeled turns as they happen, which you buffer rather than display — nobody needs a live transcript in a therapy room, and a screen in the room changes the session.

After the session: the buffered transcript goes through PHI redaction, then to an LLM for structured note generation, then into a Voice Agent session where the clinician reviews and revises conversationally.

Why stream at all if the note is generated afterward? Because the alternative is recording a 50-minute file, uploading it, and waiting. Streaming gives you the transcript the instant the session ends, which is the difference between a note the clinician signs in the room and one they get to at 8pm. If that latency doesn't matter to your product, use pre-recorded Universal-3.5 Pro instead — it's cheaper at $0.21/hr versus $0.45/hr base, and marginally more accurate.

What you need

Python 3.9 or later, the AssemblyAI Python SDK, and PyAudio for microphone capture. An API key from the docs quickstart. That's the whole dependency list.

‍

pip install assemblyai pyaudio

Step 1: stream the session with Medical Mode

The streaming endpoint is wss://streaming.assemblyai.com/v3/ws. Two things to get right in the config: the model ID and the turn detection.

import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"

# Streaming takes speech_model (singular).
# Pre-recorded takes speech_models (plural).
streaming_config = {
    "speech_model": "universal-3-5-pro",
    "domain": "medical-v1",
    "speaker_labels": True,
    "voice_focus": "far-field",
    "mode": "max_accuracy",
    "prompt": (
        "Behavioral health therapy session. Client is on sertraline "
        "and lamotrigine. Prior session addressed sleep-onset insomnia "
        "and workplace conflict."
    ),
}

Three parameters deserve explanation.

voice_focus takes near-field or far-field. Ambient capture in a therapy room is almost always far-field — a device on a side table, not a headset. Setting this correctly costs nothing and skipping it is the most common accuracy own-goal we see.

mode takes min_latency, balanced, or max_accuracy. For an ambient scribe where nothing is displayed live, use max_accuracy. Latency is irrelevant if no human is waiting on the output.

prompt is the underused one. Across 20,000 voice agent calls, scenario context cut medical-term entity errors 24% and detailed context cut them 43%. agent_context is a different parameter: it carries the reply your voice agent just spoke, so it does not apply to an ambient scribe. Populate it from the client's prior note. On pre-recorded audio the equivalent move — feeding a patient's prior-visit note as context — cut missed medical terms by 31% in an internal healthcare test.

Turn detection and therapeutic silence

Here's the part that breaks naive implementations. Therapy contains long deliberate pauses. A client sits with a question for eight seconds before answering, and that silence is clinically meaningful — it isn't the end of a turn.

Default turn detection - min_turn_silence 128ms and max_turn_silence 1280ms on the balanced preset - is right for a voice agent and wrong for a therapy room. Raise min_turn_silence to around 800ms and max_turn_silence to around 3600ms for this use case, and test against real session audio rather than a scripted recording. Getting this wrong doesn't lose words — it fragments turns, which corrupts speaker attribution, which corrupts the note.

Medical Mode itself is one parameter: domain: "medical-v1", adding $0.15/hr on top of the base model. It's the same setting on streaming and pre-recorded, and it doesn't require switching models, so you keep diarization and everything else you configured. It delivers roughly 20% fewer missed medical entities and 87% fewer entity errors against the same model without it, at a 3.2% Missed Entity Rate — the lowest across the providers we've benchmarked. Methodology is on the benchmarks page.

Diarization is the other half. Universal-3.5 Pro's is the most accurate we've shipped and is optimized for cpWER rather than DER — it's scored on getting each speaker's words right rather than on how much audio time landed in the right bucket. In streaming it supports up to 10 speakers with revision, so early attributions get corrected as the session continues. For couples and family work that revision is what makes live diarization viable at all.

Step 2: redact before you persist

The transcript you buffered contains names, dates, employers, and addresses. Redact before it touches durable storage, not after.

import assemblyai as aai

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro"],
    domain="medical-v1",
    speaker_labels=True,
).set_redact_pii(
    policies=[
        aai.PIIRedactionPolicy.person_name,
        aai.PIIRedactionPolicy.medical_condition,
        aai.PIIRedactionPolicy.date_of_birth,
        aai.PIIRedactionPolicy.phone_number,
        aai.PIIRedactionPolicy.location,
    ],
    redact_audio=True,
)

redact_audio=True is the part teams forget. PHI redaction runs across both audio and transcripts, and a redacted transcript stored next to an unredacted recording buys you very little in a compliance review.

One judgment call: redacting medical_condition removes exactly the content your note needs. The pattern that works is generating the note from the unredacted transcript in memory, then persisting only the redacted version and the note itself. That keeps your durable store clean without crippling generation.

Step 3: generate the note

Send the speaker-labeled transcript to an LLM through the LLM Gateway with a schema. Structured output, not a prose prompt — the schema is what keeps the model inside the transcript.

import json
import requests

DAP_SCHEMA = {
    "type": "object",
    "properties": {
        "data": {"type": "string"},
        "assessment": {"type": "string"},
        "plan": {"type": "string"},
        "medications_discussed": {
            "type": "array",
            "items": {"type": "string"},
        },
        "risk_discussed": {"type": "boolean"},
        "risk_detail": {"type": "string"},
    },
    "required": ["data", "assessment", "plan", "risk_discussed"],
}

prompt = f"""You are drafting a DAP progress note from a therapy session
transcript. Use only what is in the transcript. If a field was not
discussed, return an empty string. Never infer clinical content.

Prior session note (for continuity and terminology):
{prior_note}

Transcript with speaker labels:
{speaker_labeled_transcript}
"""

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers={"authorization": "YOUR_API_KEY"},
    json={
        "model": "claude-sonnet-4-6",
        "messages": [{"role": "user", "content": prompt}],
        "response_format": {
            "type": "json_schema",
            "json_schema": {"name": "dap_note", "schema": DAP_SCHEMA},
        },
    },
)

note = json.loads(response.json()["choices"][0]["message"]["content"])

Two rules that matter more than prompt wording.

Keep speaker labels in the input. Flatten the transcript and the model starts guessing who said what, which throws away everything diarization did for you.

Force empty over inferred. The risk_discussed boolean exists so the model can say "not discussed" instead of writing "denies SI/HI" about a conversation that never happened. A fabricated negative in a behavioral health note is the single worst output this pipeline can produce.

Step 4: let the clinician revise by talking

This is the step that makes the whole thing feel different. Instead of rendering the draft note in a text editor, open a Voice Agent API session with the note as context. The clinician says what they want changed, the agent applies it, reads back the modified section, and waits.

The Voice Agent API is a flat $4.50/hr with one WebSocket replacing the STT, LLM, and TTS stack you'd otherwise assemble and keep in sync. For a two-minute review conversation that's about 15 cents.

What makes it work in practice is scoping. Give the agent the note, the transcript, and permission to edit sections — not permission to add clinical content. "Add that she's tapering the sertraline" should succeed because it's in the transcript. "Add that she denied suicidal ideation" should fail if that exchange never happened.

The infrastructure side of this is a solved problem now. LiveKit, which runs voice infrastructure for a lot of these pipelines, put it this way:

"We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms."

— David Zhao, Co-founder at LiveKit

What this changes for clinicians

Three effects, in descending order of how often clinicians mention them.

Eye contact comes back. No laptop between clinician and client. In behavioral health that isn't a convenience — the therapeutic relationship is the intervention, and a screen in the room degrades it.

Documentation stops following them home. The note is drafted and signed within minutes of the session ending rather than accumulating into an evening of catch-up work.

Notes get consistent. Structured generation produces the same sections in the same order every time, which is what makes payer review and internal quality sampling tractable.

Track time-to-edit as your production metric. It collapses transcription accuracy, diarization accuracy, and template fit into one number the clinician actually feels, and unlike a benchmark it moves when your product gets better for your specific users.

Hardening checklist

The gap between this walkthrough and production is entirely in this section.

Get the BAA signed first. AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI, which makes us a business associate under HIPAA for that data. We're SOC 2 Type 2. Do this before the first real recording moves — see the BAA FAQ and the Business Associate Addendum.

Encrypt in transit and at rest, including the audio buffer you're holding during the session. That buffer is PHI in memory and it's the piece people forget.

Require clinician sign-off. Nothing enters the record unsigned. The signature is the legal basis for the note and the reason clinicians will trust the tool.

Log everything. Who accessed which transcript, when, and what they changed. Audit trails are a product feature, not a database setting.

Set a retention policy you can defend and enforce it in code. "We keep audio indefinitely" is not a policy.

Capture consent and make recording state visible in the room.

For teams with data-residency requirements, EU processing and self-hosted deployment are both available.

Extending it

Swap the schema and you've got SOAP or BIRP instead of DAP — the pipeline doesn't change, only the JSON shape. Our post on AI scribes for therapy progress notes covers how each format maps onto a diarized transcript.

For multilingual practices: the base model code-switches natively across 18 languages with no configuration, while Medical Mode's clinical tuning covers English, Spanish, German, and French. Keep those two facts separate when you decide which populations you can serve well.

Pricing across the whole pipeline, on the pricing page: $0.45/hr streaming base or $0.21/hr pre-recorded, plus $0.15/hr for Medical Mode, plus LLM cost for generation, plus $4.50/hr for the Voice Agent review session. A 50-minute session with a two-minute voice review lands well under a dollar.

Where this goes

The scribe category is about to stop being about scribing. Once the transcript is reliable and the note writes itself, the remaining work is judgment — and judgment is where the clinician's time is actually worth something.

The interesting version of this product isn't a better note generator. It's a system that noticed the client mentioned a medication change three sessions ago that never made it into the treatment plan, and asks about it. That's not a transcription feature or an LLM feature. It's what becomes possible when a year of encounters is diarized, searchable, and trustworthy — which is the actual thing you built when you built this pipeline. The note was just the first thing it was good for. More on the clinical accuracy foundations in our post on behavioral health documentation clinicians trust.

Frequently asked questions

How do I build an ambient AI scribe for therapy?

Stream session audio to Universal-3.5 Pro Realtime with domain: "medical-v1" and speaker labels on, redact PHI before persisting, generate a structured note from the speaker-labeled transcript with an explicit JSON schema, and let the clinician revise it through a Voice Agent session. The three APIs are speech-to-text, LLM Gateway, and Voice Agent; total cost lands under a dollar per session.

Which model should I use for ambient clinical capture?

Universal-3.5 Pro Realtime for streaming, at $0.45/hr base plus $0.15/hr for Medical Mode. Use pre-recorded Universal-3.5 Pro at $0.21/hr instead if you don't need the transcript the moment the session ends — it's cheaper and marginally more accurate. Set voice_focus to far-field for ambient room capture and mode to max_accuracy when nothing is displayed live.

How does the scribe handle long therapeutic pauses?

By retuning turn detection. Default turn detection - min_turn_silence 128ms, max_turn_silence 1280ms on the balanced preset - is correct for a voice agent and wrong for a therapy room where an eight-second silence is clinically meaningful. Raise min_turn_silence to around 800ms and max_turn_silence to around 3600ms, and validate against real session recordings — the failure mode isn't lost words, it's fragmented turns that corrupt speaker attribution.

How does AssemblyAI handle HIPAA and PHI?

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI, which makes us a business associate under HIPAA for that data. We're SOC 2 Type 2, PHI redaction runs across both audio and transcripts, and EU data residency and self-hosted deployment are available for teams with stricter requirements.

How accurate is speaker separation in a two-person therapy session?

Universal-3.5 Pro's diarization is the most accurate we've shipped and is optimized for cpWER rather than DER, which means it's scored on the words in each speaker's transcript rather than on audio time attributed. It captures short turns and overlapped speech — the interruptions that break most clinical pipelines. Streaming supports up to 10 speakers with revision, so couples and family sessions work too.

Can the clinician edit the note without typing?

Yes — that's what the Voice Agent step is for. A flat $4.50/hr session with one WebSocket replacing STT, LLM, and TTS lets the clinician say what they want changed and hear the revised section read back. Scope the agent to editing sections rather than adding clinical content, so it can't assert something the transcript doesn't support.