How to build an AI scribe for therapy sessions that writes progress notes

Ask any therapist where their evenings go and you'll hear the same answer: notes. The session ends, the next client is already in the waiting room, and the progress note gets pushed to a stack that gets cleared at 9pm. It's the tax on doing the work.

That's why AI scribes have taken off in behavioral health. Search "AI scribe for physical therapy" or "therapy progress notes" and you'll find a dozen finished products that record a session and hand back a SOAP note. They work. But if you're building clinical software — an EHR, a practice-management tool, a specialized scribe for psychiatry or PT — you don't want to bolt on someone else's black box. You want to build the scribe into your own product, control the note format, and own the accuracy.

So here's the build. This is a practical guide to turning a therapy session into a structured progress note using accurate medical transcription, speaker separation, and an LLM to draft the note. The pieces are simpler than you'd think — the hard part is getting the clinical vocabulary right, and that's exactly where most generic dictation falls down.

Why therapy documentation is the right job for an AI scribe

Progress notes are structured, repetitive, and high-stakes — the perfect shape for automation, and a terrible use of a clinician's time.

Most therapy notes follow one of a few formats. SOAP (Subjective, Objective, Assessment, Plan) is the default across medicine. Behavioral health often uses DAP (Data, Assessment, Plan) or BIRP (Behavior, Intervention, Response, Plan). Physical therapy leans on SOAP with functional measures baked into the Objective section. Whatever the format, the note has to reflect what was actually said, use the right clinical terms, and hold up if it's ever audited.

That last part is where generic speech-to-text breaks. A model that transcribes "sertraline" as "sir Tolline," or drops the dosage, or smears the clinician's question into the patient's answer, produces a note that a clinician has to rewrite from scratch — which defeats the entire point. The bar for a clinical scribe isn't "good enough transcription." It's transcription accurate enough on medication names, conditions, and procedures that the clinician trusts the draft.

What a good therapy scribe has to get right

Four things, in order of how badly they hurt when you get them wrong:

Clinical-term accuracy. Drug names, dosages, diagnoses, procedures, and assessment instruments (PHQ-9, GAD-7) have to land correctly. This is non-negotiable and it's the hardest part.

Speaker separation. A therapy transcript is a conversation. The note needs to know which statements came from the clinician and which from the client, or the Assessment section turns into mush.

Structured output. The transcript has to become a note in the clinician's format — not a wall of text. That's a job for an LLM with a tight prompt.

PHI handling. The moment you process a real session, you're handling protected health information. That has to be designed in from the first line of code, not patched on before launch.

Let's build each piece.

Architecture overview

The pipeline is four stages:

1. Capture the session audio (uploaded recording for async, or a live stream for real-time).
2. Transcribe with Medical Mode on, so clinical terms come through accurately, with speaker labels.
3. Generate the progress note by passing the diarized transcript to an LLM with a format-specific prompt.
4. Review — the clinician edits and signs. The AI drafts; the human approves.

You can run this async (record the session, process it after) or in real time (transcribe live, draft the note as the session ends). Most scribes start async because it's simpler and cheaper, then add a real-time mode. We'll build async first.

Step 1: Accurate clinical transcription

Start with the piece everything else depends on. Use Universal-3.5 Pro, our flagship pre-recorded model, and switch on Medical Mode by adding a single parameter: domain: "medical-v1". That one flag tunes the model for clinical audio — no separate model, no new endpoint.

import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"

config = aai.TranscriptionConfig(
    domain="medical-v1",     # enables Medical Mode (EN, ES, DE, FR)
    speaker_labels=True,     # diarization: who said what
    language_code="en",
)

transcript = aai.Transcriber().transcribe("./therapy-session.wav", config)
if transcript.error:
    raise RuntimeError(transcript.error)

diarized_text = "\n".join(f"Speaker {u.speaker}: {u.text}" for u in transcript.utterances)

Why this matters in numbers: Medical Mode reaches a 3.2% Missed Entity Rate on clinical terms — roughly 20% fewer missed medical entities than Universal-3.5 Pro alone, and about 87% fewer entity errors than the base model. Across benchmarked providers (Deepgram, Speechmatics, AWS, Google), it posts the lowest Missed Entity Rate. Those aren't vanity metrics — a missed medication name or a wrong dosage is precisely the failure that makes a clinician stop trusting the tool.

There's a second lever most builders miss: contextual prompting. If you have the client's prior-visit note, feed it to the model as context. In an internal healthcare test, priming the model with a patient's earlier note cut missed medical terms by 31% — even when the prior note was from an earlier visit. For a repeat client on a stable medication regimen, that's a big accuracy jump for almost no work. You can also pass practice-specific vocabulary (uncommon drug names, local program names, assessment tools) through keyterm prompting.

Medical Mode runs in English, Spanish, German, and French, on both pre-recorded and streaming models.

Step 2: Separate the speakers

You already turned on speaker_labels above, so the transcript comes back segmented by speaker. Universal-3.5 Pro ships the most accurate speaker diarization we've released — it produces the transcript and the speaker turns together, which is what lets it handle the rapid back-and-forth and short interjections that fill a real therapy session.

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

You'll get a clean, turn-by-turn transcript — Speaker A (clinician) and Speaker B (client). That structure is what the note generator needs to attribute observations correctly.

Step 3: Generate the progress note

Now turn the diarized transcript into a note. Pass it to an LLM through the LLM Gateway — one API that routes to GPT, Claude, or Gemini — with a prompt that specifies the exact format you want. Here's a SOAP example; swap the template for DAP or BIRP by changing the prompt.

prompt = """You are a clinical documentation assistant. Using the therapy
session transcript below, draft a progress note in SOAP format.
... (unchanged) ...
Transcript:
{transcript}
"""

# Send the diarized transcript to the LLM Gateway (OpenAI-compatible REST)
import requests

resp = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers={"authorization": "YOUR_API_KEY", "content-type": "application/json"},
    json={
        "model": "claude-sonnet-4-6",
        "messages": [{"role": "user", "content": prompt.format(transcript=diarized_text)}],
        "max_tokens": 2000,
    },
)
resp.raise_for_status()
note = resp.json()["choices"][0]["message"]["content"]
print(note)

Two prompt patterns make the output clinically usable. First, constrain the model to the transcript — "only include information present" — so it doesn't invent a diagnosis that was never discussed. Second, tell it to flag ambiguity for review rather than guessing. The goal is a strong first draft the clinician can approve in a minute, not an autonomous note-writer. Keep the human in the loop; the clinician signs.

Handling PHI the right way

A live therapy session is about as sensitive as data gets, so build for it from the start.

AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure PHI is appropriately safeguarded. The BAA is self-serve — you can sign it from the Data Controls page on any paid plan, at no extra cost, without a sales call. That means you can build and test against real requirements without a procurement cycle.

A few more controls worth wiring in:

• PHI redaction across audio and transcripts, if you need to strip identifiers before storage or downstream processing.
• EU data residency via api.eu.assemblyai.com — same price, data stays in the EU — if you serve clients under GDPR.
• Self-hosted deployment inside your own cloud if your compliance posture requires the audio never to leave your environment.
• SOC 2 Type 2 as the baseline security attestation.

Keep compliance framing accurate in your own product copy too: you sign a BAA and process PHI as a business associate — not "HIPAA-certified." (For public references, point customers to the BAA FAQ.)

Who's already building this

Behavioral-health software teams are doing exactly this today. NovoPsych, a mental-health platform used by tens of thousands of clinicians, runs an AI scribe ("NovoNote") for session documentation. Psychiatry EHRs lean on Medical Mode specifically because it's the biggest accuracy lever for drug names and dosages — the terms a psychiatric note lives or dies on. Teams displacing older transcription vendors consistently cite Medical Mode accuracy as the reason for the switch.

The pattern is the same across all of them: they didn't buy a finished scribe. They built the scribe their clinicians needed, on infrastructure accurate enough to trust.

Getting started

You can have the transcription half of this running this afternoon. Grab a free API key, transcribe a sample session with domain: "medical-v1" and speaker_labels=True, and pipe the result into an LLM with the SOAP prompt above. Then layer in contextual prompting and PHI controls as you move toward production.

The finished-product scribes on the market are good at being finished products. If you're building clinical software, the advantage is owning the whole pipeline — the format, the vocabulary, the accuracy, and the trust that comes with it.

Frequently asked questions

What is an AI scribe for therapy?

An AI scribe for therapy is software that listens to a session, transcribes it, and drafts a structured progress note the clinician reviews and signs — removing most of the manual documentation after each appointment. In behavioral health and physical therapy it typically outputs SOAP, DAP, or BIRP notes. You can adopt a finished scribe product or build your own on speech-to-text and an LLM, as this guide shows.

Can AI write therapy progress notes?

Yes. An AI scribe transcribes the session, separates the speakers, and drafts a structured note (SOAP, DAP, or BIRP) that the clinician reviews and signs. The AI produces the first draft; the clinician stays in the loop and approves the final note.

What note formats can an AI scribe generate?

Any structured format you prompt for — SOAP (Subjective, Objective, Assessment, Plan), DAP (Data, Assessment, Plan), and BIRP (Behavior, Intervention, Response, Plan) are the common ones in behavioral health and physical therapy. Because the note is generated by an LLM from the transcript, you control the template.

How much does an AI scribe cost to run?

If you build your own, the cost is mostly transcription plus LLM usage rather than a per-seat subscription. On AssemblyAI, Universal-3.5 Pro is $0.21/hr and Medical Mode adds $0.15/hr — $0.36/hr combined for clinical-grade transcription, billed per second with no minimums — plus your LLM tokens for note generation. That usage-based model often scales more predictably than per-clinician pricing on finished scribe products.

Is an AI therapy scribe safe for protected health information?

‍ It can be, if it's built correctly. AssemblyAI is considered a business associate under HIPAA and offers a standard Business Associate Addendum (BAA) for customers processing PHI, available self-serve at no extra cost. Combine the BAA with PHI redaction, SOC 2 Type 2 infrastructure, and — if needed — EU data residency or self-hosted deployment.

How accurate is AI transcription for clinical and medical terms?

With Medical Mode enabled, AssemblyAI reaches a 3.2% Missed Entity Rate on clinical terms — about 20% fewer missed medical entities than the base model, roughly 87% fewer entity errors, and the lowest Missed Entity Rate across benchmarked providers. Feeding the model a patient's prior-visit note as context cut missed medical terms by a further 31% in internal testing.

Can I build my own AI medical scribe instead of buying one?

Yes — that's the approach in this guide. Rather than adopting a finished scribe product, you build transcription (with Medical Mode via domain: "medical-v1"), speaker diarization, and LLM note generation into your own application, so you control the note format, the vocabulary, and the accuracy.

‍