Build an AI medical note-taker with one API

Open the hood on most medical note-takers and you find the same wiring every time: an accurate speech-to-text model for clinical vocabulary, speaker separation to tell the clinician from the patient, PII redaction for sensitive details, and an LLM layer that turns the raw conversation into a structured SOAP note. Four capabilities. And, for most teams, four vendors — four bills, four sets of docs, and four separate security reviews for your legal team.

It doesn't have to be four. In the demo above, the entire pipeline gets built live against a single API, and the whole thing runs on one Python script with no third-party dependencies. Here's how it works, and how to build the same thing yourself.

What a medical note-taker actually needs

A clinical intake conversation is a hard audio problem before it's an AI problem. The vocabulary is dense with conditions, medications, and dosages that general models mishear. Two people are talking, often over each other, and downstream you need to know who said what. And the deliverable a clinician actually wants isn't a transcript — it's a note.

So the pipeline breaks into four jobs:

• Medical-grade transcription that gets clinical terms right.
• Speaker diarization to separate the nurse and the patient.
• Entity detection to surface conditions, medications, and dosages inline.
• Structured note generation to produce a Subjective / Objective / Assessment / Plan (SOAP) note.

With AssemblyAI, the first three come from a single transcription request, and the fourth is one more call to the same API with the same key.

One request, three speech features

The transcription step turns on Medical Mode (the medical-v1 domain, powered by Universal-3.5 Pro), speaker labels, and entity detection at once. There's no separate model to swap in for healthcare audio — you add one parameter:

{
  "audio_url": "<recording url>",
  "domain": "medical-v1",
  "speaker_labels": true,
  "speakers_expected": 2,
  "entity_detection": true
}

Medical Mode is the piece doing the clinical heavy lifting. It reduces the missed-entity rate on drugs, conditions, procedures, and clinical terms by roughly 20% versus the base model, and it posts the lowest missed-medical-entity rate across benchmarked providers. If you have prior-visit context, contextual prompting can push that further — feeding a patient's earlier note cut missed medical terms by 31% in an internal healthcare test.

In the Python demo, that request is fired and then polled until the transcript is ready:

def transcribe(api_key):
    job = api_request(TRANSCRIPT_ENDPOINT, api_key, {
        "audio_url": AUDIO_URL,
        "domain": "medical-v1",          # Medical Mode: clinical-grade vocabulary
        "speaker_labels": True,          # speaker diarization
        "speakers_expected": 2,          # nurse + patient
        "entity_detection": True,        # conditions, medications, dosages
    })
    transcript_id = job["id"]

    while True:
        result = api_request(f"{TRANSCRIPT_ENDPOINT}/{transcript_id}", api_key)
        if result["status"] == "completed":
            return result
        if result["status"] == "error":
            sys.exit(result.get("error"))
        time.sleep(3)

One response comes back carrying all three signals: the transcript text, per-utterance speaker labels, and a list of detected entities with their types and character offsets.

Mapping speakers to nurse and patient

Diarization gives you anonymous labels — speaker A and speaker B. In an intake visit, the clinician drives the interview, so the demo maps the speaker who asks the most questions to Nurse and the other to Patient, breaking ties by who spoke first:

def map_speakers(utterances):
    question_counts = {}
    first_speaker = utterances[0]["speaker"]
    for utt in utterances:
        question_counts.setdefault(utt["speaker"], 0)
        question_counts[utt["speaker"]] += utt["text"].count("?")

    speakers = sorted(question_counts,
                      key=lambda s: (-question_counts[s], s != first_speaker))
    mapping = {speakers[0]: "Nurse"}
    for other in speakers[1:]:
        mapping[other] = "Patient"
    return mapping

It's a small heuristic, but it's the kind of glue you'd otherwise be writing across two vendors' outputs. Here it's a few lines over one response.

Highlighting medical entities inline

Entity detection returns each condition, medication, injury, procedure, and dosage with a type. The demo wraps them in highlighted spans so a reviewer can scan the transcript and see the clinically relevant terms at a glance. The entity types it surfaces map cleanly to what a note reviewer cares about:

Generating the SOAP note with LLM Gateway

The finished, speaker-labeled dialogue goes to AssemblyAI's LLM Gateway — the same API, the same key — with a prompt that asks for a structured note as JSON:

POST https://llm-gateway.assemblyai.com/v1/chat/completions

{
  "model": "claude-sonnet-4-6",
  "messages": [{ "role": "user", "content": prompt }],
  "max_tokens": 2000
}

The prompt hands the model the mapped dialogue and asks for exactly four keys — subjective, objective, assessment, plan — with a hard instruction to base every statement strictly on the transcript. The result renders into the SOAP panel next to the highlighted transcript, and the whole page is a single self-contained output.html: transcript with inline entity highlights on the left, structured note on the right.

That's the entire pipeline — transcribe, separate speakers, highlight entities, generate the note — in one script, against one API, billed on one account. The full example is on GitHub.

Working with protected health information

Healthcare audio means protected health information (PHI), and that raises a question every clinical team asks before writing a line of code. AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process PHI. AssemblyAI is considered a business associate under HIPAA, and we offer a Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI.

The part that matters for a small team: you can sign the BAA on a regular pay-as-you-go account. No enterprise contract, no spend commitment, no premium pricing — and it covers every product in this pipeline, from Medical Mode transcription to LLM Gateway. One vendor, one bill, and one security review for your legal team instead of four.

Why one API changes the build

The technical win here isn't any single feature — it's the collapse from four integrations to one. Time-to-value drops because there's a single set of docs and a single auth model. And because AssemblyAI ships docs and MCP servers built for AI coding agents, an agent can scaffold a working SOAP-note generator for you in a single session, which is exactly what the demo shows. Healthcare teams building ambient scribes and clinical documentation tools — the kind of work companies like Sully AI and Heidi Health do on AssemblyAI — start from that same one-API foundation.

Frequently asked questions

What is Medical Mode in AssemblyAI?

Medical Mode is a domain setting ("domain": "medical-v1") on Universal-3.5 Pro that tunes transcription for clinical vocabulary — conditions, medications, dosages, and procedures. You enable it by adding one parameter to a standard transcription request; there's no separate model to integrate. It adds $0.15/hr on top of the base rate.

Can one API really replace transcription, diarization, entities, and note generation?

Yes. A single AssemblyAI transcription request returns the transcript, speaker labels, and detected entities together, and the LLM Gateway generates the structured note using the same API key. The demo does all four with one Python script and no third-party dependencies.

Is AssemblyAI HIPAA-compliant?

AssemblyAI is considered a business associate under HIPAA and offers a Business Associate Addendum (BAA), which is required under HIPAA when a vendor processes PHI on your behalf. The BAA is available on a standard pay-as-you-go account with no enterprise contract or spend commitment.

What is a SOAP note and how is it generated here?

A SOAP note is a standard clinical format with four sections — Subjective, Objective, Assessment, and Plan. In this build, the speaker-labeled transcript is sent to LLM Gateway with a prompt that returns those four sections as JSON, grounded strictly in the conversation.

How accurate is AssemblyAI on medical terminology?

Medical Mode reduces the missed-entity rate on drugs, conditions, procedures, and clinical terms by roughly 20% versus the base model and posts the lowest missed-medical-entity rate across benchmarked providers. Passing prior-visit context via contextual prompting cut missed medical terms by a further 31% in an internal healthcare test.

How much does it cost to run?

Transcription with Universal-3.5 Pro is $0.21/hr, and Medical Mode adds $0.15/hr, for $0.36/hr combined. Speaker diarization and entity detection are add-ons on the async API, and LLM Gateway tokens are billed separately. Pricing is pay-as-you-go, billed per second, with no minimums.

‍