How to build an AI medical scribe with AssemblyAI

AI medical scribes are changing healthcare documentation, but building one that works in clinical settings takes more than basic transcription. You need accurate medical terminology capture, reliable speaker identification, and privacy safeguards that protect sensitive data.

This tutorial walks you through building a functional AI medical scribe in Python. You'll start with basic transcription and progressively add Medical Mode, speaker identification, PII redaction, SOAP note generation, and automatic data deletion. By the end, you'll have a working prototype that handles real-world clinical scenarios.

What makes a production-ready medical scribe?

Before diving into code, it helps to understand what separates a proof-of-concept from something you can actually deploy.

Essential features checklist

Modern Speech AI APIs like AssemblyAI handle most of this complexity. Here's how.

On the accuracy side, AssemblyAI's Medical Mode posts a 3.2% Missed Entity Rate (MER), the lowest MER across benchmarked providers including Deepgram, Speechmatics Enhanced Medical, AWS Transcribe Medical, and Google. You can see the full methodology and numbers at assemblyai.com/benchmarks.

On the privacy side, AssemblyAI provides SOC 2 Type 2 and ISO 27001:2022 certifications. AssemblyAI enables covered entities and their business associates subject to HIPAA to use AssemblyAI services to process PHI. AssemblyAI is considered a business associate under HIPAA and offers a Business Associate Addendum (BAA) required under HIPAA.

Step 1: Basic transcription setup

Start with a simple transcription request. This establishes the foundation before layering on medical-specific features.

import requests
import time
import dotenv
import os

dotenv.load_dotenv()

base_url = "https://api.assemblyai.com"
headers = {"authorization": os.getenv("AAI_API_KEY")}

Step 2: Turning on Medical Mode

This is the single most important change for clinical accuracy. Medical Mode is domain-optimized for medical entity recognition, built on Universal-3 Pro and Universal-3 Pro Streaming. It catches terminology errors before they propagate into SOAP notes, discharge summaries, or downstream LLMs.

You activate it with one parameter. Set domain to "medical-v1" on a Universal-3 Pro request:

data = {
    "audio_url": "https://storage.googleapis.com/aai-web-samples/doctor-patient-convo.mp4",
    "speech_model": "universal-3-pro",
    "domain": "medical-v1"
}

That one parameter delivers roughly 20% fewer missed medical entities versus Universal-3 Pro alone, and a 3.2% MER overall. Medical Mode is a $0.15/hr add-on on top of Universal-3 Pro at $0.21/hr, so a Medical Mode async transcript runs $0.36/hr. It works on both Universal-3 Pro (async) and Universal-3 Pro Streaming, and supports English, Spanish, German, and French. The same domain="medical-v1" parameter also covers veterinary and other non-human-patient terminology, so the scribe isn't limited to human clinical settings.

Step 3: Adding speaker identification

Medical documentation requires knowing who said what. Was it the clinician prescribing medication or the person in the visit requesting it? Speaker diarization automatically separates different speakers in the conversation.

AssemblyAI's speaker diarization identifies when different people are speaking and labels them Speaker A, Speaker B, and so on. To separate speakers further, you can use AssemblyAI's Speaker Identification feature to match specific names to speakers. Here, we'll map these labels to roles like "Doctor" and "Patient".

data = {
    "audio_url": "https://storage.googleapis.com/aai-web-samples/doctor-patient-convo.mp4",
    "speech_model": "universal-3-pro",
    "domain": "medical-v1",
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "speaker_identification": {
                "speaker_type": "role",
                "known_values": ["Doctor", "Patient"]
            }
        }
    }
}

Speaker diarization works best when each speaker talks for at least 30 seconds uninterrupted, though the model handles real-world conversations with cross-talk and short phrases.

Step 4: Protecting privacy with PII redaction

Medical conversations contain protected health information (PHI): names, birthdates, social security numbers, addresses. If this data appears in your transcripts and you store them on third-party servers, you're potentially violating privacy regulations.

AssemblyAI's PII redaction automatically identifies and removes personally identifiable information from transcripts. You configure what gets redacted and how. You can also add a keyterms prompt to lock in the spelling and formatting of specific drug names or procedures.

data = {
    "audio_url": "https://storage.googleapis.com/aai-web-samples/doctor-patient-convo.mp4",
    "speech_model": "universal-3-pro",
    "domain": "medical-v1",
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "speaker_identification": {
                "speaker_type": "role",
                "known_values": ["Doctor", "Patient"]
            }
        }
    },
    "redact_pii": True,
    "redact_pii_policies": ["person_name", "organization", "occupation"],
    "redact_pii_sub": "hash",
    "keyterms_prompt": ["Tramadol"]
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_json = response.json()
transcript_id = transcript_json["id"]
polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        for utterance in transcript["utterances"]:
            print(f"{utterance['speaker']}: {utterance['text']}")
        break
    else:
        time.sleep(3)

Step 5: Generating SOAP notes with LLM Gateway

Raw transcripts are useful, but clinicians need structured documentation. SOAP notes (Subjective, Objective, Assessment, Plan) are the clinical standard for organizing a visit.

This is where large language models come in. AssemblyAI's LLM Gateway provides a unified interface to various LLMs. Call it with your existing AssemblyAI API key.

prompt = "Create a SOAP note summary of the consultation."

llm_gateway_data = {
    "model": "gemini-3-pro-preview",
    "messages": [
        {"role": "user", "content": f"{prompt}\n\nTranscript: {transcript['text']}"}
    ]
}
llm_gateway_response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json=llm_gateway_data
)
print(llm_gateway_response.json()["choices"][0]["message"]["content"])

The LLM processes the conversation and outputs structured clinical notes. Because Medical Mode already cleaned up the terminology upstream, the LLM isn't reasoning over misheard drug names, so the SOAP note is more reliable. The clinician gets documentation ready for the EHR without manual transcription.

Step 6: Automatic data deletion

The most reliable way to protect sensitive data is to ensure it doesn't persist on third-party infrastructure. After retrieving your transcript and generating SOAP notes, delete the transcript from AssemblyAI's servers.

delete_request = requests.delete(polling_endpoint, headers=headers).json()
delete_llm_gateway_response = requests.delete(
    f"https://llm-gateway.assemblyai.com/v1/chat/completions/{llm_gateway_response.json()['reque
st_id']}",
    headers=headers
)

transcript = requests.get(polling_endpoint, headers=headers).json()
print(transcript)

Data retention for LLM Gateway

When using LLM Gateway with an executed Business Associate Addendum (BAA) and Anthropic or Google inference models, AssemblyAI provides zero data retention for inputs and outputs. The LLM Gateway processes requests ephemerally. Your transcript text and the generated SOAP notes are not stored beyond the immediate API call. Only minimal metadata is retained for logging and billing purposes.

Time-to-live for transcripts

For additional protection of transcript data, AssemblyAI offers time-to-live (TTL) settings. As of November 26, 2024, customers with a signed BAA automatically have a 3-day TTL applied to all transcripts. This TTL is subject to change. Transcripts automatically delete after this period, even if you forget to send the delete request manually.

Putting it all together

Here's the complete implementation with all features enabled, including Medical Mode:

import requests
import time
import dotenv
import os

dotenv.load_dotenv()

base_url = "https://api.assemblyai.com"
headers = {"authorization": os.getenv("AAI_API_KEY")}

data = {
    "audio_url": "https://storage.googleapis.com/aai-web-samples/doctor-patient-convo.mp4",
    "speech_model": "universal-3-pro",
    "domain": "medical-v1",
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "speaker_identification": {
                "speaker_type": "role",
                "known_values": ["Doctor", "Patient"]
            }
        }
    },
    "redact_pii": True,
    "redact_pii_policies": ["person_name", "organization", "occupation"],
    "redact_pii_sub": "hash",
    "keyterms_prompt": ["Tramadol"]
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_json = response.json()
transcript_id = transcript_json["id"]
polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        for utterance in transcript["utterances"]:
            print(f"{utterance['speaker']}: {utterance['text']}")
        break
    else:
        time.sleep(3)

prompt = "Create a SOAP note summary of the consultation."

llm_gateway_data = {
    "model": "gemini-3-pro-preview",
    "messages": [
        {"role": "user", "content": f"{prompt}\n\nTranscript: {transcript['text']}"}
    ]
}
llm_gateway_response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json=llm_gateway_data
)
print(llm_gateway_response.json()["choices"][0]["message"]["content"])

delete_request = requests.delete(polling_endpoint, headers=headers).json()
delete_llm_gateway_response = requests.delete(
    f"https://llm-gateway.assemblyai.com/v1/chat/completions/{llm_gateway_response.json()['request_id']}",
    headers=headers
)

transcript = requests.get(polling_endpoint, headers=headers).json()
print(transcript)

Why this approach works for medical AI

Building a medical scribe takes more than transcription accuracy. It's about creating a system that fits into clinical workflows while respecting privacy.

This implementation handles the real-world challenges:

• Medical terminology accuracy through Medical Mode (domain="medical-v1")
• Clear speaker attribution with speaker diarization
• Privacy protection via PII redaction and automatic deletion
• Structured output through LLM Gateway-generated SOAP notes
• Data minimization by removing all traces from third-party servers

The result is a functional prototype you can test with clinical audio. From here, add real-time transcription with Universal-3 Pro Streaming for live consultations, EHR integration, or custom SOAP note templates for different specialties.

Next steps

• AssemblyAI's Speech-to-Text documentation
• PII redaction policies and configuration
• LLM Gateway for medical use cases

[CTA — Sign-up] Get an API key and start building

Create a free AssemblyAI account, grab your API key, and run this medical scribe with Medical Mode in minutes.

Button: Get API key → https://www.assemblyai.com/dashboard/signup

Frequently asked questions

How does AssemblyAI handle PHI and PII in medical transcripts?

AssemblyAI's PII redaction automatically detects and removes identifiers like names, dates, and IDs from transcripts, and you control which policies apply and how values are substituted. Combined with automatic data deletion and time-to-live settings, this keeps protected health information from persisting on third-party servers.

Is AssemblyAI HIPAA-compliant, and do you sign a BAA?

AssemblyAI enables covered entities and their business associates subject to HIPAA to use AssemblyAI services to process PHI. AssemblyAI is considered a business associate under HIPAA and offers a Business Associate Addendum (BAA) required under HIPAA. To process PHI in production, request a BAA from the AssemblyAI team.

How accurate is Medical Mode compared to Deepgram Nova-3 Medical and Amazon Transcribe Medical?

Medical Mode posts a 3.2% Missed Entity Rate (MER), the lowest across benchmarked providers. For comparison, Deepgram Nova-3 Medical comes in around 8.7% MER and AWS Transcribe Medical around 24.4% MER. See the full methodology at assemblyai.com/benchmarks.

How does Medical Mode compare to Whisper for clinical audio?

General-purpose models like Whisper aren't tuned for medical entity recognition and can miss or invent clinical terms. Medical Mode is domain-optimized for medical terminology and benchmarks at a 3.2% MER, with about 20% fewer missed medical entities than Universal-3 Pro alone. Full benchmark numbers are at assemblyai.com/benchmarks.

What languages does Medical Mode support?

Medical Mode supports English, Spanish, German, and French for both pre-recorded and streaming transcription.

How much does Medical Mode cost?

Medical Mode is a $0.15/hr add-on. On top of Universal-3 Pro at $0.21/hr, a Medical Mode transcript runs $0.36/hr. You activate it with a single parameter, domain="medical-v1".

Building a healthcare AI product and need help with accuracy benchmarks, BAAs, or production deployment? Reach out to the AssemblyAI team at sales@assemblyai.com.