Build a real-time medical transcription analysis app with AssemblyAI and LLM Gateway

This tutorial shows you how to build a real-time medical transcription system that captures clinical conversations, separates speakers, and automatically generates SOAP notes. You'll create a streaming application that processes audio as it happens, giving clinicians instant transcription with proper clinical documentation format.

You'll use AssemblyAI's Python SDK for real-time speech-to-text with Medical Mode and multichannel speaker separation, combined with AssemblyAI's LLM Gateway to transform raw transcripts into structured clinical notes. The implementation includes microphone audio capture, streaming transcription, and FHIR integration for Electronic Health Records.

What is AI medical transcription and when to use streaming

AI medical transcription is software that converts clinical conversations into written notes automatically. You record a conversation, the AI transcribes the speech, then structures it into proper documentation like SOAP notes without manual typing.

The technology works in four steps. First, it records audio from visits in real time. Second, speech-to-text models convert spoken words into text using medical vocabulary. Third, AI models identify medical information and organize it into clinical note formats. Finally, clinicians review and approve the generated notes before adding them to records.

Streaming transcription processes audio as it happens, giving you text within milliseconds. You'd use this during live visits when you need immediate feedback and real-time documentation.

Choose streaming when you need instant results during interactions. Choose async when you're processing recorded dictations after appointments.

How AI medical transcription works in practice

The workflow transforms spoken conversations into structured notes through a specific process. Your system captures audio from clinical conversations using secure recording. Speech recognition then converts those words into text using specialized medical vocabulary trained on clinical conversations.

AI models process this raw text to identify medical entities like symptoms, medications, and diagnoses. The system organizes this into standard formats like SOAP notes with proper structure.

• Time savings: Eliminates manual note-taking during visits
• Reduced documentation burden: Cuts evening charting sessions that cause burnout
• Improved accuracy: Creates consistent notes with proper medical terminology
• Better focus: Lets clinicians maintain eye contact instead of typing

Major healthcare systems like Kaiser Permanente and UC San Francisco have already implemented AI transcription, but the technology has limitations you need to understand. Some systems create "hallucinations," inventing text that was never spoken. This happens during pauses, with background noise, or when processing unclear speech. Domain-optimized transcription like Medical Mode reduces this risk by anchoring the transcript in accurate medical entity recognition before any LLM step.

Key considerations for your implementation:

• Accuracy verification: Always require human review before finalizing notes
• Compliance: Use systems that offer encryption and a Business Associate Addendum (BAA)
• Consent: Get explicit permission before recording conversations
• Bias prevention: Choose AI models trained on diverse populations

Build real-time medical transcription with Python SDK

You'll build a streaming system that captures conversations, processes multichannel audio for speaker separation, and generates SOAP notes. This uses AssemblyAI's Python SDK for real-time transcription with Medical Mode, plus AssemblyAI's LLM Gateway to turn transcripts into clinical documentation.

Set up Python environment and streaming session

Create a new project and install the dependencies. You need Python 3.8 or higher.

# Create project directory
mkdir medical-transcription-app
cd medical-transcription-app

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install assemblyai pyaudio requests python-dotenv

Create a .env file to store your API key securely. You only need one key, since both transcription and SOAP generation run through AssemblyAI:

ASSEMBLYAI_API_KEY=your_assemblyai_api_key_here

Set up the streaming configuration with medical-optimized settings. Note the domain="medical-v1" parameter, which turns on Medical Mode:

import os
import threading
from queue import Queue, Empty

import pyaudio
from dotenv import load_dotenv

from assemblyai.streaming.v3 import (
    StreamingClient,
    StreamingClientOptions,
    StreamingParameters,
    StreamingEvents,
    TurnEvent,
    StreamingError,
)

# Load environment variables
load_dotenv()

# Audio capture configuration (these are PyAudio settings, NOT streaming params)
SAMPLE_RATE = 16000        # 16 kHz
CHANNELS = 1               # microphone capture is mono
CHUNK_SIZE = 800           # 50 ms at 16 kHz — AssemblyAI's recommended chunk size
AUDIO_FORMAT = pyaudio.paInt16


class MedicalTranscriptionStream:
    def __init__(self):
        self.transcript_queue = Queue()
        self.is_running = False
        self.pyaudio = pyaudio.PyAudio()
        self.transcriber = None

    def start_streaming(self):
        """Initialize streaming session with Medical Mode"""
        params = StreamingParameters(
            sample_rate=SAMPLE_RATE,
            encoding="pcm_s16le",
            speech_model="u3-rt-pro",     # REQUIRED; promptable model needed for keyterms_prompt
            domain="medical-v1",          # Medical Mode — boosts clinical terminology
            format_turns=True,            # punctuated, formatted final transcripts
            keyterms_prompt=[             # reinforce specific terms (max 1000)
                "hypertension", "diabetes", "metformin", "systolic", "diastolic",
            ],
            speaker_labels=True,          # separate provider vs. patient (remove if unsupported on your plan)
            max_speakers=2,
        )

        # v3 takes options (with the API key) and registers handlers via .on() —
        # it does NOT accept on_turn=/on_error= constructor callbacks.
        self.transcriber = StreamingClient(
            StreamingClientOptions(api_key=os.getenv("ASSEMBLYAI_API_KEY"))
        )
        self.transcriber.on(StreamingEvents.Turn, self.on_transcription_turn)
        self.transcriber.on(StreamingEvents.Error, self.on_error)

        self.transcriber.connect(params)
        self.is_running = True

This configuration optimizes for medical conversations at 16kHz. Medical Mode (domain="medical-v1") is the key setting: it's domain-optimized for medical entity recognition, built on Universal-3 Pro and Universal-3 Pro Streaming, and posts a 3.2% Missed Entity Rate, the lowest across benchmarked providers. The streaming endpoint is wss://streaming.assemblyai.com/v3/ws. Medical Mode is a $0.15/hr add-on on top of Universal-3 Pro at $0.21/hr (so $0.36/hr total), and supports English, Spanish, German, and French. Real-time speaker separation requires multichannel audio where each channel represents a different speaker.

Capture and stream audio with speaker separation

Implement audio capture from your microphone with real-time streaming to AssemblyAI. For speaker separation in real time, provide multichannel audio where each channel represents a different speaker.

    def capture_microphone_audio(self):
        """Capture audio from microphone and send to transcription service"""
        stream = self.pyaudio.open(
            format=AUDIO_FORMAT,
            channels=CHANNELS,
            rate=SAMPLE_RATE,
            input=True,
            frames_per_buffer=CHUNK_SIZE,
        )
        print("Listening... Press Ctrl+C to stop")
        try:
            while self.is_running:
                audio_data = stream.read(CHUNK_SIZE, exception_on_overflow=False)
                try:
                    self.transcriber.stream(audio_data)  # .stream() accepts raw PCM bytes
                except Exception:
                    break  # session closed underneath us
        finally:
            stream.stop_stream()
            stream.close()

    def on_transcription_turn(self, client, turn: TurnEvent):
        """Handle incoming transcription turns. The handler receives (client, event)."""
        # v3 emits a Turn for partial AND final segments — only buffer finals.
        if not turn.end_of_turn or not turn.transcript:
            return

        self.transcript_queue.put({
            "text": turn.transcript,                    # field is 'transcript', not 'text'
            "speaker": turn.speaker_label,              # 'A'/'B'/None when diarization is on
            "confidence": turn.end_of_turn_confidence,  # there is no turn.confidence
            "start_ms": turn.words[0].start if turn.words else None,  # no turn.start_time
            "is_final": turn.end_of_turn,
        })

        # Display real-time transcription
        print(f"\n[Speaker {turn.speaker_label or '?'}] {turn.transcript}")

    def on_error(self, client, error: StreamingError):
        print(f"Streaming error: {error}")

    def stop_streaming(self):
        """Gracefully terminate the streaming session."""
        self.is_running = False
        if self.transcriber:
            try:
                self.transcriber.disconnect(terminate=True)
            except Exception:
                pass

For single-channel audio, all transcription appears without speaker labels. To achieve speaker separation in real time, configure multichannel audio where each participant uses a separate channel. Post-processing with async transcription can provide speaker diarization from single-channel recordings.

Process transcripts and generate SOAP notes with LLM Gateway

Transform raw transcripts into structured SOAP notes using AssemblyAI's LLM Gateway. This keeps everything on one platform and one API key: AssemblyAI handles both the transcription accuracy and the LLM step, which matters when you're processing PHI under a single BAA.

import json
import requests
from datetime import datetime


class SOAPNoteGenerator:
    GATEWAY_URL = "https://llm-gateway.assemblyai.com/v1/chat/completions"

    # Structured-output schema guarantees valid JSON back (Gemini/OpenAI only — not Claude).
    SOAP_SCHEMA = {
        "type": "object",
        "properties": {
            "subjective": {"type": "string"},
            "objective": {"type": "string"},
            "assessment": {"type": "string"},
            "plan": {"type": "string"},
        },
        "required": ["subjective", "objective", "assessment", "plan"],
        "additionalProperties": False,
    }

    def __init__(self):
        self.api_key = os.getenv("ASSEMBLYAI_API_KEY")
        self.conversation_buffer = []

    def process_transcript(self, transcript_data):
        """Add transcript to conversation buffer"""
        if transcript_data["is_final"]:
            self.conversation_buffer.append(transcript_data)

    def format_conversation(self):
        """Format buffered conversation for the LLM"""
        lines = []
        for entry in self.conversation_buffer:
            lines.append(f"Speaker {entry.get('speaker') or '?'}: {entry['text']}")
        return "\n".join(lines)

    def generate_soap_note(self):
        """Convert conversation transcript to SOAP format using LLM Gateway"""
        if not self.conversation_buffer:
            return None

        instructions = (
            "Convert the following medical conversation into a SOAP note. "
            "Map the speakers to the healthcare provider and the patient.\n\n"
            "- SUBJECTIVE: complaints, symptoms, and history\n"
            "- OBJECTIVE: vital signs, physical exam findings, test results\n"
            "- ASSESSMENT: diagnosis or differential diagnoses\n"
            "- PLAN: treatment plan, medications, follow-up\n\n"
            f"Conversation:\n{self.format_conversation()}"
        )

        try:
            response = requests.post(
                self.GATEWAY_URL,
                headers={"Authorization": self.api_key},  # raw API key — NO 'Bearer' prefix
                json={
                    "model": "gemini-2.5-pro",            # a valid LLM Gateway model ID
                    "messages": [
                        {"role": "system", "content": "You are a medical documentation assistant that writes accurate SOAP notes."},
                        {"role": "user", "content": instructions},
                    ],
                    "temperature": 0.3,                   # Lower temperature for consistency
                    "max_tokens": 1500,
                    "response_format": {
                        "type": "json_schema",
                        "json_schema": {"name": "soap_note", "strict": True, "schema": self.SOAP_SCHEMA},
                    },
                },
                timeout=60,
            )
            response.raise_for_status()
            soap_json = json.loads(response.json()["choices"][0]["message"]["content"])
            return self.format_soap_note(soap_json)
        except Exception as e:
            print(f"Error generating SOAP note: {e}")
            return None

    def format_soap_note(self, soap_json):
        """Format SOAP note for display"""
        return f"""
SOAP NOTE - {datetime.now().strftime('%Y-%m-%d %H:%M')}
{'=' * 50}

SUBJECTIVE:
{soap_json.get('subjective', 'No subjective data recorded')}

OBJECTIVE:
{soap_json.get('objective', 'No objective data recorded')}

ASSESSMENT:
{soap_json.get('assessment', 'No assessment recorded')}

PLAN:
{soap_json.get('plan', 'No plan recorded')}

{'=' * 50}
Generated by AI - Requires clinician review
""".strip()

LLM Gateway processes the conversation flow and extracts medical information for each SOAP section. It identifies symptoms for Subjective, vitals for Objective, and treatment plans for Plan. Because Medical Mode already corrected the terminology upstream, the LLM isn't reasoning over misheard drug names, which makes the resulting note more reliable.

Integrate with EHR systems

Connect your transcription system to Electronic Health Records through standard FHIR APIs. Most modern EHRs support FHIR for data exchange.

import base64
import requests
from typing import Optional


class EHRIntegration:
    def __init__(self, ehr_base_url: str, api_key: str):
        self.base_url = ehr_base_url.rstrip("/")
        self.headers = {
            "Authorization": f"Bearer {api_key}",  # the EHR's own token — distinct from the AssemblyAI key
            "Content-Type": "application/json",
        }

    @staticmethod
    def encode_base64(text: str) -> str:
        return base64.b64encode(text.encode("utf-8")).decode("utf-8")

    def create_clinical_note(self, patient_id: str, soap_note: str,
                             provider_id: str) -> Optional[str]:
        """Submit SOAP note to EHR system via FHIR API"""
        fhir_document = {
            "resourceType": "DocumentReference",
            "status": "current",
            "type": {
                "coding": [{
                    "system": "http://loinc.org",
                    "code": "11488-4",
                    "display": "Consultation note",
                }]
            },
            "subject": {"reference": f"Patient/{patient_id}"},
            "author": [{"reference": f"Practitioner/{provider_id}"}],
            "date": datetime.now().isoformat(),  # datetime imported in Code Block 5
            "content": [{
                "attachment": {
                    "contentType": "text/plain",
                    "data": self.encode_base64(soap_note),
                }
            }],
        }
        try:
            response = requests.post(
                f"{self.base_url}/DocumentReference",
                json=fhir_document,
                headers=self.headers,
                timeout=30,
            )
            if response.status_code == 201:
                return response.json().get("id")
            print(f"EHR returned {response.status_code}: {response.text}")
            return None
        except Exception as e:
            print(f"Error submitting to EHR: {e}")
            return None

Run this script and it captures audio from your microphone, transcribes it in real time with Medical Mode, and generates SOAP notes through LLM Gateway when you stop recording.

Security and compliance requirements

Healthcare transcription systems must meet strict compliance requirements to protect patient information, especially as healthcare data breaches affected 276+ million patients in 2024 with an average cost of $9.77 million per incident. HIPAA mandates technical, physical, and administrative safeguards for any system handling Protected Health Information (PHI).

Your AI transcription system needs specific security measures:

AssemblyAI enables covered entities and their business associates subject to HIPAA to use AssemblyAI services to process PHI. AssemblyAI is considered a business associate under HIPAA and offers a Business Associate Addendum (BAA) required under HIPAA. Running both transcription and SOAP generation through AssemblyAI means PHI stays under a single BAA instead of being split across multiple vendors.

Consent is another critical requirement. You must get explicit permission before recording conversations.

• Document consent: Store signed consent forms with records
• Allow withdrawal: People can revoke permission at any time
• Clear disclosure: Explain how recordings will be used and stored
• State compliance: Check local laws for additional consent requirements

The accuracy challenge needs special attention. Some systems hallucinate, inventing text that was never spoken. This risk rises during pauses, background noise, or unclear speech, which is why Medical Mode and human review both matter.

• Confidence scoring: Flag transcriptions with low confidence
• Human review: Require clinician approval before finalizing documentation
• Quality monitoring: Regular audits of transcription accuracy
• Fallback procedures: Manual documentation when AI fails

Final words

This real-time medical transcription system captures conversations, processes them with streaming speech-to-text and Medical Mode, separates speakers using multichannel audio, and transforms raw transcripts into structured SOAP notes through LLM Gateway. The workflow eliminates manual note-taking during visits and reduces post-appointment documentation time through automated clinical note generation.

Frequently asked questions

How accurate is AI medical transcription for complex medical terminology?

With a domain-optimized model it's strong. AssemblyAI's Medical Mode (domain="medical-v1") posts a 3.2% Missed Entity Rate, the lowest across benchmarked providers, and roughly 20% fewer missed medical entities than Universal-3 Pro alone. See assemblyai.com/benchmarks.

How does Medical Mode compare to Deepgram Nova-3 Medical, Amazon Transcribe Medical, and Whisper?

Medical Mode posts a 3.2% MER. Deepgram Nova-3 Medical comes in around 8.7% MER and AWS Transcribe Medical around 24.4% MER on the same benchmark. General-purpose models like Whisper aren't tuned for medical entity recognition and miss more clinical terms. Full methodology is at assemblyai.com/benchmarks.

How do I handle PHI and BAAs for AI medical transcription?

AssemblyAI enables covered entities and their business associates subject to HIPAA to use AssemblyAI services to process PHI. AssemblyAI is considered a business associate under HIPAA and offers a Business Associate Addendum (BAA) required under HIPAA. Sign a BAA before processing PHI, encrypt data in transit and at rest, keep audit logs, and capture consent before recording. Running transcription and the LLM step through AssemblyAI keeps PHI under one BAA.

Does Medical Mode redact PHI and PII?

AssemblyAI offers PII redaction that automatically detects and removes identifiers from transcripts, and you control which policies apply and how values are substituted. Combine it with automatic data deletion and time-to-live settings to limit how long data persists.

What languages does Medical Mode support?

English, Spanish, German, and French, for both pre-recorded and streaming transcription.

Which EHR systems support FHIR integration for AI transcription notes?

Major EHR systems like Epic, Cerner, Allscripts, and athenahealth support FHIR R4 for document integration. Use DocumentReference resources to submit clinical notes with proper patient linking and encounter context.

Want a deeper walkthrough of evaluating Voice AI for healthcare, including clinical accuracy, speech understanding, and BAAs? See the AssemblyAI ambient AI scribes guide at https://www.assemblyai.com/ambient-ai-scribes-guide.