Now live New models for transcription and text-to-speech. AssemblyAI-STT-3.7 AssemblyAI-TTS-1
Insights & Use Cases

Extract phone call insights with LLMs in Python

Build a Python call analytics pipeline: transcribe with AssemblyAI-STT-3.7, redact PII, then extract CRM-ready JSON insights with LLM Gateway.

Abstract green cylinder illustration

Written by

Ryan O'Connor

Published on

15 September 2026

Call analytics is the practice of turning recorded phone conversations into structured, queryable data: transcribing the audio, attributing each turn to a speaker, and extracting fields like summaries, action items, sentiment, entities and compliance flags. This guide builds a working pipeline in Python using AssemblyAI speech-to-text with the AssemblyAI-STT-3.7 model for the transcript, Speech Understanding for the structured fields, and the LLM Gateway for the judgment calls a fixed schema can’t make. You’ll finish with schema-checked JSON you can write straight into a CRM, and a clear picture of what each call costs.

What is call analytics?

Call analytics converts unstructured conversation audio into fields a system can act on. The input is a recording; the output is a row: who called, what they wanted, what was promised, how it went, and what happens next.

The case for automating it comes down to arithmetic. A contact center generating a few thousand calls a month produces hundreds of hours of audio. Human QA teams sample a small fraction of that, and most reviewed calls are chosen because someone complained, which biases the sample toward the failures you already know about. An automated pipeline reads every call at a marginal cost measured in cents, which changes the questions you can ask. “Was this call bad?” becomes “which of the 340 calls that mentioned a competitor last month ended without a follow-up?”

That only works if the transcript is right. Every downstream classifier, phrase detector and scorecard inherits the errors in the text it was given.

Real-time vs. historical call analytics

Type Processing Best for
Real-time Streaming transcription over a WebSocket while the call is live Live agent assist, in-call compliance alerts, supervisor escalation
Historical Async processing of the recording after the call ends Post-call summaries, CRM writeback, QA scoring, batch trend analysis

This guide covers the historical path, which is where most conversation intelligence work lives. If you’re building the batch ingestion and orchestration layer around it, the companion post on building a call center analytics pipeline in Python goes deeper on queueing, storage and dashboards than this one does.

How a call analytics pipeline works

  1. Ingest the recording from your telephony provider, recording store or upload endpoint.
  2. Transcribe it with speaker diarization so each turn is attributed to a speaker.
  3. Redact PII before the text leaves your processing boundary.
  4. Extract structured fields (entities, sentiment, topics) with purpose-built models, and bespoke fields with an LLM.
  5. Write the result somewhere queryable: a CRM record, a warehouse table, a dashboard.

Steps 2 through 4 are what the rest of this post builds.

Getting started

You need Python 3.10 or later and an AssemblyAI API key. The complete working code is on GitHub at kelsey-aai/call-insights-llm-gateway-python, and every snippet below comes from it.

Setting up your environment

git clone https://github.com/kelsey-aai/call-insights-llm-gateway-python
cd call-insights-llm-gateway-python

python -m venv .venv
source .venv/bin/activate   # On Windows: .venv\Scripts\activate
pip install -r requirements.txt

Set your API key as an environment variable instead of hardcoding it. The repo ships a .env.example you can copy, or export the variable directly:

export ASSEMBLYAI_API_KEY=<your-key>

# or copy the template and load it:
cp .env.example .env
set -a; source .env; set +a

The sample call used throughout this post is a custom home builder inquiry: https://storage.googleapis.com/aai-web-samples/Custom-Home-Builder.mp3. Every script exits with a clear message if ASSEMBLYAI_API_KEY isn’t set.

Transcribing the call with speech-to-text

AssemblyAI-STT-3.7 is the recommended model for pre-recorded audio, at $0.21/hr. Pass its ID, assemblyai-stt-3-7, in the speech_models list (the async parameter is plural). It replaces Universal-3.5 Pro, which was deprecated on November 17, 2026; the migration guide covers what changes when you switch.

import os
import sys

import assemblyai as aai

api_key = os.environ.get("ASSEMBLYAI_API_KEY")
if not api_key:
    sys.exit("Set ASSEMBLYAI_API_KEY before running this script.")
aai.settings.api_key = api_key

AUDIO_URL = "https://storage.googleapis.com/aai-web-samples/Custom-Home-Builder.mp3"

config = aai.TranscriptionConfig(
    speech_models=["assemblyai-stt-3-7"],
    speaker_labels=True,
)

transcript = aai.Transcriber(config=config).transcribe(AUDIO_URL)

if transcript.status == aai.TranscriptStatus.error:
    sys.exit(f"Transcription failed: {transcript.error}")

print("Model used:", transcript.speech_model_used)

Three things matter here for call audio specifically.

Language coverage. AssemblyAI-STT-3.7 supports 32 languages with native code switching, so it handles a speaker moving between languages mid-sentence without you declaring a language up front or splitting the audio. That comes up constantly on real customer calls: bilingual households, bilingual agents and support queues in multilingual markets switch languages inside a single utterance. If your queue includes languages outside those 32, set speech_models to ["assemblyai-stt-3-7", "universal-2"]. AssemblyAI-STT-3.7 handles every language it supports, and Universal-2 ($0.15/hr) covers the rest of its 99. With language_detection on and only assemblyai-stt-3-7 in the list, there is no fallback. The model selection docs have the details, and transcript.speech_model_used tells you which model processed each file.

Disfluencies. AssemblyAI-STT-3.7 keeps disfluencies by default: filler words, repetitions, false starts and contractions like “gonna”. That’s useful for QA scoring and talk-pattern analysis. If you’d rather hand the LLM clean text, set disfluencies=False on the config.

Diarization. speaker_labels=True adds async speaker diarization at +$0.02/hr and gives you transcript.utterances, each with a speaker label. On a call, what matters is whether the right words landed against the right speaker, because every per-speaker metric (talk ratio, who committed to what, who raised the objection) depends on it.

Format the utterances into a speaker-labeled transcript before anything downstream touches it:

def format_transcript(transcript: aai.Transcript) -> str:
    return "\n".join(
        f"Speaker {u.speaker}: {u.text}" for u in transcript.utterances
    )


conversation = format_transcript(transcript)
print(conversation[:800])

If diarization is new to you, this explainer covers how it works and where it breaks.

Redacting PII before you analyze

Call recordings routinely contain card numbers, CVVs, expiration dates, home addresses, account IDs and Social Security numbers, usually read aloud digit by digit. If you send raw call transcripts to an LLM, a warehouse or a QA dashboard without redacting first, you’ve expanded the scope of every system downstream without anyone noticing.

Redaction runs as part of transcription, so the sensitive strings never exist in the transcript you store. The setup lines (imports, API key, AUDIO_URL) are the same as in the transcription script:

config = aai.TranscriptionConfig(
    speech_models=["assemblyai-stt-3-7"],
    speaker_labels=True,
    redact_pii=True,
    redact_pii_sub=aai.PIISubstitutionPolicy.entity_name,
    redact_pii_policies=[
        aai.PIIRedactionPolicy.credit_card_number,
        aai.PIIRedactionPolicy.credit_card_cvv,
        aai.PIIRedactionPolicy.credit_card_expiration,
        aai.PIIRedactionPolicy.banking_information,
        aai.PIIRedactionPolicy.account_number,
        aai.PIIRedactionPolicy.us_social_security_number,
        aai.PIIRedactionPolicy.phone_number,
        aai.PIIRedactionPolicy.email_address,
        aai.PIIRedactionPolicy.location_address,
    ],
    redact_pii_audio=True,
    redact_pii_audio_options={"override_audio_redaction_method": "silence"},
)

transcript = aai.Transcriber(config=config).transcribe(AUDIO_URL)

if transcript.status == aai.TranscriptStatus.error:
    sys.exit(f"Transcription failed: {transcript.error}")

print(transcript.text[:500])

# The redacted recording is a separate file. Download it within 24 hours.
print("Redacted audio:", transcript.get_redacted_audio_url())

Five parameters do the work:

  • redact_pii turns text redaction on (+$0.08/hr).
  • redact_pii_policies is the list of categories to remove. Be explicit. Redacting person_name will also strip the agent’s name, which you may want to keep for QA scoring, so this example leaves it out.
  • redact_pii_sub controls what replaces the removed span. hash writes #######; entity_name writes a readable token like [CREDIT_CARD_NUMBER], which is usually better for LLM input because it preserves the shape of the sentence.
  • redact_pii_audio produces a copy of the recording with the sensitive spans removed (+$0.05/hr). Use it if the audio itself is retained for training or dispute resolution. The redacted file is available for 24 hours, so download it promptly.
  • redact_pii_audio_options with override_audio_redaction_method set to "silence" replaces PII with silence instead of the default beep.

Run redaction before the LLM step so the raw values stay out of every system except the transcription request itself.

Run a call through the pipeline in your browser

Upload a real recording, switch on diarization, PII redaction and Speech Understanding, and see the JSON before you write a line of code.

Try playground

Speech Understanding vs. an LLM prompt

The original version of this tutorial extracted entities and sentiment by asking an LLM to find them. That works, but it’s the wrong default. AssemblyAI ships purpose-built models for these tasks, and they’re deterministic, typed and priced per audio hour. All of them run on AssemblyAI-STT-3.7 transcripts.

Feature Parameter Price What it gives you on a call
Entity detection entity_detection +$0.08/hr Names, addresses, dates, dollar amounts, products and more, each with its type and timestamps
Sentiment analysis sentiment_analysis +$0.02/hr Per-sentence positive / neutral / negative with confidence and speaker, so you can chart sentiment across the call
Topic detection iab_categories +$0.15/hr Consistent IAB topic labels across your whole corpus, which is what makes cross-call trend analysis possible
Key phrases auto_highlights +$0.01/hr The salient phrases in the conversation, with counts, for search and tagging
PII redaction (text) redact_pii +$0.08/hr Card numbers, SSNs, addresses removed from the stored transcript
PII redaction (audio) redact_pii_audio +$0.05/hr The same spans removed from the recording itself
Content moderation content_safety +$0.15/hr Flags sensitive content for escalation and agent-safety reporting

All of them are flags on the same transcription request:

config = aai.TranscriptionConfig(
    speech_models=["assemblyai-stt-3-7"],
    speaker_labels=True,
    redact_pii=True,
    redact_pii_sub=aai.PIISubstitutionPolicy.entity_name,
    redact_pii_policies=[aai.PIIRedactionPolicy.credit_card_number],
    entity_detection=True,
    sentiment_analysis=True,
    iab_categories=True,
    auto_highlights=True,
)

transcript = aai.Transcriber(config=config).transcribe(AUDIO_URL)

if transcript.status == aai.TranscriptStatus.error:
    sys.exit(f"Transcription failed: {transcript.error}")

for entity in transcript.entities:
    print(f"{entity.entity_type.value}: {entity.text}")

negative = [
    s for s in transcript.sentiment_analysis
    if s.sentiment == aai.SentimentType.negative
]
print(f"{len(negative)} negative sentences")

for label, relevance in transcript.iab_categories.summary.items():
    print(f"{label} ({relevance:.2f})")

for phrase in transcript.auto_highlights.results[:5]:
    print(f"{phrase.text} (mentioned {phrase.count}x)")

When to use which

Use a Speech Understanding feature when the field is the same on every call. Structured features are deterministic: the same audio produces the same labels, with the same enum values, every time. They’re typed, so you can put them in a database column without a validation layer. They come with character offsets and timestamps, so you can jump to the moment in the recording. They’re billed per hour of audio, so cost scales with volume in a way you can forecast. And they don’t drift when someone edits a prompt.

Use an LLM prompt when the output is bespoke or requires judgment. “Did the agent follow our five-step objection-handling framework?” is not an entity type. Neither is “summarize this call in the format our sales ops team uses,” or “what is the single most likely reason this deal stalled?” Anything that depends on your internal rubric, changes as your business changes, or requires reasoning across the whole conversation belongs in a prompt.

In practice, do both. Redact first, extract the structured fields with Speech Understanding, then pass the redacted transcript plus the structured fields into the LLM as context. The LLM does less work, hallucinates less because the facts are handed to it, and you spend fewer tokens.

Extracting call insights with the LLM Gateway

The LLM Gateway gives you one API and one bill for 55+ models from OpenAI, Anthropic, Google and others, plus models AssemblyAI hosts itself. It speaks the standard chat completions shape at https://llm-gateway.assemblyai.com/v1/chat/completions, and it authenticates with the same AssemblyAI API key, so the OpenAI SDK or a plain HTTP request both work. This post uses requests to keep every field visible.

First, a helper that turns the Speech Understanding output into compact context for the prompt:

def summarize_fields(transcript: aai.Transcript) -> str:
    """Turn the Speech Understanding output into compact context for the LLM."""
    entities = sorted({f"{e.entity_type.value}: {e.text}" for e in transcript.entities or []})
    sentiment = Counter(s.sentiment.value for s in transcript.sentiment_analysis or [])
    return json.dumps(
        {"entities": entities, "sentence_sentiment_counts": dict(sentiment)},
        indent=2,
    )

Then the prompt and the call itself. qwen3.5-4b-32k-fast is a fast model AssemblyAI hosts, and it’s a good fit for free-text summaries at volume:

GATEWAY_URL = "https://llm-gateway.assemblyai.com/v1/chat/completions"
INSIGHTS_MODEL = "qwen3.5-4b-32k-fast"
EXTRACTION_MODEL = "claude-haiku-5-5"

PROMPT = """ROLE
You are a sales operations analyst reviewing an inbound customer call.

CONTEXT
Below is a speaker-labeled, PII-redacted transcript. Structured fields already
extracted from the audio are provided alongside it. Treat those fields as ground
truth and do not contradict them.

INSTRUCTION
Produce a concise summary, the concrete action items each party committed to,
the customer's primary objection if one was raised, and a recommended next step.
If something was not discussed, say so rather than inferring it.

TRANSCRIPT
{conversation}

STRUCTURED FIELDS
{structured}
"""


def gateway_headers() -> dict:
    return {"authorization": os.environ["ASSEMBLYAI_API_KEY"]}


def analyze(conversation: str, structured: str, model: str = INSIGHTS_MODEL) -> str:
    response = requests.post(
        GATEWAY_URL,
        headers=gateway_headers(),
        json={
            "model": model,
            "max_tokens": 1000,
            "messages": [
                {
                    "role": "user",
                    "content": PROMPT.format(
                        conversation=conversation, structured=structured
                    ),
                }
            ],
        },
        timeout=60,
    )
    response.raise_for_status()
    return response.json()["choices"][0]["message"]["content"]

The ROLE / CONTEXT / INSTRUCTION structure does real work. Telling the model to treat the Speech Understanding output as ground truth cuts down on invented details, because the entity list and sentiment counts are no longer things the model has to guess at.

Choosing a model

Model ID Runs on Price per 1M tokens (input / output, global) Best for on a call pipeline
qwen3.5-4b-32k-fast AssemblyAI $0.10 / $0.50 · 32,768-token context Free-text summaries, transcript cleanup and formatting at volume. No response_format or tools
claude-haiku-5-5 Anthropic via AWS Bedrock $0.10 / $0.50 · +10% in-region Structured JSON with response_format, tool calling, EU processing
gpt-6-luna OpenAI (served via AWS Bedrock by default) $0.10 / $0.50 · +10% in-region The choose-a-model pick for extracting action items from 2-minute calls (October 29, 2026 run)
claude-sonnet-4-6 Anthropic via AWS Bedrock $3 / $15 · +10% in-region QA rubrics, multi-step reasoning, and steps that need tool calling
gemini-3.7-flash Google via Vertex $0.75 / $3.75 · +10% in-region Long-context batch analysis, with a context window above 1M tokens

Check supported parameters before you pick. The Models page lists qwen3.5-4b-32k-fast with max_tokens, temperature and stream only, so it can’t take response_format or tools. That’s why the structured step below runs on claude-haiku-5-5. For a ranked shortlist by task and audio length, the choose-a-model guide scores every Gateway model on call transcripts, and the tool calling docs cover function-calling patterns.

The per-call arithmetic

Model choice looks like a rounding error on one call and like the entire budget on a hundred thousand, so work it out concretely.

A 12-minute two-party call is 0.2 hours of audio and produces a transcript of roughly 2,400 tokens. Add a ~300-token prompt and you’re sending about 2,700 input tokens and getting back about 400.

Per call, with AssemblyAI-STT-3.7, diarization, entity detection, sentiment, PII text redaction, and claude-haiku-5-5 for the structured JSON:

Line item Rate One 12-min call 10,000 calls/month (2,000 hrs)
Transcription (AssemblyAI-STT-3.7) $0.21/hr $0.042 $420
Speaker diarization +$0.02/hr $0.004 $40
Entity detection +$0.08/hr $0.016 $160
Sentiment analysis +$0.02/hr $0.004 $40
PII text redaction +$0.08/hr $0.016 $160
LLM JSON extraction (claude-haiku-5-5, in-region) $0.11 / 1M in, $0.55 / 1M out ~$0.0005 ~$5.20
Total ~$0.083 ~$825

LLM Gateway prices on the pricing page are for global routing. A request without "model_region": "global" is processed in-region (US or EU) at 10% more, and the table uses that in-region rate. The Models page lists no regional surcharge for qwen3.5-4b-32k-fast.

The useful way to reason about the LLM line is per million tokens against your own volume. At 10,000 calls a month you’re sending roughly 27M input tokens and receiving 4M output tokens. Every $1.00 per 1M input tokens adds about $27/month, and every $1.00 per 1M output tokens adds about $4/month. Switching the extraction step to claude-sonnet-4-6 ($3 / 1M input, $15 / 1M output, plus 10% in-region) moves that line to roughly $155/month, still under a fifth of the $820 of audio processing. Spend on a larger model for the calls where judgment matters, and use the fast models everywhere else.

Start building your call analytics pipeline

Get an API key and run your first transcription, redaction and LLM extraction in a few minutes.

Sign up free

Getting structured JSON output for CRM integration

Asking a model nicely for JSON gets you JSON most of the time. A pipeline that writes to a CRM needs it every time, so use a schema contract: pass response_format with a json_schema and the gateway constrains generation to it.

CALL_INSIGHTS_SCHEMA = {
    "type": "object",
    "properties": {
        "summary": {"type": "string"},
        "action_items": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "owner": {"type": "string", "enum": ["agent", "customer"]},
                    "task": {"type": "string"},
                    "due": {"type": "string"},
                },
                "required": ["owner", "task", "due"],
                "additionalProperties": False,
            },
        },
        "primary_objection": {"type": "string"},
        "call_outcome": {
            "type": "string",
            "enum": ["closed_won", "follow_up_scheduled", "no_next_step", "lost"],
        },
        "overall_sentiment": {
            "type": "string",
            "enum": ["positive", "neutral", "negative"],
        },
    },
    "required": [
        "summary",
        "action_items",
        "primary_objection",
        "call_outcome",
        "overall_sentiment",
    ],
    "additionalProperties": False,
}

SCHEMA_RULES = (
    "\nReturn JSON only. Use an empty string for primary_objection when the "
    "customer raised none, and for due when no date was agreed."
)


def extract_structured(conversation: str, structured: str) -> dict:
    response = requests.post(
        GATEWAY_URL,
        headers=gateway_headers(),
        json={
            "model": EXTRACTION_MODEL,
            "max_tokens": 1000,
            "messages": [
                {
                    "role": "user",
                    "content": PROMPT.format(
                        conversation=conversation, structured=structured
                    )
                    + SCHEMA_RULES,
                }
            ],
            "response_format": {
                "type": "json_schema",
                "json_schema": {
                    "name": "call_insights",
                    "strict": True,
                    "schema": CALL_INSIGHTS_SCHEMA,
                },
            },
            "post_processing_steps": [{"type": "json-repair"}],
        },
        timeout=60,
    )
    response.raise_for_status()
    return json.loads(response.json()["choices"][0]["message"]["content"])

The enum constraints are the part that pays off. call_outcome can only ever be one of four values, so the field maps cleanly to a CRM picklist and you can group by it without normalizing free text after the fact. Every property is listed in required, with an empty string standing in for “not discussed,” which keeps the schema valid under strict mode across providers. post_processing_steps with json-repair fixes trailing commas and unescaped quotes server-side before the response reaches you. Full details are in the structured outputs documentation.

For resilience, configure a fallback chain so a provider outage degrades to a second model instead of dropping the call from your pipeline.

Data residency, retention and compliance

Call recordings are among the most sensitive data most companies hold, so the handling model matters as much as the extraction quality.

Zero data retention. LLM Gateway doesn’t store the content of prompts or completions for any request. It keeps only usage metadata (the model, token counts, status and request ID) for logging and billing. AssemblyAI has also opted out of data training with all LLM Gateway providers.

EU processing. Point the gateway at https://llm-gateway.eu.assemblyai.com/v1/chat/completions to keep LLM requests in the EU. The EU endpoint serves Anthropic Claude and Google Gemini models, so claude-haiku-5-5 works there and qwen3.5-4b-32k-fast doesn’t. For transcription, set aai.settings.base_url to https://api.eu.assemblyai.com so audio and transcripts are processed in the EU. The data residency docs list which models each region serves.

Certifications and BAAs. AssemblyAI holds SOC 2 Type 2, ISO 27001:2022 and PCI DSS v4.0 certifications. PCI DSS matters here, since payment details spoken aloud on a support line are a routine part of call recordings. For healthcare workloads, AssemblyAI is considered a business associate under HIPAA, and offers a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards protected health information. See the BAA FAQ or contact sales to get one in place.

Error handling

Four failure classes, four handlers. process_call wraps the whole flow so one bad file doesn’t stop a batch:

def process_call(audio_url: str) -> dict | None:
    try:
        transcript = aai.Transcriber(config=config).transcribe(audio_url)
        if transcript.status == aai.TranscriptStatus.error:
            print(f"Transcription failed: {transcript.error}")
            return None

        conversation = format_transcript(transcript)
        insights = extract_structured(conversation, summarize_fields(transcript))
        insights["transcript_id"] = transcript.id
        return insights

    except aai.AssemblyAIError as e:
        print(f"AssemblyAI API error: {e}")
    except requests.exceptions.RequestException as e:
        print(f"LLM Gateway request failed: {e}")
    except json.JSONDecodeError as e:
        print(f"Model returned unparseable output: {e}")
    return None
Failure Where it surfaces What to do
Unreachable or corrupt audio TranscriptStatus.error Log the URL, mark the call for manual review, don’t retry blindly
Auth, quota or malformed request aai.AssemblyAIError Surface immediately; these are configuration bugs, not transients
Gateway timeout or 5xx requests.exceptions.RequestException The gateway already retries 5xx responses; add backoff on your side, then fall through to a configured fallback model
Unparseable model output json.JSONDecodeError Rare with json_schema plus json-repair; if it happens, retry once, then escalate

Scaling call analytics in production

Polling for transcript completion is fine for one file and wrong for ten thousand. Use webhooks: submit the job, return immediately, and let AssemblyAI call you back. The callback payload carries only transcript_id and status, so the worker fetches the transcript by ID. Your endpoint has to return a 2xx within 10 seconds, or AssemblyAI retries the callback, up to 10 attempts in total.

Your webhook handler should do one thing: validate the request, enqueue the transcript ID, and return 200. All extraction work happens in a worker.

WEBHOOK_SECRET = os.environ.get("WEBHOOK_SECRET", "change-me")
app = Flask(__name__)
jobs: "queue.Queue[str]" = queue.Queue()  # swap for SQS, Celery or RQ in production


def submit(audio_url: str, webhook_url: str) -> str:
    config = pipeline.config.set_webhook(
        webhook_url,
        auth_header_name="X-Webhook-Secret",
        auth_header_value=WEBHOOK_SECRET,
    )
    transcript = aai.Transcriber(config=config).submit(audio_url)
    return transcript.id


@app.post("/assemblyai-webhook")
def assemblyai_webhook():
    if request.headers.get("X-Webhook-Secret") != WEBHOOK_SECRET:
        abort(401)
    payload = request.get_json(silent=True) or {}
    if payload.get("status") == "completed" and payload.get("transcript_id"):
        jobs.put(payload["transcript_id"])
    return "", 200


def worker() -> None:
    while True:
        transcript_id = jobs.get()
        transcript = aai.Transcript.get_by_id(transcript_id)
        insights = pipeline.extract_structured(
            pipeline.format_transcript(transcript),
            pipeline.summarize_fields(transcript),
        )
        print(transcript_id, insights)  # write to your CRM or warehouse here
        jobs.task_done()
Component Purpose Implementation
Task queue Buffer transcription jobs and extraction work Celery, RQ, AWS SQS
Webhook receiver Accept completion callbacks, enqueue, return fast FastAPI or Flask behind a load balancer
Worker pool Run the LLM Gateway calls and schema validation Background workers with retry and a dead-letter queue
Data layer Store and surface the structured insights Postgres or a warehouse table, plus a dashboard

Store the raw structured output alongside the parsed fields. When you change a prompt or a schema, you’ll want to re-run against history without re-transcribing 2,000 hours of audio.

Use cases

  • Lead intelligence. Capture contact details, stated budget, timeline and next step from every inbound call, then write them to the CRM without an agent typing anything.
  • Cross-call trend analysis. With consistent topic labels across your corpus, you can answer “what changed in customer objections this quarter” from data instead of anecdotes.
  • Sales coaching. Talk ratios, objection handling and script adherence per rep, scored on every call instead of the handful a manager had time to listen to.
  • Compliance monitoring. Verify required disclosures were read, and flag calls where they weren’t, with content_safety catching the escalations that need human eyes.
  • Support triage. Route follow-ups by sentiment trajectory and detected entities, so the calls that ended badly surface before the customer churns.

These are the workloads contact center and conversation intelligence teams build first, and they all rest on the same three-step pipeline: accurate diarized transcript, redacted and structurally enriched, then interpreted by a model.

Next steps

The full pipeline is about 200 lines of Python, because the hard parts are handled by the API. The decisions that shape your build are which model transcribes the audio, which fields come from structured Speech Understanding features versus an LLM prompt, and which LLM you point at the judgment layer. Clone the companion repo, run pipeline.py on one of your own recordings, and compare the JSON against what your team would have written by hand.

Talk through your call analytics architecture

If you’re processing call recordings at volume, or need a BAA, EU data residency or zero data retention in place, our team can help you scope the deployment.

Talk to AI expert

Frequently Asked Questions (FAQs)

How do I extract per-speaker insights from a call?

Set speaker_labels=True in your TranscriptionConfig to get diarized utterances, then format them as Speaker A: {text} before passing the transcript to an LLM or a scoring function. Use speech_models=["assemblyai-stt-3-7"] for the transcript itself. Async speaker diarization adds $0.02/hr, and per-speaker sentiment comes back on each sentence when sentiment_analysis is on.

What’s the best way to process very long phone calls?

Estimate the transcript’s token count and compare it against your model’s context window before you send it. qwen3.5-4b-32k-fast has a 32,768-token context, which fits most calls of an hour or less, and claude-haiku-5-5 takes up to 1,000,000 tokens, so chunking is often unnecessary. When a call does exceed the window, split on speaker turns or fixed time intervals instead of raw character counts, summarize each chunk, then summarize the summaries. Splitting mid-turn is what causes the model to lose track of who said what.

How can I get structured JSON output for CRM or database integration?

Pass response_format with a json_schema object and "strict": true to a model that supports it, such as claude-haiku-5-5 or claude-sonnet-4-6 on LLM Gateway. A schema lets you constrain fields to enums, so call_outcome can only ever be one of your CRM picklist values, and mark every field as required, so the output maps to database columns without a normalization pass. Add post_processing_steps: [{"type": "json-repair"}] to fix malformed JSON server-side.

Should I use Speech Understanding or an LLM prompt to extract entities and sentiment?

Use a Speech Understanding feature when you need the same field on every call: entity detection, sentiment analysis, topic detection and key phrases are deterministic, typed, timestamped and billed per hour of audio, so cost and output are predictable. Use an LLM prompt when the output is bespoke or requires judgment, like a QA rubric, a summary in your team’s format or a recommended next action. The strongest pipelines do both: extract structured fields first, then pass them into the LLM as ground-truth context so it reasons over facts.

How do I redact credit card numbers and PII from call recordings?

Set redact_pii=True on your TranscriptionConfig and list the categories in redact_pii_policies: credit_card_number, credit_card_cvv, banking_information, us_social_security_number and so on. Use redact_pii_sub to control the replacement (entity_name writes readable tokens like [CREDIT_CARD_NUMBER], which works better as LLM input than a hash), and add redact_pii_audio=True to remove the same spans from the recording itself. Text redaction is +$0.08/hr and audio redaction is +$0.05/hr; run both before any transcript reaches your warehouse or an LLM.

What does call analytics cost per call?

A 12-minute call transcribed with AssemblyAI-STT-3.7 plus diarization, entity detection, sentiment analysis, PII text redaction and an LLM-generated JSON record comes to roughly $0.083, or about $825/month at 10,000 calls. Audio processing dominates that figure. The LLM step is about $5/month at the same volume using claude-haiku-5-5 at in-region rates. See pricing for current rates on every component.