From transcript to summary to follow-up items: Speech understanding gets an LLM upgrade

What's new in Summarization

Where summarization used to give you a single flat summary of your entire file, our latest version works the way people actually summarize: summaries broken out by topic, each with its own headline and start/end timestamps—as scannable bullets (the default) or prose paragraphs. Here it is on a real team meeting, run with summary_type set to paragraph:

"summarization": {
  "summary": [
    {
      "start": 32,
      "end": 213402,
      "text": "The meeting begins with an introduction to the new group name, 'Govern Growth and Data Science', which covers Applied ML, MLOps, and Anti-Abuse...",
      "headline": "Organizational Changes and New Leadership"
    },
    {
      "start": 62869,
      "end": 244341,
      "text": "The meeting covers updates on engineering allocation, error budgets, and reliability...",
      "headline": "Engineering Metrics and Security Automation"
    },
    ...
  ],
  "summary_type": "paragraph",
  "effort": "low",
  "status": "success"
}

Timestamps mean you can link each summary section straight back to the audio. Headlines mean you can render a table of contents without any post-processing.

How to use it

One request, transcription and summary together—the SDK handles polling for you:

import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

config = aai.TranscriptionConfig(
    speech_understanding={
        "request": {
            "summarization": {"summary_type": "paragraph"}
        }
    }
)

transcript = aai.Transcriber().transcribe(
    "https://assembly.ai/wildfires.mp3", config
)

print(transcript.speech_understanding.response.summarization)

Two parameters control the output:

• summary_type—bullets (default) for short, scannable summaries per topic, or paragraph for longer, more detailed prose.
• effort—low (default) covers the majority of use cases. Switch to medium when missed details are costly: high-stakes meetings, multilingual audio, or very long files (1.5 hours and up).

Full details are in the Summarization docs.

Action Items: the follow-ups, extracted

Most teams summarizing meetings want two outputs: what was discussed, and what happens next. The new Action Items shipped as its own feature, tuned for its own task:

import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

config = aai.TranscriptionConfig(
    speech_understanding={
        "request": {"action_items": {}}
    }
)

transcript = aai.Transcriber().transcribe(
    "https://assembly.ai/wildfires.mp3", config
)

print(transcript.speech_understanding.response.action_items)

"action_items": {
  "status": "success",
  "items": [
    {
      "action_item": "Consider a better name for the meeting.",
      "quote": "We might get a better name over time.",
      "timestamp": 8421
    },
    {
      "action_item": "Alan will step in as acting full stack manager for security policies while an EM is hired.",
      "quote": "I've asked Alan to step in as acting full stack manager for security policies while I hire an EM.",
      "timestamp": 82978
    },
    ...
  ],
  "effort": "low"
}

You get back a structured list of commitments and to-dos from the conversation—ready to push into a task tracker, CRM, or follow-up email. Run it with summarization in the same request, or on its own.

One request object, a growing set of native audio tasks

Summarization and Action Items join a growing set of speech understanding features designed to help you transform and process audio data natively in the API—no separate vendors, no glue code, no shipping transcripts to a second platform:

• Summarization—topic-based, timestamped summaries as bullets or paragraphs
• Action Items—structured follow-ups and commitments
• Translation—translated transcripts across languages, down to the utterance level
• Speaker Identification—map diarized speakers to real names and roles
• Custom Formatting—normalize dates, phone numbers, and emails to your format

Stack them in a single request and get one response with every result. Transcribe a sales call once; get back the transcript, a summary, the follow-ups, and speaker names in the same payload.

And when the built-in tasks don't cover exactly what you need, like a custom JSON schema or your own extraction logic, the LLM Gateway lets you run any LLM-based task on dozens of market-leading models following transcription, with configurable fallbacks, from the same API too.

Simple pricing, per hour of audio

• Summarization—$0.05/hr
• Action Items—$0.02/hr

Summarization and Action Items are billed on audio duration, both work alongside any other features in your request, and neither requires a commitment or minimum. Summarize a full hour of audio and extract its action items for seven cents. No reserved capacity, no standing up servers, no upfront commitments.

Upgrading from the legacy parameters

With the upgrade live, the legacy parameters are deprecated:

• summarization, summary_model, and summary_type (top-level) → replaced by speech_understanding.request.summarization
• auto_chapters → replaced by chapter generation through the LLM Gateway

The legacy parameters continue to work with Universal-2 until September 15, 2026, but aren't supported on our latest Universal models. After September 15, they'll be removed entirely.

If you're using legacy parameters, moving summarization over is a config change—shift the parameter into the speech_understanding object and you're done. For chapters, the LLM Gateway approach gives you something the old parameter never could: control. Pull paragraphs from the transcript, group them to set your chapter length, and prompt any Gateway model for headlines, gists, and summaries in exactly the format your product needs. The Auto Chapters docs walk through the full pattern in Python and JavaScript.

If you're using any of the deprecated parameters, you'll get an email with migration steps and code samples this week.

Frequently asked questions