How to build an AI lecture summarizer app

Search for "lecture summarizer" and you get a wall of sign-up pages. Upload your recording, get bullet points, pay $9.99 a month. Every one of them is a thin wrapper around two API calls: transcribe the audio, then hand the transcript to a language model with a good prompt.

That's the whole product. And once you've built it yourself, you own it — you choose the model, you control the prompt, you decide what a "summary" means for your users, and you can point the same pipeline at anything with a speaker in it.

So that's what this tutorial does. By the end you'll have a working Python app that:

• Transcribes a recorded lecture with speaker labels and timestamps
• Produces a structured summary with topic-by-topic sections
• Answers free-form questions about the lecture ("what did she say about eigenvalues?")
• Extends, with about twenty lines of extra code, into a meeting scribe that produces notes and action items

Everything runs on two AssemblyAI APIs: speech-to-text for the transcript, and the LLM Gateway for the summarization and Q&A. No vector database, no orchestration framework, no front-end build step.

Prerequisites:

• Python 3.9 or later
• An AssemblyAI API key (free to create — the same key authenticates both APIs)
• A recorded lecture in any common audio or video format
• About 15 minutes

How lecture summarization actually works

Before the code, it's worth being precise about what's happening, because "AI summarizes your lecture" hides two very different problems.

Problem one is getting the words right. A lecture is not clean audio. It's one person talking for fifty minutes in a room with HVAC noise, a lapel mic that's slightly too far away, students interrupting from across the hall, and a vocabulary that includes words the speech model has rarely seen — heteroskedasticity, Hamiltonian, the name of a researcher from a 1987 paper. If the transcript gets those wrong, no summarization step can recover them. The summary will confidently describe a lecture that didn't happen.

Problem two is compression with structure. A 50-minute lecture is roughly 7,500 words. A useful summary isn't a shorter version of all of it — it's a map. Which topics were covered, in what order, what the professor emphasized, what was assigned, what's going to be on the exam.

Older speech APIs, including ours, used to ship fixed features for the second problem: a chapter-detection endpoint that segmented audio into topics, and a highlights endpoint that pulled out key phrases. They worked, but they were rigid. You got the summary shape the vendor decided on, and if you wanted "structure this as exam revision notes with a definitions glossary," you were out of luck.

Modern practice separates the two cleanly. Run a good speech model to get an accurate, speaker-labeled, timestamped transcript. Then run a language model over that transcript with a prompt that describes exactly the output you want. The second half becomes configuration instead of a feature request, which is why every summarization feature in this tutorial is a string you can edit.

That's the architecture here:

Setup

Install the two dependencies. The AssemblyAI SDK handles transcription; requests talks to the LLM Gateway over plain HTTP.

pip install -U assemblyai requests

Set your API key as an environment variable so it never ends up in your source:

export ASSEMBLYAI_API_KEY="your-api-key"

One key covers both APIs. If you're on Windows, use set ASSEMBLYAI_API_KEY=your-api-key in cmd or $env:ASSEMBLYAI_API_KEY="your-api-key" in PowerShell.

Step 1: Transcribe the lecture

Create lecture.py and start with the transcription step:

from __future__ import annotations

import os

import assemblyai as aai

aai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]


def transcribe_lecture(audio_path: str, context: str | None = None):
    """Transcribe a lecture recording with speaker labels and timestamps."""
    config = aai.TranscriptionConfig(
        speech_models=["universal-3-5-pro"],
        speaker_labels=True,
        prompt=context,
    )

    transcript = aai.Transcriber(config=config).transcribe(audio_path)

    if transcript.status == aai.TranscriptStatus.error:
        raise RuntimeError(f"Transcription failed: {transcript.error}")

    return transcript


if __name__ == "__main__":
    transcript = transcribe_lecture(
        "lecture.mp3",
        context=(
            "A university lecture in an introductory linear algebra course. "
            "Expect terms like eigenvalue, eigenvector, determinant, "
            "orthonormal basis, and Gram-Schmidt."
        ),
    )
    print(f"Transcript ID: {transcript.id}")
    print(f"Duration: {transcript.audio_duration} seconds")
    print(transcript.text[:500])

Three things in that config are doing real work.

speech_models=["universal-3-5-pro"] pins the request to Universal-3.5 Pro, the current flagship async model. It runs at $0.21/hr, handles 18 languages with native code-switching, and has the most accurate speaker diarization we've shipped. If you'd rather always ride the latest Universal Pro model without touching your code again, omit speech_models entirely.

speaker_labels=True turns the transcript into a list of utterances attributed to Speaker A, Speaker B, and so on. In a lecture that usually means one dominant speaker plus student questions, which is exactly the distinction you want the summary to respect — a student's guess is not the professor's answer.

prompt is contextual prompting, and it's the single highest-leverage line in this file. You're telling the model what the audio is about before it hears a word. Domain vocabulary that would otherwise come back phonetically mangled comes back spelled correctly. If you have the syllabus, the course title, or last week's notes, put them here.

One line that isn't about the API: from __future__ import annotations at the top. Without it, the str | None annotation on context is a syntax the interpreter evaluates at definition time, and on Python 3.9 that raises TypeError: unsupported operand type(s) for |: 'type' and 'NoneType' before your program does anything. The future import defers annotation evaluation and makes the file run unchanged on 3.9 through 3.13.

Run it and you'll get a transcript ID back. Hold onto that ID — the LLM Gateway can fetch the transcript text for you by ID, which means you don't have to shuttle 7,500 words through your own process on every question.

Step 2: Summarize the transcript with the LLM Gateway

The LLM Gateway is one OpenAI-compatible endpoint that fronts 25+ models from Anthropic, OpenAI, Google, and others. You authenticate with the same AssemblyAI key you used for transcription, and you swap models by changing a string.

Add this to lecture.py:

import requests

LLM_GATEWAY_URL = "https://llm-gateway.assemblyai.com/v1/chat/completions"
DEFAULT_MODEL = "claude-sonnet-4-6"


def call_gateway(
    messages: list[dict],
    transcript_id: str | None = None,
    model: str = DEFAULT_MODEL,
    max_tokens: int = 2000,
) -> str:
    """Send a chat completion request to the LLM Gateway."""
    body = {
        "model": model,
        "messages": messages,
        "max_tokens": max_tokens,
    }

    # The Gateway fetches the transcript and substitutes it into the first
    # "{{ transcript }}" tag it finds, so you never have to hold the full
    # transcript in your own process. No tag, no transcript.
    if transcript_id:
        body["transcript_id"] = transcript_id

    response = requests.post(
        LLM_GATEWAY_URL,
        headers={"authorization": os.environ["ASSEMBLYAI_API_KEY"]},
        json=body,
        timeout=120,
    )
    response.raise_for_status()

    return response.json()["choices"][0]["message"]["content"]

The one detail that decides whether any of this works

transcript_id on its own does nothing.

The Gateway does not silently prepend your transcript to the conversation. It looks for the literal tag {{ transcript }} in the first message that contains one, and replaces that tag with the transcript text. If no message contains the tag, the request still succeeds — HTTP 200, a perfectly well-formed completion — and the model simply never sees a word of your lecture. You get a confident summary of nothing, or a polite request to paste the transcript.

That failure is silent and it is easy to ship. Here is the same request with and without the tag, against the same transcript ID:

The token count is the tell. If your prompt tokens look like the length of your instructions rather than the length of your lecture, the transcript isn't there.

Three rules worth memorising:

• The tag must be exactly {{ transcript }}, with the spaces. {{transcript}} and {{ TRANSCRIPT }} are not substituted.
• Only the first occurrence in the first message containing it is replaced. Extra tags are left as literal text.
• An ID that doesn't exist, or belongs to another account, returns 404 — but only once you've given it a tag to substitute into.

Now the summarizer itself. This is where you decide what a lecture summary means for your users — and the answer is different for a student cramming than for an instructor checking coverage.

SUMMARY_SYSTEM_PROMPT = """You summarize university lectures for students who
missed the class. Work only from the transcript you are given. If something is
unclear in the transcript, say so rather than guessing.

Produce, in this order:

1. A two-sentence overview of what the lecture covered.
2. "Topics covered" — a numbered list. Each entry has a short heading and two
   or three sentences explaining the idea in plain language.
3. "Definitions" — every term the lecturer explicitly defined, with the
   definition they gave, not a textbook one.
4. "Emphasized for the exam" — anything the lecturer flagged as important,
   likely to be tested, or worth reviewing. Omit this section if nothing
   qualifies.
5. "Assigned work" — readings, problem sets, and deadlines mentioned.
   Omit this section if nothing qualifies.

Use markdown headings. Do not invent content that is not in the transcript."""


def summarize_lecture(transcript_id: str, model: str = DEFAULT_MODEL) -> str:
    return call_gateway(
        messages=[
            {"role": "system", "content": SUMMARY_SYSTEM_PROMPT},
            {
                "role": "user",
                "content": (
                    "Summarize this lecture transcript using the structure "
                    "described in your instructions.\n\n{{ transcript }}"
                ),
            },
        ],
        transcript_id=transcript_id,
        model=model,
    )

Notice what isn't here: no chunking loop, no map-reduce, no embedding step. A 50-minute lecture is well under the context window of every model in the table on the Gateway's model page. Chunking a lecture is a solved problem that no longer needs solving, and skipping it means the model sees the whole arc of the argument instead of twelve disconnected windows.

The instruction "work only from the transcript you are given" matters more than it looks. Without it, models happily fill gaps with textbook knowledge about linear algebra, which produces a summary that's correct and not about this lecture. That's the failure mode that makes students stop trusting the tool.

Step 3: Answer questions about the lecture

Summaries are the front door. Questions are what people actually come back for. Same endpoint, different prompt, plus a conversation history so follow-ups work:

QA_SYSTEM_PROMPT = """You answer questions about a specific university lecture,
using only its transcript.

Transcript:
{{ transcript }}

Rules:
- Answer in two to four sentences unless the question demands more.
- Quote the lecturer directly when the exact phrasing matters.
- If the transcript does not contain the answer, say "That wasn't covered in
  this lecture" and stop. Do not answer from general knowledge.
- When a student question and the lecturer's answer are both relevant, make
  clear which speaker said what."""


class LectureQA:
    """Multi-turn Q&A over a single lecture transcript."""

    def __init__(self, transcript_id: str, model: str = DEFAULT_MODEL):
        self.transcript_id = transcript_id
        self.model = model
        self.history: list[dict] = [
            {"role": "system", "content": QA_SYSTEM_PROMPT}
        ]

    def ask(self, question: str) -> str:
        self.history.append({"role": "user", "content": question})

        answer = call_gateway(
            messages=self.history,
            transcript_id=self.transcript_id,
            model=self.model,
            max_tokens=800,
        )

        self.history.append({"role": "assistant", "content": answer})
        return answer

The tag lives in the system prompt here, not the user message, and that's deliberate. The history grows with every exchange, but the system message stays at index 0, so it's always the first message containing the tag and the transcript gets substituted on every turn. Put the tag in a user message instead and only the first question would see the lecture.

The cost of that is real and worth knowing: the transcript is re-sent on every turn, so a long conversation over a long lecture is not free. If that matters, cache the substituted transcript yourself and stop sending transcript_id after the first call. For a study tool with a handful of questions per session, the simplicity is worth more than the tokens.

The refusal rule is doing the heavy lifting again. A lecture Q&A tool that answers everything is a chatbot. One that says "that wasn't covered" is a study aid, because now a "no" is information: the student knows to look elsewhere instead of assuming they missed it.

Step 4: Put it together

Here's the complete runnable app. Save it over lecture.py — it's the three pieces above plus a transcript cache and an interactive loop, with nothing elided.

"""An interactive AI lecture summarizer.

Usage:
    python lecture.py lecture.mp3
    python lecture.py lecture.mp3 --context "Intro to linear algebra, week 6"
"""

from __future__ import annotations

import argparse
import json
import os
from pathlib import Path

import assemblyai as aai
import requests

aai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

LLM_GATEWAY_URL = "https://llm-gateway.assemblyai.com/v1/chat/completions"
DEFAULT_MODEL = "claude-sonnet-4-6"
CACHE_PATH = Path(".lecture_cache.json")

SUMMARY_SYSTEM_PROMPT = """You summarize university lectures for students who
missed the class. Work only from the transcript you are given. If something is
unclear in the transcript, say so rather than guessing.

Produce, in this order:

1. A two-sentence overview of what the lecture covered.
2. "Topics covered" — a numbered list. Each entry has a short heading and two
   or three sentences explaining the idea in plain language.
3. "Definitions" — every term the lecturer explicitly defined, with the
   definition they gave, not a textbook one.
4. "Emphasized for the exam" — anything the lecturer flagged as important,
   likely to be tested, or worth reviewing. Omit this section if nothing
   qualifies.
5. "Assigned work" — readings, problem sets, and deadlines mentioned.
   Omit this section if nothing qualifies.

Use markdown headings. Do not invent content that is not in the transcript."""

QA_SYSTEM_PROMPT = """You answer questions about a specific university lecture,
using only its transcript.

Transcript:
{{ transcript }}

Rules:
- Answer in two to four sentences unless the question demands more.
- Quote the lecturer directly when the exact phrasing matters.
- If the transcript does not contain the answer, say "That wasn't covered in
  this lecture" and stop. Do not answer from general knowledge.
- When a student question and the lecturer's answer are both relevant, make
  clear which speaker said what."""


# --- Transcription -----------------------------------------------------------


def transcribe_lecture(audio_path: str, context: str | None = None):
    config = aai.TranscriptionConfig(
        speech_models=["universal-3-5-pro"],
        speaker_labels=True,
        prompt=context,
    )

    transcript = aai.Transcriber(config=config).transcribe(audio_path)

    if transcript.status == aai.TranscriptStatus.error:
        raise RuntimeError(f"Transcription failed: {transcript.error}")

    return transcript


def load_cache() -> dict:
    if CACHE_PATH.exists():
        return json.loads(CACHE_PATH.read_text())
    return {}


def save_cache(cache: dict) -> None:
    CACHE_PATH.write_text(json.dumps(cache, indent=2))


def get_transcript_id(audio_path: str, context: str | None) -> str:
    """Transcribe once per file, then reuse the transcript ID forever."""
    cache = load_cache()
    key = str(Path(audio_path).resolve())

    if key in cache:
        print(f"Using cached transcript {cache[key]}")
        return cache[key]

    print("Transcribing… (a few minutes for an hour of audio)")
    transcript = transcribe_lecture(audio_path, context)

    cache[key] = transcript.id
    save_cache(cache)

    print(f"Done. Transcript ID: {transcript.id}")
    return transcript.id


# --- LLM Gateway -------------------------------------------------------------


def call_gateway(
    messages: list[dict],
    transcript_id: str | None = None,
    model: str = DEFAULT_MODEL,
    max_tokens: int = 2000,
) -> str:
    body = {"model": model, "messages": messages, "max_tokens": max_tokens}

    # Requires a literal "{{ transcript }}" tag in one of the messages.
    if transcript_id:
        body["transcript_id"] = transcript_id

    response = requests.post(
        LLM_GATEWAY_URL,
        headers={"authorization": os.environ["ASSEMBLYAI_API_KEY"]},
        json=body,
        timeout=120,
    )
    response.raise_for_status()

    return response.json()["choices"][0]["message"]["content"]


def summarize_lecture(transcript_id: str, model: str = DEFAULT_MODEL) -> str:
    return call_gateway(
        messages=[
            {"role": "system", "content": SUMMARY_SYSTEM_PROMPT},
            {
                "role": "user",
                "content": (
                    "Summarize this lecture transcript using the structure "
                    "described in your instructions.\n\n{{ transcript }}"
                ),
            },
        ],
        transcript_id=transcript_id,
        model=model,
    )


class LectureQA:
    def __init__(self, transcript_id: str, model: str = DEFAULT_MODEL):
        self.transcript_id = transcript_id
        self.model = model
        self.history: list[dict] = [
            {"role": "system", "content": QA_SYSTEM_PROMPT}
        ]

    def ask(self, question: str) -> str:
        self.history.append({"role": "user", "content": question})

        answer = call_gateway(
            messages=self.history,
            transcript_id=self.transcript_id,
            model=self.model,
            max_tokens=800,
        )

        self.history.append({"role": "assistant", "content": answer})
        return answer


# --- CLI ---------------------------------------------------------------------


def main() -> None:
    parser = argparse.ArgumentParser(description="Summarize and query a lecture.")
    parser.add_argument("audio", help="Path or URL to the lecture recording")
    parser.add_argument(
        "--context",
        default=None,
        help="Course title, syllabus notes, or expected vocabulary",
    )
    parser.add_argument("--model", default=DEFAULT_MODEL)
    args = parser.parse_args()

    transcript_id = get_transcript_id(args.audio, args.context)

    print("\n" + "=" * 70)
    print("SUMMARY")
    print("=" * 70 + "\n")
    print(summarize_lecture(transcript_id, model=args.model))

    qa = LectureQA(transcript_id, model=args.model)

    print("\n" + "=" * 70)
    print("Ask anything about the lecture. Press Enter on a blank line to quit.")
    print("=" * 70 + "\n")

    while True:
        try:
            question = input("> ").strip()
        except (EOFError, KeyboardInterrupt):
            print()
            break

        if not question:
            break

        print("\n" + qa.ask(question) + "\n")


if __name__ == "__main__":
    main()

Run it:

python lecture.py lecture.mp3 --context "Intro to linear algebra, week 6: eigenvalues"

The cache file means transcription happens once per recording. Every subsequent run is a Gateway call against a transcript ID that already exists, which takes seconds and costs a fraction of a cent.

Testing and validating the output

A summarizer that's wrong is worse than no summarizer, because nobody checks it. Four tests worth running before you put this in front of anyone.

Check that the transcript is actually reaching the model. Print response.json()["usage"]["input_tokens"] on your first summary call. If it's in the low tens, your {{ transcript }} tag is missing or misspelled and you are summarizing an empty string. If it's in the thousands, you're fine. This takes ten seconds and it is the single most valuable check in this list, because every other failure at least looks like a failure.

Test the transcript, not the summary. Pick three minutes of the hardest audio you have — the part with the noisy room and the technical vocabulary — and read the transcript against the audio yourself. Count the words that matter: names, terms, numbers. If the transcript is wrong there, add those terms to your prompt and re-run. This is the step people skip, and it's the one that determines whether the whole app works.

Test refusal. Ask the Q&A a question the lecture definitely doesn't answer. "What's the professor's office number?" If you get a confident invented answer, tighten the system prompt until you get "that wasn't covered."

Test against a summary you'd write yourself. Summarize one lecture by hand, then compare. You're not looking for identical text — you're looking for whether the model picked the same five topics you did, and whether it caught the "this will be on the exam" line. If it consistently misses emphasis, that's a prompt problem, and you fix it by naming the cue explicitly ("the lecturer often signals importance with phrases like 'remember this' or 'you'll see this again'").

One thing worth knowing about diarization: speaker labels are assigned per-audio, not per-person. Speaker A in this week's lecture is not necessarily Speaker A in next week's. If you're building a multi-lecture product, map labels to real names yourself using the first utterance and a bit of context.

Getting started with the LLM Gateway

If you came here for the summarization half specifically, here's the short version of how the Gateway works on its own.

It's a single OpenAI-compatible endpoint at https://llm-gateway.assemblyai.com/v1/chat/completions. You authenticate with your AssemblyAI API key in the authorization header — no Bearer prefix — and you pick a model by ID in the request body. Claude, GPT, Gemini, Qwen, Kimi, and more sit behind that one URL, so switching providers is a one-string diff rather than a new SDK and a new key.

Because it's OpenAI-compatible, the official OpenAI Python client works against it directly:

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://llm-gateway.assemblyai.com/v1",
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
)

response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Summarize this in three bullets: ..."}],
)

print(response.choices[0].message.content)

Three features matter for anything built on transcripts. Transcript injection — the transcript_id field used above, paired with a {{ transcript }} tag — lets the Gateway fetch the transcript text itself, so your app doesn't have to store or re-send it. Model fallbacks let you name an alternative model (and even an alternative prompt) that gets used automatically when your first choice is unavailable, which matters when a summary button has to work at 2am the night before an exam. And structured outputs constrain the response to a JSON schema, which is what you want the moment a summary stops being displayed as prose and starts populating database columns. One caveat on that last one: not every model behind the Gateway implements response_format, so check before you depend on it.

There's a deeper walkthrough in our LLM Gateway introduction, and the full request and response schema lives in the chat completions reference.

From lecture notes to meeting notes and action items

Here's the thing about the app you just built: nothing in it is about lectures. It's a pipeline that takes recorded speech and returns structured text. Point it at a Zoom recording instead of a lecture hall and it becomes an AI meeting scribe — which is, not coincidentally, one of the most common things developers build on this exact stack.

The changes are small, and there are three of them.

One: the transcript needs real speaker attribution. A lecture is mostly one voice. A meeting is six people interrupting each other, and a summary that can't tell who committed to what is useless. Speaker labels were optional before; here they're the whole point. Use utterances rather than the flat transcript text so the model sees the turn structure:

def format_utterances(transcript) -> str:
    """Render a diarized transcript as a speaker-labeled script."""
    return "\n".join(
        f"Speaker {u.speaker}: {u.text}" for u in transcript.utterances
    )

Note that this path sends the transcript yourself rather than using transcript_id — which is the right call here, because you want the speaker-labeled rendering, not the flat text the Gateway would substitute.

Two: the context prompt changes from a syllabus to a meeting agenda. Universal-3.5 Pro's contextual prompting takes whatever you know about the conversation before it happens. For a meeting, that's the calendar title, the attendee names, the product names, and the customer accounts that are going to come up. Names are exactly where transcription usually fails, and exactly what an action item needs to be right about.

This is the part Metaview, which builds AI notetaking for recruiting interviews, pushed on hardest. Shahriar Tajbakhsh, their Co-founder and CTO, put it this way:

“Since moving to AssemblyAI, we've seen a meaningful improvement in the confidence tail of our production transcripts….What stands out is not just the model quality, but the way [they] let us bring real meeting context into transcription, from calendar titles to organizations, domains, and participant names, so recruiting conversations come through with the nuance our customers depend on.”

Three: the output needs a schema, not prose. Meeting notes get filed, searched, and synced into a task tracker. That means JSON:

MEETING_SYSTEM_PROMPT = """You produce meeting notes from a speaker-labeled
transcript. Work only from the transcript.

Return JSON matching this shape exactly:

{
  "summary": "two to four sentences on what the meeting was about",
  "decisions": [
    {"decision": "...", "owner": "Speaker A or a real name if stated"}
  ],
  "action_items": [
    {"task": "...", "owner": "...", "due": "date or null if not stated"}
  ],
  "open_questions": ["..."],
  "topics": [{"topic": "...", "detail": "one or two sentences"}]
}

Rules:
- An action item requires someone accepting responsibility for something
  concrete. "We should probably look at that" is an open question, not an
  action item.
- Never invent a due date. If nobody said one, use null.
- Map speaker labels to real names only when someone is addressed by name in
  the transcript. Otherwise keep the label."""


def meeting_notes(transcript) -> dict:
    """Produce structured meeting notes from a diarized transcript."""
    script = format_utterances(transcript)

    raw = call_gateway(
        messages=[
            {"role": "system", "content": MEETING_SYSTEM_PROMPT},
            {"role": "user", "content": script},
        ],
        max_tokens=3000,
    )

    # Strip a markdown fence if the model added one, then parse.
    cleaned = raw.strip().removeprefix("```json").removeprefix("```").removesuffix("```")
    return json.loads(cleaned)

Use it like this:

transcript = transcribe_lecture(
    "standup.m4a",
    context=(
        "A weekly product standup at Acme. Attendees: Priya, Marcus, Dana. "
        "Projects discussed: Atlas migration, Beacon dashboard, Q4 pricing."
    ),
)

notes = meeting_notes(transcript)

print(notes["summary"])
for item in notes["action_items"]:
    due = item["due"] or "no date"
    print(f"  [ ] {item['task']} — {item['owner']} ({due})")

The action-item rule in that prompt is the one worth stealing. Every meeting-notes tool that's annoying to use is annoying for the same reason: it treats every sentence containing a verb as a task. Distinguishing a commitment from a musing is a prompt-engineering problem, and you now own the prompt.

If you're building this as a product rather than a script, the AI notetakers solution page covers the architecture at production scale, and there's a focused walkthrough in summarize meeting transcripts with LLMs in Python.

Next steps

You now have a lecture summarizer, a Q&A interface, and a meeting scribe sharing one pipeline. A few directions worth taking it:

Add timestamps to the summary. Every word in the transcript carries millisecond timings. Ask the model to cite the timestamp of the passage each topic came from, then render those as clickable seeks into the audio player. This is the feature that turns a summary into a navigation tool.

Batch a whole semester. Summarizing audio and video at scale is mostly a queueing problem — submit transcriptions in parallel, collect the IDs, run the Gateway over them afterward. A course's worth of lectures summarizes in the time it takes to make coffee.

Go cross-lecture. With a semester of transcript IDs in hand, "when did we first cover eigenvectors?" becomes a question you can answer across the whole course rather than one recording at a time.

The reason to build this yourself instead of paying for one of those landing pages isn't cost. It's that the prompt is the product. Every tool in that search results page made a decision about what a summary should look like, once, for everyone. You get to make that decision for your users, and change it on a Tuesday when they tell you it's wrong.

Frequently asked questions

What is a lecture summarizer?

A lecture summarizer is an application that converts a recorded lecture into structured notes — usually an overview, a topic breakdown, key definitions, and any assigned work. It works in two stages: a speech-to-text model produces an accurate transcript of the audio, then a language model compresses that transcript into the structure you asked for. The quality of the second stage is capped by the first, which is why transcription accuracy matters more than model choice.

How do I summarize a lecture with AI in Python?

Transcribe the recording with a speech-to-text API, then send the transcript to a language model with a prompt describing the summary structure you want. With AssemblyAI that's two calls: aai.Transcriber().transcribe() for the transcript, then a chat completion against the LLM Gateway at https://llm-gateway.assemblyai.com/v1/chat/completions. If you pass a transcript_id rather than the text, remember to include a literal {{ transcript }} tag in one of your messages — without it the request succeeds and the model receives no transcript at all. The complete script is in this post; one API key authenticates both steps.

Does AssemblyAI provide customizable summary types?

Yes. Summarization runs through the LLM Gateway, so the "summary type" is whatever you describe in your prompt — bullets, a topic-by-topic outline, exam revision notes, a clinical note, or a JSON object matching a schema you define. You also choose which model produces it from 25+ options across Anthropic, OpenAI, and Google. That's more flexible than fixed summary-type parameters, because you can change the output shape without waiting on a product release.

What's the best API for meeting transcription and summarization?

For meetings specifically, look for three things: accurate speaker diarization, the ability to pass meeting context before transcription, and a summarization layer you can shape yourself. AssemblyAI's Universal-3.5 Pro handles the first two — it produces the transcript and the speaker-change boundaries jointly, which is what captures short turns and cross-talk — and the LLM Gateway handles the third. Meeting and notetaking products including Granola, Fireflies, and Metaview run on this stack.

Can I build an AI meeting scribe that generates summaries and action items?

Yes, and it's the same pipeline as a lecture summarizer with two changes: turn on speaker labels and use the diarized utterances instead of the flat transcript, and ask the model for structured JSON with separate decisions, action_items, and open_questions fields. The hard part is instructing the model to distinguish a real commitment from a passing suggestion — the section above has a working prompt for that.

How do I get started with AssemblyAI's API?

Create a free account, copy your API key from the dashboard, and install the SDK with pip install -U assemblyai. A first transcription is three lines: set aai.settings.api_key, build a TranscriptionConfig, and call .transcribe() with a file path or URL. Billing is per second with no minimums, so you can test on real audio before committing to anything.