How to build a lecture capture system with speaker identification

This tutorial shows you how to build a lecture capture system that automatically works out who is speaking in a classroom recording. You'll write a Python application that records the room, sends the audio through speaker diarization, and produces speaker-attributed captions students can search. It handles the things real classrooms throw at you: HVAC noise, a professor who wanders away from the mic, and a Q&A section where six people talk in ninety seconds.

You'll build two paths. The async path uploads a finished recording and returns the most accurate speaker labels available — that's what you want for the version students watch later. The streaming path labels speakers live over a WebSocket, for captions on the projector during class. Both run on Universal-3.5 Pro.

What is a lecture capture system?

A lecture capture system records classroom audio and video, then makes those recordings available for students to review later. That's the whole job in one sentence. The interesting part is what you do with the recording once you have it.

Speaker diarization is what turns a recording into something usable. It answers "who spoke when," which means a transcript stops being a wall of text and becomes navigable. A student who missed Thursday can jump to the three minutes where the professor explained the thing they don't understand, instead of scrubbing through a 75-minute file.

Four things get better once speaker labels exist:

• Search that works. Students can search within a speaker, not just within a transcript.
• Captions that name the speaker. Required for accessibility, useful for everyone.
• Skippable Q&A. Lecture content and student questions become separable.
• Study notes that read like a conversation. Because they were one.

Why speaker diarization matters for lecture capture

A transcript without speaker labels is a search index. A transcript with them is a study tool.

The difference shows up most for the students who need it most. International learners can anchor on who is speaking while they work through an unfamiliar accent. Screen readers can announce speaker changes. And any downstream tool you build — a summarizer, a study-guide generator, a question extractor — gets dramatically better input when it knows the professor's words from a student's.

A note on privacy before you write any code. Diarization returns anonymous labels: Speaker A, Speaker B. It does not know who anyone is, and in a classroom that's a feature. Several privacy frameworks — FERPA in the US chief among them — counsel real care around creating and storing identifiable records of individual student voices. The approach this tutorial takes: generic labels for students, role labels at most, informed consent before recording, and access controls on the transcript. There's an implementation section on this below.

Lecture capture system architecture options

You can buy a hardware lecture capture appliance or you can build a software system on a computer with a decent microphone. The choice mostly determines how good your speaker labels get, because diarization quality is a model problem, not a microphone problem.

Software-based systems for speaker diarization

Software wins on diarization for a boring reason: you can swap the model. A hardware appliance ships with whatever speaker separation its vendor built, and you get the next version when you buy the next box. A software system calls an API, and the model improves underneath you.

For reference on the cost side: AssemblyAI's pre-recorded transcription runs $0.21/hr on Universal-3.5 Pro with speaker diarization at +$0.02/hr, and the free tier covers 185 hours of pre-recorded audio. A 15-week course with three 75-minute lectures a week is about 56 hours of audio.

How to build a lecture capture system with speaker diarization

Four pieces: capture the audio, transcribe it with speaker labels, attribute the speakers, and write out captions. We'll build each as its own module.

Set up audio capture for speaker diarization

Diarization works by building a voice embedding for each speaker and clustering on it. Everything that muddies those embeddings — noise, cross-talk, wildly uneven volume — costs you accuracy. So the audio setup matters more here than it does for plain transcription.

Record at 16 kHz minimum, 16-bit, mono. Higher sample rates don't hurt but they don't help diarization much either; clean separation between voices helps far more than extra bandwidth. The docs' guidance is worth internalizing: each speaker should get at least 30 seconds of speech before you should expect a stable label for them. A student who says "yeah" once is not going to get their own cluster, and that's correct behavior.

Test your room before you trust it:

Listen for four things: are both voices clearly distinct, is the HVAC audible, does the volume collapse when someone speaks from the back, and is there slap echo off a hard wall. Fix those in the room. No model fixes them for you — see the hard cases in speaker diarization for what happens when you don't.

Configure your Python environment

assemblyai needs to be 1.0 or newer for the assemblyai.prerecorded.v2 namespace used below. One gotcha: pyaudio builds against PortAudio, so on macOS you'll want brew install portaudio first, and on Debian or Ubuntu sudo apt install portaudio19-dev.

Lay the project out like this:

Configuration first. Read the key from the environment; never put it in source:

Record the lecture

Standard PyAudio callback recorder, with a timer so it stops on its own if everyone forgets:

Transcribe with speaker diarization

Here's the part that does the actual work. Set speaker_labels=True and the transcript comes back as an array of utterances — one uninterrupted stretch of speech from one speaker — instead of a single block of text.

Two things worth knowing before you read the code.

First, the speaker count parameters are hard boundaries, not hints. max_speakers_expected is a strict cap: if eleven people speak and you set it to four, speakers five through eleven get merged into the four labels you allowed. And min_speakers_expected is a strict floor, which will happily split one person into three if you set it too high. The defaults are duration-based and sensible for lectures — no cap under 2 minutes, 10 speakers for 2 to 10 minutes, and 30 speakers for anything over 10 minutes. A full lecture lands in that last bucket, so the honest advice is to leave the range unset unless you know your room.

Second, pin your model. Passing speech_models=["universal-3-5-pro", "universal-2"] gets you Universal-3.5 Pro — 18 languages natively, with the most accurate diarization we've shipped — and automatic fallback to Universal-2 for anything outside that set, which covers 99 languages in total. That fallback matters more in a university than almost anywhere else. If you'd rather see how multilingual rooms behave in practice, there's a separate walkthrough on detecting, diarizing, and transcribing across languages.

Note the speech_model_used field on the response. It tells you which model actually ran, which is how you find out that last Tuesday's guest lecture in Portuguese quietly routed to Universal-2.

Put roles on the speakers, carefully

"Speaker A" is safe but not especially useful. The natural next step is naming, and this is exactly where a lecture capture system should slow down.

Speaker Identification maps the generic labels to real names or roles. You give it the labels you expect and it returns a mapping — {"A": "Instructor", "B": "Student"} — and rewrites utterances[].speaker for you. It costs $0.02/hr.

Use speaker_type: "role", not "name". A role label tells a student everything they need ("the professor said this, a classmate said that") and creates no record tying a specific person's voice to their words. That distinction is the whole ballgame under FERPA. If you're fuzzy on the terminology here, diarization vs. recognition vs. identification is worth ten minutes.

Note the speech_model_used field on the response. It tells you which model actually ran, which is how you find out that last Tuesday's guest lecture in Portuguese quietly routed to Universal-2.

Put roles on the speakers, carefully

"Speaker A" is safe but not especially useful. The natural next step is naming, and this is exactly where a lecture capture system should slow down.

Speaker Identification maps the generic labels to real names or roles. You give it the labels you expect and it returns a mapping — {"A": "Instructor", "B": "Student"} — and rewrites utterances[].speaker for you. It costs $0.02/hr.

Use speaker_type: "role", not "name". A role label tells a student everything they need ("the professor said this, a classmate said that") and creates no record tying a specific person's voice to their words. That distinction is the whole ballgame under FERPA. If you're fuzzy on the terminology here, diarization vs. recognition vs. identification is worth ten minutes.

Live captions with streaming diarization

Everything so far assumes the lecture is over. For captions on the projector while class is happening, you want the streaming API instead — and streaming diarization is supported on all three streaming models, so enabling it is one parameter — see the Streaming Diarization documentation for the full message shapes.

It's a genuinely harder problem than async diarization, and it's worth being clear about why. Async diarization sees the whole recording before it decides anything. Streaming diarization has to label a turn the moment it ends, with only the audio it has heard so far. Three consequences you have to design around:

• Turns under about a second come back as "PENDING". There isn't enough audio to build a reliable embedding, and guessing would split one speaker across several labels. Show the text, hold the label.
• The first one or two turns of a session may be wrong. The model hasn't built speaker profiles yet. It self-corrects quickly.
• Overlapping speech gets assigned to one speaker. The model can't split simultaneous voices. Frequent cross-talk degrades everything.

The fix for the first two is SpeakerRevision. When the session ends, the server does a final refinement pass with the whole conversation in view and sends back corrections for any turn whose label changed — one message, about 400 ms, arriving before termination. So you get fast labels during class and correct labels for the file you save. There's more on the design of that in One stream, two jobs.

Here's a complete live-captioning script that handles all of it:

Two parameters there deserve a note. max_speakers accepts 1 through 10 and behaves like its async cousin — a hard cap, so leave headroom. And min_turn_silence / max_turn_silence control turn detection, which on Universal-3.5 Pro is punctuation-based rather than confidence-based. The defaults are tuned for voice agents, where people interrupt; a lecture hall wants longer, so 560 ms and 2000 ms are better starting points than the defaults. If you're migrating from an older streaming model, note that format_turns and end_of_turn_confidence_threshold don't apply here — formatting is always on, and turn detection changed.

Streaming diarization bills at +$0.12/hr on top of the $0.45/hr streaming rate — the full breakdown is on the pricing page.

Generate speaker-attributed captions

Now turn that JSON into files a video player and a learning management system can actually use.

WebVTT has a feature most tutorials skip: the voice span, <v Speaker A>. Players that understand it style the speaker separately from the caption text; players that don't just render the name. Either way the speaker survives into the caption, which is the entire point and something a plain text-only caption loses.

One detail that bites people: those timestamp helpers use integer divmod on the raw millisecond value rather than datetime.timedelta. A 75-minute lecture crosses the one-hour mark, and a naive implementation that reads timedelta.seconds silently drops the hour. Run a caption from the back half of a long file and check that it says 01: and not 00:.

Wire it together

Validate before you trust it

Diarization either works on your rooms or it doesn't, and you find out empirically. Three checks worth running on a real lecture, not a test clip:

1. Count the labels. More speakers than people in the room means over-splitting — usually noise or a too-high min_speakers_expected. Fewer means merging, usually short contributions or a too-low cap.
2. Spot-check the Q&A. It's the hardest part of any lecture recording and the part students use most.
3. Measure it, if you're deploying at scale. There's a real metric for this, and how to measure speaker diarization accuracy with cpWER in Python walks through it. Guessing from vibes across 40 rooms doesn't scale.

Technical requirements and compliance considerations

Audio quality targets, restated as a checklist: 16 kHz sample rate minimum, 16-bit depth or better, background noise low enough that voices stay distinct, consistent levels from every seat you expect speech from, and at least 30 seconds of speech per person you want labeled.

Privacy is the part that gets systems shut down, so treat it as a design constraint rather than a disclaimer. Three frameworks usually apply. FERPA (US) protects students' educational records and restricts unauthorized disclosure, which is the direct argument against building a persistent voice-to-name record of individual students. ADA and Section 508 (for federal institutions) require accessible content — speaker-labeled captions help satisfy this. GDPR (EU) requires a lawful basis and data minimization for processing identifiable personal data, and voice recordings qualify.

That last row is why processor.py prints the transcript ID on every run. You need it to delete a transcript later, and "we can't find it" is not an answer to a deletion request.

On institutional agreements: AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and offers a standard Business Associate Addendum (BAA). That's relevant if your recordings touch a teaching hospital or a clinical program; for ordinary coursework it isn't.

For accessibility, the speaker-labeled captions this pipeline produces support Section 508 and ADA captioning requirements for recorded content. Live captioning during class is what live_captions.py is for.

Final words

The system in this post is four modules and about 400 lines, and the interesting design decision isn't in any of them. It's the choice to stop at roles.

Everything technical here pushes toward more identification: diarization separates the voices, Speaker Identification can name them, and the API will happily accept a roster. But a lecture capture system that knows which student asked which question has created a record that a system labeling them "Student" has not — and the second system is just as useful to the person the tool is actually for. The constraint makes the product better. That's rarer than it sounds.

If you're extending this: the searchable transcript is the obvious hook for a summarizer, and role labels make a much better prompt than Speaker A ever did.

Frequently asked questions

Which Python packages do I need for lecture capture with speaker diarization?

Four: assemblyai (1.0 or newer, for the assemblyai.prerecorded.v2 namespace), pyaudio for recording, sounddevice for the microphone test, and python-dotenv to load your API key. pyaudio compiles against PortAudio, so install that first — brew install portaudio on macOS, sudo apt install portaudio19-dev on Debian or Ubuntu.

What's the right max_speakers_expected value for a lecture?

Usually none. max_speakers_expected is a hard cap, not a hint, and the duration-based defaults already suit lectures: 30 speakers for any file over 10 minutes. Set a range only if you know your room — and when you do, set max_speakers_expected a couple above the count you expect, because setting it too high causes over-splitting and setting it too low merges people together. Use speakers_expected only when you're certain of the exact number; a wrong exact count hurts accuracy more than no constraint at all.

How accurate is speaker diarization in a real classroom?

It depends on the audio, and the biggest lever is how much each person speaks. The model builds a voice embedding per speaker and clusters on it, so accuracy improves the longer someone talks — the docs suggest at least 30 seconds of speech per speaker. Cross-talk, background noise, and one-word contributions are what degrade it. A professor and a handful of question-askers is a well-behaved case; a seminar where twelve people interrupt each other is not.

Can I identify speakers by name instead of "Speaker A"?

Yes, with Speaker Identification (+$0.02/hr), which maps the generic labels to names or roles. In an educational setting, use speaker_type: "role" rather than "name". Role labels like "Instructor" and "Student" give students everything they need without creating a record that links an individual voice to an identity — which is the specific thing FERPA counsels care around. Check with your institution's privacy office before naming anyone.

Is real-time speaker diarization available for live lectures?

Yes. Add speaker_labels: true to your streaming connection parameters; it's supported on all three streaming models, including universal-3-5-pro. Three limitations to design around: turns under about a second are labeled "PENDING" rather than guessed, the first one or two turns of a session may be misassigned while the model builds speaker profiles, and overlapping speech gets attributed to a single speaker. The end-of-session SpeakerRevision message fixes the first two in the saved transcript.

Can I process lecture recordings I already have?

Yes. Any audio or video file at 16 kHz or above works — pass a local path or a URL to transcriber.transcribe() and the SDK handles the upload. Nothing about the async path depends on when the recording was made, which makes back-cataloging an existing archive the cheapest win available to most departments.

What does this cost to run for a semester?

Pre-recorded transcription on Universal-3.5 Pro is $0.21/hr with diarization at +$0.02/hr, so a 15-week course with three 75-minute lectures a week — roughly 56 hours — runs about $13. Role-based Speaker Identification adds $0.02/hr. Live captioning is priced differently: $0.45/hr for streaming plus $0.12/hr for streaming diarization. The free tier covers 185 hours of pre-recorded audio and 333 hours of streaming, which is enough to pilot a couple of courses before you pay anything.

How should I handle student privacy in a diarized lecture recording?

Five things, in order of how often they get skipped: use generic or role labels rather than names, get written consent before recording and put a notice in the syllabus, keep the API key in an environment variable rather than source control, set an automatic deletion policy tied to the end of term, and log every transcript ID so you can honor a deletion request. FERPA restricts unauthorized disclosure of student educational records — talk to your institution's privacy office before deploying anything that creates a persistent, identifiable record of student voices.