Speaker diarization: Speaker labels for mono channel files

Most audio arrives as one channel with several people in it. A recorded sales call, a Zoom export, a clinic visit, a podcast episode recorded on a single room mic — one waveform, multiple voices, no metadata telling you which is which. Transcribe that and you get a wall of accurate text that is nearly useless, because "I'll send the contract over tomorrow" means one thing from the seller and something else entirely from the buyer.

Speaker diarization is the step that fixes it.

It's the process of automatically partitioning an audio stream into segments by speaker identity — answering "who spoke when" without knowing in advance who anyone is, or even how many people are in the room. Combine it with transcription and every word carries an owner. That's the difference between a transcript and a conversation.

What is speaker diarization?

Speaker diarization takes a single-channel recording and produces time-stamped segments labeled by speaker: Speaker A from 0:00 to 0:14, Speaker B from 0:14 to 0:31, and so on. The labels are relative, not absolute. The system doesn't know that Speaker A is Priya from procurement; it knows that the voice in those segments is the same voice, and that it's different from the other one.

That distinction matters more than it sounds. Diarization is an unsupervised clustering problem — no enrollment, no voiceprint database, no prior samples. Speaker recognition, by contrast, matches a voice against known individuals. They're often confused, and the confusion leads teams to expect names when what they'll get is letters. If you want the ground-level explainer first, start with what speaker diarization is and how it works.

You can bridge that gap. AssemblyAI's Speaker Identification feature maps generic labels onto real names or roles when you have something to map from — a participant list, a calendar invite, a CRM record.

The mono-channel constraint is the whole reason this is hard. If you have a stereo recording where each participant sits on their own channel — the way most VoIP systems record outbound calls — you don't need diarization at all. Split the channels and you're done. But the interesting audio is almost always mixed down: conference room mics, mobile recordings, podcast stems bounced to a single track, telehealth sessions captured through a browser. One channel, several voices, and a model that has to sort them out from the acoustics alone.

Why speaker diarization matters

Speaker-attributed text unlocks a category of analysis that undifferentiated text can't support.

In sales, talk-time ratio is one of the most reliable predictors of deal outcome, and you cannot compute it without knowing who spoke. In contact centers, agent-versus-customer separation is what makes call center analytics work at all — sentiment scored across a merged transcript averages the frustrated customer and the calm agent into meaningless neutrality. In healthcare, separating clinician speech from patient speech is what turns a recording into a usable clinical note. In media, it's how you build a searchable archive where "find every time the CFO discussed margins" returns something.

There's a second-order effect too. Diarization improves the transcript itself. Speaker turn boundaries give downstream models structure to work with, which is why speaker-aware conversation intelligence systems consistently outperform ones built on flat text.

How speaker diarization works

Nearly every production diarization system runs the same four stages, whether it's open source or a hosted API.

Voice activity detection. First, find the speech. VAD separates speech regions from silence, music, keyboard noise and the HVAC hum that a surprising number of pipelines mistake for a very quiet speaker. Everything downstream inherits VAD's mistakes, so this stage matters more than its simplicity suggests.

Segmentation. Cut the speech into homogeneous chunks, each ideally containing one speaker. Classical systems cut at fixed intervals and hoped; modern ones detect change points, looking for the acoustic discontinuity where one voice stops and another starts.

Embedding extraction. Convert each segment into a fixed-length vector that encodes voice characteristics — pitch, formant structure, vocal tract shape, speaking style — while discarding the words themselves. Two segments from the same speaker should land near each other in that space regardless of what was said.

Clustering. Group the embeddings. Each cluster becomes a speaker label. This is where the speaker count gets decided, and where most diarization errors originate.

The field's biggest jump came from replacing the embedding step. Traditional systems used i-vector embeddings, a statistical representation borrowed from speaker verification. Neural d-vectors, typically LSTM-based, learn the representation directly from data and hold up far better under noise, compression and channel variation.

Main approaches to speaker diarization

Clustering-based approaches

The classical pipeline: segment, embed, cluster. K-means with KMeans++ initialization is common when you know the speaker count; spectral clustering is the stronger default when you don't, since it estimates the number of clusters from the eigenstructure of the affinity matrix rather than being told. The standard recipe is to construct the affinity matrix, apply refinement operations, run eigen-decomposition on the refined matrix, then cluster the resulting embeddings.

Clustering approaches see the entire recording before deciding anything, which is exactly why they're accurate on batch audio and unusable on a live stream.

Neural clustering methods like UIS-RNN replaced the hand-tuned clustering step with a learned one, letting the model infer speaker count as part of inference.

End-to-end neural approaches

Instead of a pipeline, one model. End-to-end neural diarization takes audio in and emits per-speaker activity directly, optimizing the whole task jointly rather than optimizing four stages that were never trained to work together.

The practical win is overlapping speech. A clustering pipeline assigns each segment to exactly one speaker by construction, so simultaneous talking is unrepresentable. An end-to-end model can output two speakers active in the same frame, which is what actually happens in real conversation. Related work on source separation takes the further step of pulling apart the overlapping signals themselves.

Hybrid pipeline systems

Most production systems, ours included, are hybrids: neural components in the stages where neural components win, classical machinery where it's more predictable and cheaper. You get to optimize each stage independently and swap one without retraining everything. The cost is complexity — more moving parts, more places for a regression to hide.

What AssemblyAI's diarization actually does

Here's the part the research literature can't tell you: what you get when you turn it on.

Diarization on recorded audio

For pre-recorded files, set speaker_labels: true and the response includes an utterances array where every utterance carries a speaker label alongside its text and timestamps. You don't have to declare the speaker count — the model infers it — but you can constrain it with speakers_expected when you know the exact number, or with min_speakers_expected and max_speakers_expected when you only know a range. These are hard boundaries rather than hints, so an incorrect exact count will split one speaker in two or merge two into one.

Diarization runs against whichever model handles the file. Our async flagship, Universal-3.5 Pro, is described on our pricing page as our most accurate speaker diarization yet — the diarization gains came with the model, not as a bolt-on.

Diarization on live streams

Streaming diarization is a genuinely different problem, and for a long time the honest answer in this industry was "wait for the file." Universal-3.5 Pro Realtime returns word-level speaker labels as the audio arrives, via the same speaker_labels: true flag, with meaningful reductions in phantom speakers and spurious turn counts compared with naive online clustering.

The trick is that it doesn't pretend online clustering is as good as offline clustering. It isn't — an online system has to commit to a label before it has heard the rest of the conversation. So the model re-clusters when the stream ends and sends a single correction pass, typically within about half a second of stream end. Live labels while you need them, batch-quality labels once the call is over. Up to 10 speakers, on our streaming speech-to-text surface.

What it costs

Two cents an hour for pre-recorded diarization is small enough that the build-versus-buy math almost never favors building. Everything is pay-as-you-go and billed per second, with no minimums.

Evaluating speaker diarization quality

Diarization error rate

DER is the standard metric, and it's the sum of three failure modes: false alarm (speech detected where there was none), missed speech (speech the system didn't detect), and speaker error (speech attributed to the wrong person). Leading commercial systems reach DER below 5% under favorable conditions — clean audio, few speakers, minimal overlap.

Treat that number the way you'd treat any benchmark: as a ceiling, not an expectation. DER on curated corpora and DER on your actual audio are different numbers, and the gap is usually large.

Speaker count accuracy

Getting the count right is often more important than getting boundaries perfect. Splitting one person into two speakers scrambles every downstream metric — talk-time ratios, per-speaker sentiment, question counts — in a way a few misplaced boundary milliseconds never will.

Robustness

The metric that predicts production behavior is how gracefully quality degrades as conditions worsen: background noise, codec compression, cross-talk, speakers who sound alike. A system that posts excellent numbers on clean audio and collapses on a speakerphone is worse in practice than one that's merely good at both. If you want to measure this properly on your own files, we published a walkthrough of measuring diarization accuracy with cpWER in Python.

Challenges and limitations

Overlapping speech

People interrupt. They agree out loud while someone else is still talking. Any pipeline that assigns each frame to exactly one speaker is structurally incapable of representing that, and one recent study documents how sharply performance falls as overlap increases. End-to-end neural approaches are the current best answer.

Many speakers

Accuracy degrades as the roster grows, and it degrades fast. Research findings demonstrate word diarization error rates jumping from 2.68% in two-speaker scenarios to 11.65% with three speakers. Most systems perform well with two to five speakers and can handle 10 to 15 with declining accuracy.

Design around this. A four-person meeting is a solved problem; a 20-person all-hands is not, and treating them as the same workload will disappoint you.

Short speaker turns

A one-word "right" gives the embedding model almost nothing to work with. Backchannels, quick confirmations and rapid-fire exchanges are where diarization systems produce their most visible errors — and unfortunately those short turns are exactly where conversational signal lives. We dug into this and other failure modes in our post on speaker diarization hard cases.

Environmental factors

Reverberant rooms, distant mics, road noise, aggressive codecs. Modern neural embeddings are considerably more robust than i-vectors here, with documented improvements around 30% in noisy environments, but noise still costs you.

Similar voices

Two speakers of the same gender, similar age and similar regional accent produce embeddings that sit close together. Family members and long-time colleagues are the hard cases — people who share vocal characteristics and, often, speech patterns.

Active areas of research

Four threads are moving fastest right now. Joint modeling of ASR and diarization, so the words and the speakers are predicted together rather than stitched afterward. Target-speaker extraction, which conditions the model on a reference voice. Self-supervised speaker representations that learn from unlabeled audio at scale. And online diarization, where recent research found processing time can be as low as 3% of the speech segment's duration — efficient enough that the constraint on live diarization is accuracy, not compute.

Speaker diarization use cases

Telemedicine and clinical documentation

Clinical conversations are the highest-stakes diarization workload there is. Attributing a symptom description to the clinician instead of the patient doesn't just produce a messy note — it produces a wrong one. And the record matters: research shows patient history contributes to 76% of initial diagnoses.

Two things matter here beyond the labels themselves.

The first is terminology. Drug names, dosages and procedure names are exactly the words a general model drops, and a diarized transcript with the wrong medication in it is worse than no transcript. Medical Mode is a single parameter — domain: "medical-v1", paired with speech_models: ["universal-3-5-pro"] on pre-recorded audio or speech_model: "universal-3-5-pro" on streaming — with no model switch and no separate endpoint. It posts a 3.2% Missed Entity Rate and makes 87% fewer medical entity errors than the base model, and it's the lowest Missed Entity Rate across the providers we benchmark against. The numbers and methodology live on our benchmarks page. It covers English, Spanish, German and French across pre-recorded and streaming, and runs $0.15/hr on top of the base rate — $0.36/hr all-in with the async flagship.

The second is the legal posture, which people get wrong in both directions. Here's the accurate version:

AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI.

The BAA can be signed in minutes without a sales call. PII and PHI redaction is available across both audio and transcripts, and the compliance stack behind it is SOC 2 Type 2, ISO 27001:2022 and PCI DSS v4.0. If you're building in this space, our medical transcription solutions page is the right starting point.

Contact centers and conference calls

Agent-versus-customer separation is the foundation of nearly every contact center metric worth tracking: talk-time balance, interruption counts, silence duration, per-side sentiment, adherence to script. Without diarization you can measure none of them.

Calabrio builds workforce engagement software on top of exactly this kind of separated transcript. As Raj Shankar, SVP Product, Calabrio put it:

The transcription accuracy, reliability, and speed of AssemblyAI's API have greatly enhanced our operations, reinforcing our trust in their technology and solidifying our partnership.

For live deployments — real-time agent assist, supervisor monitoring, in-call compliance checks — streaming diarization is the enabling piece, since a label that arrives after the call ends can't change how the call goes. More on that pattern in our contact center solutions overview and our post on real-time agent assist.

Hiring platforms

Interview intelligence needs to know who asked and who answered, and it needs to survive the acoustics of a laptop mic in an open-plan office. It also needs context — names, companies, role titles — that no acoustic model can invent.

Metaview built recruiting-specific interview intelligence on AssemblyAI. Shahriar Tajbakhsh, Co-founder and CTO, Metaview:

Since moving to AssemblyAI, we've seen a meaningful improvement in the confidence tail of our production transcripts....What stands out is not just the model quality, but the way [they] let us bring real meeting context into transcription, from calendar titles to organizations, domains, and participant names, so recruiting conversations come through with the nuance our customers depend on.

Podcast and video hosting

Speaker-labeled transcripts make episodes searchable by person, let you generate per-guest clips automatically, and turn an archive into something navigable. For platforms hosting other people's content, it's the difference between storing media and offering a product. Both benefit from transcript search built on labeled turns.

Broadcast media

Panel shows, live debates, news roundtables — many speakers, frequent interruptions, variable mic quality. It's the hardest common case, and the one where speaker count accuracy and overlap handling matter most.

How to enable speaker diarization with AssemblyAI

Enabling speaker diarization is one parameter. The Python SDK handles authentication, file upload and polling for you, so the whole thing is a few lines.

import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    speaker_labels=True,
)

transcriber = aai.Transcriber(config=config)
transcript = transcriber.transcribe("./audio/meeting.mp3")

if transcript.status == "error":
    raise RuntimeError(transcript.error)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

‍

Set the models explicitly rather than riding the account default. speech_models takes an ordered list — Universal-3.5 Pro handles the request, and Universal-2 catches anything outside the flagship's 18 supported languages, which is how you get diarization across 99 languages without branching your code. The response field speech_model_used tells you which one ran.

For live audio, diarization is a connection parameter on the streaming socket:

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "speaker_labels": True,
}

Note the singular speech_model on streaming versus the plural speech_models array on pre-recorded audio. It trips people up constantly. Full parameter reference and language options are in the speaker diarization documentation.

Getting started with speaker diarization

The build-versus-buy question here is less interesting than it used to be. Open-source diarization is genuinely good, and if you're doing research or have unusual constraints, it's a reasonable path. But at +$0.02/hr for pre-recorded audio, the API cost is not what decides this. What decides it is that diarization quality is set by the hard cases — the overlapping speech, the three-second turns, the two colleagues who sound alike — and those cases are where a self-hosted pipeline quietly costs you a quarter of an engineer forever.

Here's the thing worth internalizing, though: diarization is no longer a post-processing step you bolt onto transcription. The gains in our own system came from the speech model itself, and the streaming design — live labels, then a single re-clustering correction at stream end — only works because the labeler and the recognizer are the same system. Teams still architecting around "transcribe, then diarize" are leaving accuracy on the table that the pipeline shape can't recover.

Run it on your own audio. Not a benchmark clip — your worst real recording, the one with the speakerphone and the four people. That's the number that matters.

Frequently asked questions

What is speaker diarization and how does it work in speech recognition?

Speaker diarization is the process of automatically partitioning an audio recording into segments labeled by speaker, answering "who spoke when" without knowing the participants in advance. It works in four stages: voice activity detection to find speech, segmentation to cut it into single-speaker chunks, embedding extraction to convert each chunk into a voice vector, and clustering to group those vectors into speakers. In a speech recognition pipeline the labels are then aligned to the transcript, so every utterance carries a speaker field alongside its text and timestamps. Modern systems use neural d-vector embeddings rather than older i-vectors, which is most of the reason accuracy improved so sharply under noisy conditions.

How does AssemblyAI identify and separate multiple speakers?

AssemblyAI separates speakers with a hybrid neural pipeline that you enable by setting speaker_labels: true — no enrollment, no voiceprints and no need to declare the speaker count. On pre-recorded audio the response includes an utterances array where each entry carries a speaker label; Universal-3.5 Pro is described on our pricing page as our most accurate speaker diarization yet. Labels are generic by default (Speaker A, Speaker B), and Speaker Identification maps them to real names or roles when you have a participant list to match against. Pre-recorded diarization is a +$0.02/hr add-on.

Is speaker diarization supported for both recorded audio and live streams?

Yes — AssemblyAI ships diarization on both, using the same speaker_labels: true flag. On live streams, Universal-3.5 Pro Realtime returns word-level speaker labels as audio arrives, handles up to 10 speakers, and sends a single re-clustering correction within roughly half a second of the stream ending, so you get live labels during the call and batch-quality labels once it's over. Streaming diarization with revision is +$0.12/hr; pre-recorded diarization is +$0.02/hr. The design is written up in full in our post on streaming speaker diarization.

How many different voices can a transcription API distinguish at once?

AssemblyAI's streaming diarization supports up to 10 speakers, and pre-recorded diarization infers the count from the audio rather than capping it at a fixed number. Practically, accuracy is strongest with two to five speakers and declines beyond that across every system on the market — published research shows word diarization error rates rising from 2.68% with two speakers to 11.65% with three. If your audio routinely has more than five active participants, benchmark it before committing to an architecture.

How do I configure the API to know how many speakers are present in the audio?

You don't have to — the model infers the speaker count on its own once speaker_labels: true is set. When you do know the number in advance, passing an expected speaker count (speakers_expected) constrains the clustering step, and it is a hard boundary rather than a hint, so only set it when you are certain of the exact number. Use min_speakers_expected and max_speakers_expected to describe a range when you're genuinely unsure: an incorrect count is worse than no count at all. The model also returns confidence-bearing utterance boundaries either way, so you can inspect how it segmented the conversation before trusting the count it landed on.

How does the API handle people talking simultaneously in a recording?

Overlapping speech is the hardest case in diarization, and it's the failure mode most worth testing before you commit to a provider. Classical clustering pipelines assign each frame to exactly one speaker by construction, so they can't represent two people talking at once; end-to-end neural approaches can emit multiple speakers active in the same frame, which is why modern systems handle interruptions and backchannels far better. Accuracy still degrades as overlap increases — published research quantifies the drop — so if your audio is full of cross-talk, test with your worst files rather than clean ones. AssemblyAI's pre-recorded diarization sees the full recording before assigning labels, which is why offline results hold up better on overlap than any online system can.

‍