The hard cases in speaker diarization: overlap, short turns, and noise

Speaker diarization looks solved right up until it isn't. Point a decent model at a clean two-person phone call and it'll nail who said what, turn after turn, with almost no effort. That's the demo everyone shows.

Now drop the same model into a dinner party. Four people, a cross-talk moment, somebody laughing mid-sentence, a "yeah, exactly" fired off while the host is still finishing a thought. The clean-call model falls apart—and it falls apart in specific, nameable ways.

This post is that list. Not "diarization is hard, be careful," but the actual taxonomy: the exact cases that break systems, why each one breaks them, and—the part almost nobody writes down—how you'd even know it's failing. Because here's the uncomfortable part: the metric most people quote barely notices some of the worst failures.

Why diarization is still hard

Diarization is two jobs stapled together. First, figure out how many distinct speakers exist. Second, draw the boundaries that say person A talked from here to here, then person B took over. Do both perfectly and you get a clean speaker-labeled transcript. Miss on either and the errors compound.

On a clean call, both jobs are easy. Two clearly different voices, one talking at a time, generous pauses between turns—the acoustic signal practically labels itself. If you want the full mechanics of how this works under the hood, we wrote a complete explainer on speaker diarization that walks through the pipeline.

Real conversations don't cooperate. People interrupt. They react. They talk over each other. They mumble from across a room. And the moment reality gets messy, the failures sort into a handful of recurring categories. Let's name them.

Overlapped speech

Two people talking at once is the single biggest source of diarization error. Full stop.

Here's why it's so brutal. Most diarization systems are built on an assumption baked deep into their architecture—that at any given instant, exactly one person is speaking. That assumption makes the math tractable. It also makes overlap a blind spot, because when two voices land in the same slice of audio, the system has to pick one. The other speaker's words just vanish.

Some systems handle overlap by masking it—detecting the overlapped region and excluding it from analysis so it doesn't corrupt the clustering. That keeps the speaker labels cleaner, sure. But think about what you just did: you threw away the audio where two people were talking. Every word in that region is gone from the transcript. For a meeting-summarization use case or a call-center QA workflow, those are often the words that matter most—the interruption, the objection, the correction.

So what does good look like? A model that treats overlap as a first-class event, not noise to be filtered. It should attribute overlapping speech to the right speakers and keep both sets of words, instead of picking a winner and dropping the loser. That's a real bar, and most legacy pipelines don't clear it.

Short turns and back-channels

"Yep." "No." "Right." A quick laugh. A "wait, what?" thrown in while someone else holds the floor.

These are back-channels—the tiny verbal reactions that keep a conversation moving—and they wreck diarization in a sneaky way. The system credits them to whoever was already speaking. Person A is mid-sentence, Person B says "exactly," and the transcript hands that "exactly" to Person A. The turn was too short and too embedded to trigger a speaker switch, so it gets swallowed.

Now here's the part that should genuinely change how you evaluate diarization. You might never see this failure in your metrics.

The standard metric, DER (diarization error rate), weights errors by seconds of audio. A back-channel is maybe 300 milliseconds. Misattribute it and DER shrugs—it barely moves. So you can have a system that systematically misfiles every short reaction in your data and still posts a DER that looks great on a slide.

We ran the numbers on this and the gap is almost comical. In our own testing, a catastrophic attribution output—words credited to the wrong people all over the place—scored just 15.1% DER. Sounds fine, right? The same output scored 30.7% cpWER. Twice as bad by the metric that actually tracks whether each person's words landed in the right place.

It cuts the other way too. A flawless transcript—every word right, every word attributed correctly—scored 51.5% DER but 0.0% cpWER. DER called a perfect result a failure. If you're optimizing for DER, you're optimizing for the wrong thing, and short turns are exactly where that bites. We broke this whole comparison down in DER vs cpWER if you want the full teardown.

Background noise and far-field audio

Not every conversation happens through a headset an inch from someone's mouth. A lot of the most valuable audio is captured in the worst possible conditions.

Think about a drive-thru: engine noise, wind, a speaker box, someone in the back seat. Or a self-service kiosk in a busy lobby. Or a conference room where the mic is in the center of the table and half the speakers are eight feet away. Or a phone on speaker sitting on a desk. This is far-field audio, and it stacks two problems at once.

First, the target voices are quieter and smeared by room reverberation, which blurs the acoustic fingerprint the system uses to tell speakers apart. Second, background noise and background speech—a TV, a nearby conversation, ambient chatter—give the model extra "voices" to trip over. It can invent speakers that don't exist or fail to separate the ones that do.

The fix isn't just a better microphone you don't control. It's a model that can suppress background speech and noise before it corrupts the speaker labels—tuned for whether the audio is near-field or far-field, because those are genuinely different acoustic problems. A single-mic room recording and a multichannel setup call for different strategies too; if you've got separate audio channels per speaker, multichannel diarization sidesteps a lot of this by design.

Getting the speaker count wrong

Every diarization output has a hidden claim baked into it: this conversation had N speakers. Get N wrong and everything downstream is wrong too.

There are two ways to blow it. Merging is when the system collapses two people into one—two similar-sounding colleagues become "Speaker A," and now half of one person's words are filed under the other. Splitting is the opposite: one person's voice shifts—they get excited, they lower their voice, the line quality changes—and the system decides that's a new person, so "Speaker A" and "Speaker C" are actually the same human.

Similar-sounding voices make merging worse. Two men with comparable pitch and cadence, or two people with the same regional accent, sit close together in the model's internal space, and the clustering can't pull them apart.

This is why you need a roster metric—speaker count error—that asks a blunt question: did the system get the number of people right? It's separate from word-level accuracy, and it catches structural failures that a per-word score can smear across the whole transcript. Merges and splits are cheap to make and expensive to ignore.

Ready to test your own hard cases? Try our API for free and run your messiest audio through it—overlap, cross-talk, far-field, all of it.

How to tell if your diarization is actually failing

By now the theme should be clear. The failures that hurt most—short turns, misattribution, cross-talk—are the ones DER is worst at seeing. So if you only track DER, you're flying blind on your hardest cases.

Here's the evaluation stack that actually works, in two numbers.

cpWER (concatenated minimum-permutation word error rate). For each person in the conversation, what fraction of their words did the system get wrong—misrecognized, dropped, or credited to someone else? You concatenate everything each speaker said, find the speaker-mapping that gives the best possible match, and score the errors. Because it's grounded in words, not seconds, a misattributed "exactly" costs exactly as much as it should. cpWER is the number that punishes the failures your users will actually notice.

Speaker count error. The roster metric from the last section—did we get the number of people right? Track it alongside cpWER so structural merges and splits can't hide.

That's the honest scorecard: cpWER for word-level attribution, speaker count error for the roster. If you want to run this yourself, we published a companion walkthrough on measuring cpWER in Python so you can score your own audio instead of trusting a vendor's slide.

How modern models handle the hard cases

Naming the failures is one thing. Building around them is another. Here's how we approach it.

The biggest architectural shift is refusing to treat transcription and diarization as separate steps. Universal-3.5 Pro produces the transcript and the speaker labels jointly—one model, one pass—instead of transcribing first and then bolting speaker labels on afterward. That joint modeling is exactly what lets it catch the short turns, rapid back-and-forth, and overlapped speech that break two-stage pipelines. And it's optimized for cpWER, not DER, because that's the metric that reflects real quality.

The numbers back it up. Universal-3.5 Pro averages 30.17 cpWER against Deepgram Nova-3 English at 37.92, ElevenLabs Scribe v2 at 35.26, and Gladia at 36.87. It runs async at $0.21/hour. Getting started is about as light as it gets:

import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

config = aai.TranscriptionConfig(speaker_labels=True)
transcript = aai.Transcriber().transcribe("./meeting.wav", config)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

Real-time is where diarization gets genuinely thorny, because you can't cluster speakers you haven't heard yet. When someone new talks 30 seconds in, a naive streaming system either guesses early and gets it wrong or stalls waiting for certainty.

Universal-3.5 Pro Realtime handles this with streaming diarization plus revision. It labels speakers live as the audio comes in, then—once it has the full picture—re-clusters and sends a single correction within about half a second of the stream ending. You get low-latency labels during the call and a cleaned-up roster the moment it's over, handling up to 10 speakers. We wrote up the mechanics in streaming speaker diarization and the major streaming upgrade that made this possible.

For the far-field and noise problems, voice_focus with near_field and far_field settings suppresses background speech and noise before it can poison the speaker labels—the difference between a drive-thru transcript that's usable and one that's full of phantom speakers.

Want to see it on your own audio? Poke at it in the playground, read the speaker diarization feature page for the full spec, or just wire it into your stack and run your worst files through it.

See how it handles your hard cases. Try our API for free—no overlap or cross-talk sample too messy.

Frequently asked questions

What are the limitations of speaker diarization?

The main limitations of speaker diarization are overlapping speech, short turns and back-channels, background noise and far-field audio, speaker-count errors, and similar-sounding voices. Overlapping speech is the single biggest source of error, because most systems assume one person talks at a time and drop words when two voices collide. Short reactions like "yep" or "no" get credited to the wrong speaker, and noisy or far-field audio blurs the acoustic differences the system relies on. Modern joint transcription-and-diarization models built and measured for cpWER handle these cases far better than legacy two-stage pipelines.

What is overlapping speech?

Overlapping speech is when two or more people talk at the same time, so their voices land in the same slice of audio. It's the #1 source of diarization error because most systems are built to attribute each moment to a single speaker, which forces them to pick one voice and lose the other's words. Systems that mask overlap keep cleaner speaker labels but delete the transcribed words in the overlapped region—often the interruptions and corrections that matter most.

How does background noise affect speaker diarization?

Background noise degrades diarization by masking the acoustic features a system uses to tell speakers apart and by adding stray sounds it can mistake for extra speakers. Far-field audio—kiosks, drive-thrus, conference rooms, speakerphones—makes it worse, since distance and reverberation blur each voice's fingerprint. Noise suppression tuned for near-field versus far-field capture, like AssemblyAI's voice_focus setting, strips out background speech and noise before it corrupts the speaker labels.

Why do diarization systems get the number of speakers wrong?

Diarization systems miscount speakers in two ways: merging and splitting. Merging collapses two similar-sounding people into one speaker, while splitting treats one person's changing voice—excitement, volume shifts, line-quality changes—as a brand-new speaker. Similar-sounding voices make merging especially common. Tracking speaker count error as a dedicated roster metric is the only reliable way to catch these structural mistakes.

How do you measure speaker diarization accuracy?

Measure diarization accuracy with cpWER (concatenated minimum-permutation word error rate) plus speaker count error, not DER alone. cpWER asks, for each person, what fraction of their words the system got wrong—misrecognized, dropped, or credited to someone else—so it catches short-turn misattribution that DER's seconds-based weighting misses. In our testing a catastrophic attribution output scored just 15.1% DER but 30.7% cpWER, while a flawless transcript scored 51.5% DER yet 0.0% cpWER—proof that DER can grade a perfect result as a failure and a broken one as fine.

‍