How to measure speaker diarization accuracy (cpWER) in Python

A hands-on runbook: compute cpWER in Python to see how good your speaker diarization really is — then score a Universal-3.5 Pro transcript end to end.

Here's a scorer you can paste into a notebook right now:

import numpy as np
import jiwer
from scipy.optimize import linear_sum_assignment

def _error_count(ref, hyp):
    ref = ref.strip() or "<empty>"
    hyp = hyp.strip() or "<empty>"
    out = jiwer.process_words(ref, hyp)
    return out.substitutions + out.deletions + out.insertions

def cpwer(reference, hypothesis):
    """
    reference, hypothesis: dict mapping speaker label -> that speaker's full text.
    Returns (cpWER, best_mapping). cpWER is invariant to how speakers are named.
    """
    ref_speakers = list(reference.keys())
    hyp_speakers = list(hypothesis.keys())
    n = max(len(ref_speakers), len(hyp_speakers))
    ref_padded = ref_speakers + [None] * (n - len(ref_speakers))
    hyp_padded = hyp_speakers + [None] * (n - len(hyp_speakers))

    cost = np.zeros((n, n), dtype=int)
    for i, r in enumerate(ref_padded):
        for j, h in enumerate(hyp_padded):
            cost[i, j] = _error_count(reference.get(r, ""), hypothesis.get(h, ""))

    rows, cols = linear_sum_assignment(cost)
    total_errors = cost[rows, cols].sum()
    total_ref_words = sum(len(t.split()) for t in reference.values())
    mapping = {ref_padded[i]: hyp_padded[j] for i, j in zip(rows, cols)}
    return total_errors / max(total_ref_words, 1), mapping

That's the whole thing. By the end of this post you'll know exactly what it computes, why it beats the metric most people reach for, and how to point it at a real AssemblyAI transcript. Let's dig in.

Why "who spoke when" needs its own metric

Plain word error rate is great at one job: did the system get the words right? Take a two-person interview, transcribe it perfectly, and standard WER can read 0% even if the model credited every one of the host's sentences to the guest. WER never looks at the speaker labels, so speaker mistakes are invisible to it. That's a problem the second your product shows "Speaker A" and "Speaker B" to a user.

Our research team wrote the full argument for why cpWER is the honest way to measure speaker diarization and why DER quietly lies — read the DER-vs-cpWER pillar if you want the deep theory. This post is the other half: the code. So rather than send you off to read a definition, I'll restate it here and then we'll build it.

What cpWER actually measures

cpWER stands for concatenated minimum-permutation word error rate. Three moving parts, and the name tells you all of them.

Concatenated. For each speaker, glue all their words into one stream — one blob of text per person, in both the reference and the hypothesis.

Minimum-permutation. Your system's "Speaker A" isn't guaranteed to be the reference's "Speaker A." So try every way of matching hypothesis speakers to reference speakers and keep the pairing that produces the fewest total word errors.

Word error rate. Under that best pairing, compute a normal WER across everything.

In plain English: for each person in the conversation, what fraction of their words did the system get wrong — misrecognized, dropped, or handed to the wrong speaker? One number, and it only goes up when something a human would actually notice goes wrong.

Now contrast that with DER, the diarization error rate. DER scores seconds of audio, and it scores things a user never sees: silent pauses, segment padding, non-speech annotations. The gap between the two is not subtle. In our testing, a flawless transcript scored 51.5% DER but 0.0% cpWER — DER screaming failure at a perfect result. Flip it around and a catastrophic speaker-attribution output — words shoved onto the wrong people — scored just 15.1% DER but 30.7% cpWER. DER shrugged at output that was genuinely broken. If you're evaluating speech recognition models, that's the exact kind of inversion you can't afford.

Setting up

One install line:

pip install jiwer scipy numpy

jiwer does the word-level alignment, scipy gives you the optimal assignment solver, and numpy holds the cost matrix. That's it — no GPU, no models, no audio processing.

What you actually need to run an evaluation:

• A reference transcript with speaker labels — your ground truth, shaped as {speaker: text}.
• A system hypothesis in the same shape — whatever your diarizer produced.

And here's an underrated part of the pitch. To annotate for cpWER, a human just needs to write down who said what — a speaker-labeled transcript. To annotate for DER, someone has to mark frame-level speech boundaries down to the fraction of a second. That transcript-only labeling runs roughly 10x cheaper than DER's frame-level labels, which matters a lot when you're building an eval set by hand.

Computing cpWER step by step

Look back at the scorer. It does three things in order.

Per-speaker concatenation. The reference and hypothesis are dicts of {speaker: text}, so the concatenation is already done by the time text lands in the dict — one string per speaker.

The assignment step. This is the clever bit. We build an n x n cost matrix where cost[i, j] is the number of word errors you'd get if reference speaker i were matched to hypothesis speaker j. Then linear_sum_assignment — the Hungarian algorithm — finds the pairing that minimizes the total across the diagonal. Padding with None handles the case where the two sides have a different speaker count (the model found three speakers, the truth has two). That's the "minimum-permutation" in the name, done efficiently instead of by brute force.

Scoring with jiwer. Each cell's cost comes from jiwer.process_words, summing substitutions, deletions, and insertions. Total errors under the best mapping, divided by total reference words, gives you cpWER.

Now the payoff — a worked example that proves the metric doesn't care what you name your speakers:

reference = {
    "A": "hey did you finish the report",
    "B": "yeah i sent it this morning",
}

# Same words, correct attribution — but the labels are swapped.
hypothesis = {
    "spk_1": "yeah i sent it this morning",
    "spk_2": "hey did you finish the report",
}

score, mapping = cpwer(reference, hypothesis)
print(f"cpWER: {score * 100:.1f}%")   # cpWER: 0.0%
print(mapping)                         # {'A': 'spk_2', 'B': 'spk_1'}

Words correct, labels flipped, and the score is 0.0%. The assignment step lines A up with spk_2 and B with spk_1, then finds zero word errors. That's label-invariance — exactly what you want, because "Speaker A" is an arbitrary tag, not a fact about the world.

Break the attribution instead of the naming and the score moves. Merge both people into one speaker and cpWER jumps toward the high 80s; introduce a single misrecognized word and you'll see it tick up by one word's worth of error. The metric tracks the mistakes a listener would actually catch — nothing more, nothing less. If you've ever felt burned by how word error rate hides real failures, this is the diarization-aware fix.

Scoring a real transcript with AssemblyAI

Now let's feed it something real. AssemblyAI's speaker diarization runs on Universal-3.5 Pro, our async flagship at $0.21/hr, which produces the transcript and the speaker labels jointly and is optimized for cpWER — not DER. It's tuned to catch short turns, rapid back-and-forth, and overlapped speech, which is where diarizers usually fall apart. (Standard async diarization adds $0.02/hr.)

Install the SDK and transcribe with speaker labels on:

pip install assemblyai

import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

audio_file = "https://assembly.ai/wildfires.mp3"  # or a local path like "./meeting.wav"

config = aai.TranscriptionConfig(speaker_labels=True)

transcript = aai.Transcriber().transcribe(audio_file, config)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

Need an API key? Get your free API key and you can run this in a couple of minutes. The full parameter list lives in the AssemblyAI docs.

The transcript comes back as a list of utterances, each with a speaker tag and text. To score it, fold those utterances into the {speaker: text} shape our scorer expects — concatenate every utterance per speaker:

from collections import defaultdict

hypothesis = defaultdict(str)
for utterance in transcript.utterances:
    hypothesis[utterance.speaker] += " " + utterance.text

# `reference` is your ground-truth transcript in the same {speaker: text} shape
score, mapping = cpwer(reference, dict(hypothesis))
print(f"cpWER: {score * 100:.1f}%")

at's the entire loop: transcribe, group by speaker, score against your ground truth. The mapping tells you which model speaker the scorer matched to each reference speaker, which is handy when you want to eyeball a specific person's errors.

Reading your score: what's good and the traps

So you've got a number. Now what?

For context, Universal-3.5 Pro posts an average cpWER of 30.17. Where do competitors land on the same measure? Deepgram Nova-3 English sits at 37.92, ElevenLabs Scribe v2 at 35.26, and Gladia at 36.87. Lower is better, and the spread between providers is real — this is the kind of gap you can only see once you're measuring the thing users actually experience.

A few things to keep in mind as you interpret your own runs:

Short turns and overlap dominate the error. A one-word "yeah" dropped in the middle of someone else's sentence is brutal for any diarizer, and those moments contribute a disproportionate share of cpWER. If your score is high, listen to the rapid back-and-forth sections first — that's usually where it's coming from.

Formatting choices swing DER but not cpWER. This is the trap that sends people down rabbit holes. In our testing, changing a single segmentation setting — max_gap — swung DER from 47.8% to 14.5% on identical model output. Toggling whether laughter was included in the reference swung DER from 23.4% to 1.1%. Same words out of the model, wildly different DER, purely from annotation and segmentation knobs. cpWER doesn't budge for any of that, because it scores words, not seconds. When your metric moves and the transcript didn't, that's DER noise — not a real quality change.

Compare like with like. cpWER is text-normalization sensitive (casing, punctuation, numbers). Keep your normalization identical across reference and hypothesis, and identical across the providers you're comparing, or you'll be measuring your preprocessing instead of the model.

Next steps

You've now got a self-contained, tested cpWER scorer and a way to point it at production diarization output. From here:

• Build a small labeled eval set from your own audio — remember, transcript-only annotation is about 10x cheaper than DER labeling, so this is more achievable than it sounds.
• Run the same set through multiple providers and rank them on cpWER, not DER or plain WER. If you're surveying options, our roundup of speaker diarization libraries and APIs is a good starting map.
• Kick the tires without writing any setup code in the AssemblyAI playground, then wire the SDK into your pipeline.

When you're ready to score a real transcript, get your free API key and run the two code blocks above end to end. Measure the thing your users actually see — then go make it better.

Frequently asked questions

What is cpWER?

cpWER is concatenated minimum-permutation word error rate. It concatenates each speaker's words into one stream, finds the reference-to-hypothesis speaker mapping that minimizes total word error, and reports the word error rate under that best mapping. In plain terms: for each person, what fraction of their words did the system get wrong — misrecognized, dropped, or credited to the wrong speaker.

What's the difference between cpWER and DER?

cpWER scores words a listener actually reads; DER scores seconds of audio, including things users never see like silent pauses, segment padding, and non-speech annotations. The two can disagree dramatically — a flawless transcript scored 51.5% DER but 0.0% cpWER in our testing, while a badly mis-attributed one scored just 15.1% DER but 30.7% cpWER. cpWER is also cheaper to annotate, since it needs only a speaker-labeled transcript.

How do you measure speaker diarization accuracy in Python?

Represent your reference and hypothesis as {speaker: text} dicts, then use jiwer for word-level alignment and scipy's linear_sum_assignment to find the best speaker mapping. The cpwer() function in this post does exactly that — install with pip install jiwer scipy numpy and pass in your two dicts.

Does AssemblyAI support speaker diarization?

Yes. Set speaker_labels=True in your TranscriptionConfig and Universal-3.5 Pro returns per-utterance speaker labels alongside the transcript. It's optimized for cpWER and built to handle short turns, rapid back-and-forth, and overlapped speech. Standard async diarization adds $0.02/hr.

Why is diarization error rate (DER) misleading?

Because DER moves when the words don't. In our testing, changing one segmentation setting swung DER from 47.8% to 14.5% on identical model output, and toggling whether laughter was in the reference swung it from 23.4% to 1.1%. Those are annotation and formatting artifacts, not quality changes. cpWER ignores them because it scores words instead of seconds.

How much labeled data do I need to compute cpWER?

Just a speaker-labeled transcript of your audio — who said what, in order. You don't need frame-level speech boundaries, which is why cpWER annotation runs roughly 10x cheaper than DER annotation. Start with a handful of representative clips and grow the set as you compare providers.