Does Whisper do speaker diarization? Whisper + pyannote, and its limits

Does Whisper do speaker diarization?

No. Whisper is OpenAI's automatic speech recognition (ASR) model — it turns audio into text, and that's the whole job. It doesn't tell you who spoke. Hand it a two-person interview and you'll get an accurate transcript with zero speaker labels, no "Speaker A" and "Speaker B," nothing that separates one voice from another.

So if you want speaker diarization — the "who spoke when" layer on top of the words — you have to bolt it on yourself. Most people reach for pyannote.audio (or WhisperX, which wraps a similar idea). That works. But it's a pipeline you assemble and own, and the seams show up exactly where diarization is hardest: short turns, rapid back-and-forth, and overlapping speech.

Let's build the DIY version first, then measure it honestly.

How people add diarization to Whisper

The standard recipe has three stages. Transcribe with Whisper to get words and timestamps. Run a separate diarization model — pyannote — to get speaker segments. Then align the two: for each word, figure out which speaker segment it belongs to.

Here's the shape of it in Python. This is illustrative — the point is the structure, not a production-ready alignment engine:

# pip install openai-whisper pyannote.audio torch
import whisper
from pyannote.audio import Pipeline

# 1. Transcribe with Whisper (word-level timestamps)
asr = whisper.load_model("large-v3")
result = asr.transcribe("meeting.wav", word_timestamps=True)

# 2. Diarize with pyannote (needs a free Hugging Face access token)
diarizer = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.1",
    use_auth_token="<HF_TOKEN>",
)
diarization = diarizer("meeting.wav")

# 3. Align: for each Whisper word, find the diarization segment its
#    midpoint falls in, and attach that speaker label. (You write this glue.)

Looks clean. Three steps, a few dozen lines. But here's where it gets interesting — every one of those steps hides real friction.

The pyannote models are gated. You need a free Hugging Face account, you have to accept the model's user conditions, and you generate an access token to pass into from_pretrained. Miss that and the download fails. It's not hard, but it's a step, and it's a credential you now manage.

You'll want a GPU. Whisper's large-v3 and pyannote both run painfully slow on CPU. For anything beyond a short clip, you're provisioning and maintaining a GPU — locally or in the cloud — plus the CUDA and PyTorch versions that keep them happy.

There's voice-activity detection in the mix too. pyannote leans on VAD to decide where speech is before it decides who's speaking, and tuning that on noisy or far-field audio is its own rabbit hole.

And then there's the alignment glue — step 3, the innocent-looking comment. That's your code. You're mapping Whisper's word timestamps onto pyannote's speaker segments, deciding what to do when a word straddles a boundary, what to do when the diarizer and the ASR disagree about where speech even is. The midpoint heuristic in the snippet is the naive version. It works until it doesn't, and when it doesn't, you own the bug.

None of this is a knock on Whisper or pyannote. They're solid open-source tools, and for an offline project or a weekend build, this pipeline is genuinely fine. The question is what happens to accuracy when the conversation gets messy — and how you'd even know.

Where the cracks show: measuring it with cpWER

Most diarization tutorials measure quality with DER — diarization error rate. DER asks, roughly, what fraction of audio time got assigned to the wrong speaker. It's the classic academic metric, and it has a blind spot big enough to drive a truck through: it barely notices short turns, and it can wave through catastrophic attribution errors.

The metric that actually reflects what your users experience is cpWER — concatenated minimum-permutation word error rate. Here's the plain-English version: for each person in the conversation, what fraction of their words did the system get wrong? Wrong meaning misrecognized, dropped, or credited to the wrong speaker. You concatenate everything each speaker actually said, compare it against what the system attributed to them (under the best speaker-to-speaker matching), and count the damage. It folds transcription errors and attribution errors into one honest number.

Why does the choice of metric matter so much? Because DER and cpWER can tell completely different stories about the same output. In AssemblyAI's testing, a catastrophic attribution failure — a case where speakers got badly swapped — scored a tidy 15.1% DER while landing at 30.7% cpWER. The DER made it look like a B-plus. The cpWER told the truth: nearly a third of the words were attributed to the wrong person. If you're reading a transcript, that's the difference between "mostly right" and "who said that?"

Now think about where the Whisper + pyannote pipeline is most likely to break.

Short turns. When someone jumps in with a quick "Right," "No, wait," or "Exactly," that turn might be half a second long. The midpoint-alignment heuristic has almost no signal to work with, and the interjection frequently gets swallowed into the neighboring speaker's block. DER shrugs — it's a sliver of audio. cpWER counts every one of those words as misattributed.

Overlap. Two people talking at once is the reality of real meetings, debates, and calls. A pipeline that runs VAD, then diarization, then alignment tends to collapse overlapped speech into a single speaker — one voice wins, the other's words vanish or get reassigned. Those dropped and misattributed words hit cpWER directly.

Speaker merging. On long recordings, a diarizer can decide two people are one, folding one speaker's turns into another. DER, again, can under-penalize this depending on the timing. cpWER makes the merge obvious, because a whole speaker's worth of words landed under the wrong label.

The uncomfortable takeaway: you can stand up the Whisper + pyannote pipeline, run DER, see a respectable number, and ship something that mangles exactly the moments people care about most. If you want to measure this properly on your own audio, AssemblyAI's companion walkthrough on measuring cpWER in Python shows how, and the DER vs. cpWER breakdown explains why the metric you pick changes the verdict.

Want to skip the pipeline and get diarization that's tuned for cpWER out of the box? Get your free API key and try it on your own audio.

The managed alternative: diarization built into the model

Here's the other way to do this. Instead of stitching an ASR model to a diarization model and writing the glue between them, you call one API that produces the transcript and the speaker labels together.

AssemblyAI's Universal-3.5 Pro is the async flagship, and it's built for exactly this. It generates the transcript and speaker labels jointly, and it's optimized for cpWER — the honest metric — not just DER. Because diarization is produced alongside the words rather than aligned after the fact, it holds up on the hard cases: short turns, rapid back-and-forth, and overlapped speech.

Here's the whole thing in Python:

import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

audio_file = "https://assembly.ai/wildfires.mp3"  # or "./meeting.wav"

config = aai.TranscriptionConfig(speaker_labels=True)

transcript = aai.Transcriber().transcribe(audio_file, config)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

That's pip install assemblyai and a single speaker_labels=True flag. No Hugging Face token. No gated model downloads. No alignment heuristic to write and debug. No GPU to provision, patch, or keep alive. The utterances come back already labeled by speaker, in order, ready to print or store.

On price, Universal-3.5 Pro runs $0.21/hr, and standard diarization is an additional $0.02/hr — a rounding error against the cost of running your own GPU box and maintaining the pipeline around it. On accuracy, the numbers hold up where it counts: Universal-3.5 Pro averages 30.17 cpWER, compared with 37.92 for Deepgram Nova-3 English, 35.26 for ElevenLabs Scribe v2, and 36.87 for Gladia. Lower is better, and the gap is largest exactly on the short-turn and overlap cases that a DIY pipeline fumbles. The full Universal-3.5 Pro async release has the details.

This isn't magic — it's just diarization treated as a first-class part of the model instead of a post-processing step you assemble. And it means the messy moments in real conversations get attributed correctly instead of collapsed.

Whisper + pyannote vs. Universal-3.5 Pro: when to use which

Both are legitimate choices. Here's the honest comparison.

So when should you reach for each?

Go with Whisper + pyannote if you need everything to run offline or air-gapped, if you're doing research or a hobby build where the pipeline itself is the point, if you have a GPU sitting idle, and if cost pressure means you'd rather spend engineering time than dollars. It's a fine tool for that.

Go with Universal-3.5 Pro if accuracy on real conversations matters — short turns, cross-talk, overlap — if you don't want to own a GPU and an alignment script, if you'd rather ship in an afternoon than tune VAD for a week, and if you want a number you can trust because it's measured with cpWER. For most production speech-to-text work, that's the trade that pays off.

The thing is, most teams reach for Whisper + pyannote because it looks free and turns out to be expensive — in GPU time, in glue code, and in the accuracy you quietly lose on the hardest turns. Priced against that, a managed API optimized for the honest metric is usually the cheaper answer.

Next steps

If you're building diarization into a product, the fastest way to compare is to run the same audio through both. Grab a messy clip — an interview with interruptions, a meeting with cross-talk — and see which one attributes the short turns correctly.

• Try it live in the playground — paste a file, get labeled speakers, no code.
• Read the speaker diarization docs for the async API details.
• See the speaker diarization feature overview for how it fits into a full pipeline.
• Compare options in our roundup of the top speaker diarization libraries and APIs and the top free speech-to-text APIs and open-source engines.

Ready to skip the GPU and the glue code? Get your free API key and add speaker labels with one line of config.

Frequently asked questions

Does Whisper support speaker diarization?

 No. Whisper is an ASR model — it transcribes speech to text but doesn't identify who's speaking. To get speaker labels you have to pair it with a separate diarization model like pyannote.audio, or use a managed API that does both jointly.

How do you add speaker diarization to Whisper?

 The common approach is a three-stage pipeline: transcribe with Whisper for word-level timestamps, run pyannote.audio (or WhisperX) to get speaker segments, then align each word to a speaker turn with your own glue code. pyannote's models are gated, so you'll need a free Hugging Face token, and a GPU is strongly recommended.

Is Whisper + pyannote free? 

The software is open source, so there's no license fee. But "free" is misleading — you'll typically need a GPU to run it at any real speed, plus engineering time to build and maintain the alignment step, manage credentials, and tune voice-activity detection. The dollars move from a per-hour bill to your infrastructure and your team's hours.

How accurate is Whisper speaker diarization? 

Whisper itself is a strong transcriber; the diarization accuracy depends on pyannote and your alignment code. Measured with cpWER — the metric that counts misattributed words — DIY pipelines tend to struggle on short turns, rapid back-and-forth, and overlapping speech, precisely the cases DER can hide. Always measure with cpWER, not just DER.

What's the difference between DER and cpWER? 

DER (diarization error rate) measures the fraction of audio time attributed to the wrong speaker, and it under-penalizes short turns and attribution swaps. cpWER (concatenated minimum-permutation word error rate) measures, per speaker, the fraction of their words that were misrecognized, dropped, or credited to someone else. In one AssemblyAI test, a bad attribution output scored just 15.1% DER but 30.7% cpWER — same output, very different story.

What's a simpler alternative to Whisper + pyannote?

 AssemblyAI's Universal-3.5 Pro async model produces the transcript and speaker labels together from a single API call — no Hugging Face token, no alignment script, no GPU to manage. It's optimized for cpWER, averages 30.17 cpWER, and costs $0.21/hr plus $0.02/hr for standard diarization.

‍