How to improve speaker diarization accuracy: context, speaker counts, and speaker identification

To improve speaker diarization accuracy, give the model context it can't infer from audio alone. Pass an expected speaker count or range, record each speaker on their own channel when you can, keep audio clean, and add names or roles with Speaker Identification. Then measure with cpWER on your own recordings, not a generic benchmark.

Speaker labels go wrong in predictable ways. Two similar voices merge into one label. One person gets split across three. A quick "yeah" lands on the wrong side of a sales call. Each fix below targets one of those failures, and most take a single API parameter. The examples use AssemblyAI's Universal-3.5 Pro, which posts the lowest average cpWER (30.17%) of 10 systems on our published diarization benchmark.

Why speaker labels go wrong

Diarization detects speech, embeds each segment as a voiceprint, and clusters the voiceprints into speakers. (New to the topic? Start with what is speaker diarization.) Errors come from five places:

• Speaker count: the model guesses too few or too many speakers.
• Short turns: back-channels like "mm-hmm" carry too little audio for a confident voiceprint.
• Overlap: two people talk at once, and only one label wins.
• Similar voices: same gender, same accent, same mic, so the embeddings sit close together.
• Noise and far-field audio: room echo and background chatter blur the embeddings.

We break down each failure, with examples, in the hard cases in speaker diarization: overlap, short turns, and noise.

The speaker diarization accuracy checklist

1. Use the most accurate diarization model you can. Set speaker_labels: true on Universal-3.5 Pro (universal-3-5-pro). It produces the transcript and the speaker changes jointly, which helps on short turns and rapid back-and-forth.
2. Pass the exact speaker count when you know it. Use speakers_expected for a two-party support call or a one-on-one interview. Use it only when you're certain.
3. Otherwise, pass a range. Use speaker_options with min_speakers_expected and max_speakers_expected. Set the maximum a little above what you expect, roughly the minimum plus two. The maximum is a hard cap: extra speakers get merged into existing labels, and setting it too high can over-split.
4. Know the defaults. Without a range, the cap is no default for 0–2 minute files, 10 speakers for 2–10 minutes, and 30 speakers above 10 minutes.
5. Give every speaker enough airtime. The docs recommend at least 30 seconds of uninterrupted speech per speaker for reliable labels.
6. Record on separate channels when you can. If each participant has their own channel, use multichannel: true. Channel separation is ground truth, so there's nothing to guess. See using multichannel and speaker diarization.
7. Clean up the capture. Close mics beat room mics. Reduce cross-talk, echo, and background audio. Avoid mixing a phone leg and a studio mic into one mono file when you can keep them apart.
8. Name speakers with context, not enrollment. Add Speaker Identification with a participant list or roles. The model uses the conversation's content to map labels to people.
9. Flag low-confidence attributions. Set include_speaker_confidence: true inside speaker_options to get a speaker_confidence field per word and per utterance. Route low scores to human review.
10. Measure with cpWER on your own audio. cpWER checks that each word is right and attributed to the right speaker. DER can rank systems backwards, as our research team explains in DER vs. cpWER.

How context improves speaker labeling

Audio alone tells a model that two voices differ. Context tells it who those voices are and how many to expect. Three kinds of context matter.

Audio context: what the conversation reveals

People name each other. "Thanks, Priya" is strong evidence about who speaks next. Roles show up in phrasing too: the agent reads the disclosure, and the patient describes the symptoms. Speaker Identification uses these cues to resolve labels.

Metadata context: what you provide before transcription

Calendar invites, CRM records, and scheduling systems already know who's on the call. Pass that as a speaker count, a range, or a participant list. Metaview, an AI notetaker for recruiting, uses this kind of context:

"Since moving to AssemblyAI, we've seen a meaningful improvement in the confidence tail of our production transcripts....What stands out is not just the model quality, but the way [they] let us bring real meeting context into transcription, from calendar titles to organizations, domains, and participant names, so recruiting conversations come through with the nuance our customers depend on."

— Shahriar Tajbakhsh, Co-founder and CTO, Metaview

Structural context: how the audio is formatted

Separate channels per participant beat any clustering algorithm. Dual-channel call recordings, per-participant meeting streams, and stereo interview rigs all qualify. If you have the channels, use them.

Speaker Identification: from generic labels to names and roles

Diarization stops at "Speaker A." Speaker Identification, a Speech Understanding task, maps those labels to names or roles. It needs no voice enrollment and costs +$0.02/hr.

data = {
    "audio_url": upload_url,
    "speech_models": ["universal-3-5-pro"],
    "speaker_labels": True,
    "speaker_options": {
        "min_speakers_expected": 2,
        "max_speakers_expected": 4,
        "include_speaker_confidence": True,
    },
    "speech_understanding": {
        "request": {
            "speaker_identification": {
                "speaker_type": "name",
                "speakers": [{"name": "Michel Martin"}, {"name": "Peter DeCarlo"}]
            }
        }
    }
}

Output:

Michel Martin: Good morning, and welcome to the show.
Peter DeCarlo: Thanks for having me.

Role-based identification

Set "speaker_type": "role" and list roles like Agent and Customer, or Clinician and Patient. This is the common choice for contact centers, where the names change on every call but the roles don't.

Name-based identification

Set "speaker_type": "name" and pass the participant list. You can add a description per speaker, such as what they usually discuss, and raise effort to "medium" for complex audio.

The limitation to be clear about: per-file, not cross-file

Speaker Identification works within one recording. AssemblyAI doesn't store voiceprints or offer cross-file speaker enrollment. If you need the same person recognized across every meeting by voice alone, you'll need a separate voice-biometrics layer. The trade-offs are in speaker recognition vs. speaker verification.

Accuracy in live streams

Real-time diarization is harder than pre-recorded, because the model can't hear the end of the conversation first. On Universal-3.6 Pro Realtime, enable it with speaker_labels: true and cap speakers with max_speakers. Four practices help:

• Use the revised labels for anything you store. At session end, a refinement pass sends one SpeakerRevision message for turns that changed, adding about 400 ms.
• Turn on mid-session revisions for long calls. Set speaker_labels_revision_interval_ms (300,000 ms is the recommended value).
• Expect short turns to wait. Turns under about one second are labeled PENDING instead of guessed.
• Give it a few sentences. The first one or two turns can be misassigned while the model builds speaker profiles.

The full walkthrough is in streaming speaker diarization.

How to measure whether it worked

Pick 20 to 50 recordings that look like production: same channels, same noise, same speaker counts. Hand-label who said what. Then score each configuration with cpWER, changing one setting at a time. Track the errors that matter to your product, such as swapped agent and customer turns, not just the average. Universal-3.5 Pro averages 30.17% cpWER across six public datasets, against 30.35% for Azure and 35.26% for ElevenLabs Scribe v2. Your own audio is the benchmark that counts.

Real-world use cases

• Contact centers: role-based identification for agent and customer analytics and QA.
• Meeting notetakers: participant lists from the calendar plus a speaker range. See AI notetakers.
• Healthcare: clinician vs. patient roles in ambient documentation.
• Podcasts and media: host and guest names from the episode notes.

Final words

Most diarization accuracy problems aren't model problems. They're missing-context problems. Tell the model how many people to expect, give it channels when you have them, and name speakers with what you already know. Then measure with cpWER and fix the errors your users actually see. Start with Universal-3.5 Pro: $0.21/hr, plus +$0.02/hr for diarization and +$0.02/hr for Speaker Identification.

Frequently asked questions

How do I improve speaker diarization accuracy?

Pass an expected speaker count or range, record speakers on separate channels when possible, and keep audio clean. Make sure each speaker talks for at least 30 seconds. Use a cpWER-optimized model like Universal-3.5 Pro, and test changes with cpWER on your own recordings.

How many speakers should I set?

Set speakers_expected only when you know the exact count. Otherwise pass speaker_options with a minimum and a maximum a little above what you expect, about the minimum plus two. A maximum that's too high can split one speaker into several labels.

Should I use multichannel or speaker diarization?

Use multichannel when each speaker is recorded on their own channel, because channel separation is exact. Use diarization when everyone shares one mixed channel. Many call recordings are dual-channel, so check your audio before choosing.

Can AssemblyAI label speakers by name automatically?

Yes. Speaker Identification maps diarized labels to names or roles using the conversation's content and an optional participant list. No voice enrollment is needed. It costs +$0.02/hr and works within each file, not across files.

How does context influence automatic speaker labeling?

Context narrows the model's choices. Spoken names and role-specific phrasing help resolve who is who. Metadata like participant lists and speaker counts constrain the clustering. Separate channels remove the guesswork entirely.

How is speaker diarization accuracy measured?

The two common metrics are DER, which scores time, and cpWER, which scores words attributed to the right speaker. cpWER tracks what readers of a transcript actually experience, so AssemblyAI optimizes and reports on it.