Skip to main content

Overview

Speaker diarization identifies individual speakers in your audio and labels each segment of the transcript with the speaker. When enabled, the transcript is returned as a list of utterances, where each utterance represents an uninterrupted segment of speech from a single speaker. Accuracy improves the more each speaker talks as the model accumulates embedding context. For best results, each speaker should have at least 30 seconds of continuous speech.

Quickstart

To enable Speaker Diarization, set speaker_labels to True in the POST request body:
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "speaker_labels": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

for utterance in transcription_result['utterances']:
  print(f"Speaker {utterance['speaker']}: {utterance['text']}")

Configuration

You can constrain the number of speakers with speakers_expected, or with min_speakers_expected/max_speakers_expected. These are hard boundaries on the number of speaker labels, not hints: max_speakers_expected is a strict cap — if more people speak than that, the additional speakers are merged into existing labels — and min_speakers_expected is a strict floor.
  • If you know the exact number of speakers, set speakers_expected.
  • If you have a rough idea, use min_speakers_expected and max_speakers_expected to set a range. Set max_speakers_expected a little higher than the number of speakers you expect so the model has room to identify any additional speakers. Setting it too high can cause the model to over-split and return more speaker labels than are actually present.
KeyTypeDefaultDescription
speaker_labelsbooleanfalseEnable Speaker Diarization.
speakers_expectednumberSet the exact number of speakers.
speaker_optionsobjectSet range of possible speakers.
speaker_options.min_speakers_expectednumberA hard lower limit on the number of speaker labels. The model won’t return fewer speakers than this.
speaker_options.max_speakers_expectednumber0–2 minutes: —
2–10 minutes: 10 speakers
10+ minutes: 30 speakers
A hard upper limit on the number of speaker labels. If more people speak than this, the additional speakers are merged into existing labels. Give the model a little headroom above the number of speakers you expect; setting it too high can cause over-splitting and return more speakers than are actually present.
Only set speakers_expected when you are certain of the exact speaker count. If you’re unsure, use min_speakers_expected and max_speakers_expected to describe a range instead — providing an incorrect exact count can negatively affect diarization accuracy.

Reading the response

When diarization is enabled, the transcript includes an utterances array in place of a single text block. Each object in the array represents one uninterrupted segment of speech from a single speaker.
{
  "utterances": [
    {
      "speaker": "A",
      "text": "Smoke from hundreds of wildfires in Canada is triggering air quality alerts.",
      "start": 250,
      "end": 28840,
      "confidence": 0.97,
      "words": [
        {
          "text": "Smoke",
          "speaker": "A",
          "start": 250,
          "end": 650,
          "confidence": 0.99
        }
      ]
    }
  ]
}
The utterances array contains objects with the following fields:
FieldTypeDescription
speakerstringThe speaker label, assigned as sequential letters such as A, B, and C.
textstringThe transcribed text for the utterance.
startnumberThe start time of the utterance in milliseconds.
endnumberThe end time of the utterance in milliseconds.
confidencenumberA score between 0 and 1 indicating the model’s confidence in the transcribed text.
wordsarrayA word-level breakdown of the utterance.
Each object in the words array contains the following fields:
FieldTypeDescription
textstringThe transcribed word.
speakerstringThe speaker label for the word.
startnumberThe start time of the word in milliseconds.
endnumberThe end time of the word in milliseconds.
confidencenumberA score between 0 and 1 indicating the model’s confidence in the word.

Identify speakers by name

Speaker Diarization assigns generic labels like “Speaker A” and “Speaker B” to each speaker. If you want to replace these labels with actual names or roles, you can use Speaker Identification to transform your transcript. Before Speaker Identification:
Speaker A: Good morning, and welcome to the show.
Speaker B: Thanks for having me.
After Speaker Identification:
Michel Martin: Good morning, and welcome to the show.
Peter DeCarlo: Thanks for having me.
The following example shows how to transcribe audio with Speaker Diarization and then apply Speaker Identification to replace the generic speaker labels with actual names.
maxLines=30
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

audio_url = "https://assembly.ai/wildfires.mp3"

# Configure transcript with speaker diarization and speaker identification
data = {
    "audio_url": audio_url,
    "language_detection": True,
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "speaker_identification": {
                "speaker_type": "name",
                "known_values": ["Michel Martin", "Peter DeCarlo"]
            }
        }
    }
}

# Submit the transcription request
response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

# Poll for transcription results
while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()

    if transcript["status"] == "completed":
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)

# Print utterances with identified speaker names
for utterance in transcript["utterances"]:
    print(f"{utterance['speaker']}: {utterance['text']}")
For more details on Speaker Identification, including how to identify speakers by role and how to apply it to existing transcripts, see the Speaker Identification guide.

Best practices for accurate diarization

Follow these tips to get the best results from Speaker Diarization:
  • Ensure sufficient speech per speaker. Each speaker should speak for at least 30 seconds uninterrupted. The model may struggle to create separate clusters for speakers who only contribute short phrases like “Yeah”, “Right”, or “Sounds good”.
  • Minimize cross-talk. Overlapping speech between speakers can reduce diarization accuracy. Where possible, ensure speakers take turns.
  • Reduce background noise. Background noise, echoes, or playback of recorded audio during a conversation can interfere with speaker separation.
  • Use speaker_options instead of speakers_expected when uncertain. Only use speakers_expected when you are confident about the exact number of speakers. If this number is incorrect, the model may produce random splits of single-speaker segments or merge multiple speakers into one. It’s generally recommended to use min_speakers_expected and set max_speakers_expected slightly higher (e.g., min_speakers_expected + 2) to allow flexibility.
  • Avoid setting max_speakers_expected too high. Setting the maximum too high may reduce accuracy, causing sentences from the same speaker to be split across multiple speaker labels.
  • Be aware of speaker similarity. If speakers sound similar, the model may have difficulty distinguishing between them.