How to apply LLMs to multi-speaker audio recordings

Ask an LLM to summarize a panel discussion and it will happily give you an answer. The answer will also be wrong in a specific, hard-to-notice way: it will report what "the group" thinks, because a flat transcript never told it that four different people were talking.

That's the whole problem with pointing a language model at multi-speaker audio. Transcription gets you text. It doesn't get you attribution. And attribution is where most of the value in a conversation lives — who committed to what, who disagreed, who asked and who answered, how many people were even in the room.

The fix is a two-step pattern, and it's cheap: diarize the audio so every utterance carries a speaker label, then hand that speaker-attributed transcript to the LLM. This post walks through both steps with working code — the direct LLM Gateway path for one-shot questions, and a retrieval pipeline in Haystack for when you're querying hours of recordings.

What a flat transcript actually costs you

The gap is easiest to see with three questions you'd genuinely want to ask a recording.

"What are each speaker's opinions on building in-house versus using third parties?" Run that against a plain transcript of a panel discussion and you get a single blended paragraph — the group's average opinion, which nobody actually holds. Run it against a diarized transcript and you get each person's position separately, including the fact that one of the "speakers" was the moderator asking the question rather than answering it.

"What are the two opposing opinions, and how many people are on each side?" A flat transcript of a debate contains both arguments, so the model can usually name them. It cannot count heads. Speaker labels turn a vague "there are differing views" into a tally.

"How many speakers and moderators are in this call?" This one is the clearest failure. Without labels, the best an LLM can do is infer that more than one person is present. It has no way to arrive at the actual number. With speaker diarization, the count is just a property of the data.

None of these are exotic queries. They're the default questions people ask about meetings, earnings calls, interviews, and support calls — which is why conversation intelligence products treat diarization as table stakes rather than an add-on.

Step 1: diarize the recording

Set speaker_labels to true and the transcript comes back as an utterances array instead of one block of text. Each utterance is an uninterrupted stretch of speech from a single person, with a label, timings, and word-level detail.

from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# You can use a local filepath:
# audio_file = "./example.mp3"

# Or use a publicly-accessible URL:
audio_file = (
    "https://assembly.ai/wildfires.mp3"
)

config = TranscriptionConfig(
  language_detection=True,
  speaker_labels=True,
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>")
transcript = transcriber.transcribe(audio_file, config)

for utterance in transcript.utterances:
  print(f"Speaker {utterance.speaker}: {utterance.text}")

That's the entire diarization step. The output looks like this:

Speaker A: Good morning, and welcome to the show.
Speaker B: Thanks for having me.

A few parameters are worth knowing before you point this at production audio, because the defaults are deliberately permissive:

These are boundaries, not hints. max_speakers_expected is a strict cap, so if eight people speak and you set it to four, four of them get folded into someone else's label. The docs are blunt about the other direction too: setting the maximum too high causes over-splitting, where one person's sentences get scattered across several labels.

So the practical rule is to use the range, not the exact count. Set min_speakers_expected to the number you're confident about and max_speakers_expected a couple higher. Reach for speakers_expected only when you truly know — an incorrect exact count actively hurts accuracy.

Three more things that move diarization quality more than any parameter:

• Speech per speaker. Accuracy improves as the model accumulates embedding context, so aim for at least 30 seconds of continuous speech per person. Someone who only ever says "Right" or "Sounds good" is hard to cluster separately.
• Cross-talk. Overlapping speech reduces accuracy. Universal-3.5 Pro jointly produces the transcript and the speaker changes in one pass, which is why it handles short turns and rapid back-and-forth better than a bolt-on diarizer, but physics still applies.
• Similar voices. If two speakers sound alike, expect the model to struggle. Nothing in the config fixes that.

If your audio is already split per-participant — separate tracks from a conferencing SDK, for instance — you may not need diarization at all. Our breakdown of multichannel versus speaker diarization covers when each one is the right tool.

Optional: swap Speaker A for a real name or role

Diarization gives you Speaker A, Speaker B, Speaker C. Useful, but not what you'd show a user. Speaker Identification replaces those generic labels with actual names or roles, inferred from what's said in the conversation — no voice enrollment, no reference recordings. It requires speaker_labels: true, so it layers on top of the step above rather than replacing it.

Pass names when you know who's in the room:

data = {
    "audio_url": upload_url,
    "language_detection": True,
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "speaker_identification": {
                "speaker_type": "name",
                "speakers": [{"name": "Michel Martin"}, {"name": "Peter DeCarlo"}] 
# Change these values to match the names of the speakers in your file
            }
        }
    }
}

The transcript comes back as Michel Martin: and Peter DeCarlo: instead of Speaker A: and Speaker B:. Names can only be associated when they're actually present somewhere in the transcript — the model can't synthesize a name from nothing.

When you know the roles but not the names, switch speaker_type to "role" and pass role labels instead. The docs list the combinations that come up most: Agent/Customer for support calls, Interviewer/Interviewee for interviews, Host/Guest for podcasts, and Moderator/Panelist for panel discussions — which would have answered that "how many speakers and moderators" question directly.

There's also an effort parameter with two modes, low (the default) and medium. Low effort is fine for clean, clearly segmented audio: short transcripts, podcasts, support calls where the first line gives the role away. Reach for medium on conference-room recordings, arguments where people talk over each other, and transcripts where names or roles are only weakly signposted. Medium costs more, so it's a deliberate trade rather than a default.

This capability didn't exist when the video above was recorded. If you're rebuilding this pattern in 2026, it's the first thing to add.

Step 2, the fast path: send the diarized transcript to an LLM

For a single question about a single recording, you don't need a vector store. You need one request. LLM Gateway is an OpenAI-compatible endpoint that fronts 25+ models from Anthropic, OpenAI, Google, and others, and it can pull a transcript in by ID so you never have to hold the text yourself:

curl -X POST "https://llm-gateway.assemblyai.com/v1/chat/completions" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-2.5-flash-lite",
    "messages": [
      {"role": "user", "content": "hi there"},
      {"role": "assistant", "content": "Hi! How can I help?"},
      {"role": "user", "content": "Here is a transcript: {{ transcript }}. Return the text verbatim."}
    ],
    "transcript_id": "065a71ac-dc3e-4e38-9374-e54c0bea564f"
  }'

The API replaces the first {{ transcript }} tag with the transcript's text and runs the completion. Two gotchas: the tag must be exactly {{ transcript }} with the spaces ({{transcript}} is left alone), and only the first occurrence in the first message containing it gets substituted.

One thing to be explicit about in your prompt: tell the model what it's looking at. A transcript where every line starts with Speaker B: is a format the model needs to be told about, not one it should have to guess at. The prompt used in the Haystack cookbook below does exactly this, and it's the single highest-leverage line in the whole pipeline.

If summarization is what you're after specifically, we have a full walkthrough on summarizing meeting transcripts with LLMs in Python, and a broader guide to building voice AI apps with LLM Gateway.

Step 2, the retrieval path: a Haystack RAG pipeline

The single-request approach stops working when the recording is long or you're querying across many of them. A 40-minute panel plus an hour-long earnings call is a lot of tokens to resend on every question, and stuffing everything into context isn't the same as retrieving the relevant parts.

That's where retrieval comes in. Haystack is an open-source Python framework for building LLM applications, and the AssemblyAI integration ships an AssemblyAITranscriber component that drops straight into a Haystack pipeline. The code below is from the deepset cookbook notebook — it's the current version, which has moved on from what's shown in the video.

Install the dependencies:

%%bash
pip install haystack-ai
pip install assemblyai-haystack
pip install huggingface-api-haystack
pip install sentence-transformers-haystack
pip install --upgrade gdown

Then build the indexing pipeline. It transcribes with speaker labels, splits the result into chunks, embeds them, and writes them to a document store:

from haystack.components.writers import DocumentWriter
from haystack.components.preprocessors import DocumentSplitter
from haystack_integrations.components.embedders.sentence_transformers import SentenceTransformersDocumentEmbedder
from haystack import Pipeline
from haystack.document_stores.in_memory import InMemoryDocumentStore
from assemblyai_haystack.transcriber import AssemblyAITranscriber
from haystack.document_stores.types import DuplicatePolicy
from haystack.utils import ComponentDevice

speaker_document_store = InMemoryDocumentStore()
transcriber = AssemblyAITranscriber(api_key=ASSEMBLYAI_API_KEY)
speaker_splitter = DocumentSplitter(
    split_by = "sentence",
    split_length = 10,
    split_overlap = 1
)
speaker_embedder = SentenceTransformersDocumentEmbedder(device=ComponentDevice.from_str("cuda:0"))
speaker_writer = DocumentWriter(speaker_document_store, policy=DuplicatePolicy.SKIP)

indexing_pipeline = Pipeline()
indexing_pipeline.add_component(instance=transcriber, name="transcriber")
indexing_pipeline.add_component(instance=speaker_splitter, name="speaker_splitter")
indexing_pipeline.add_component(instance=speaker_embedder, name="speaker_embedder")
indexing_pipeline.add_component(instance=speaker_writer, name="speaker_writer")

indexing_pipeline.connect("transcriber.speaker_labels", "speaker_splitter")
indexing_pipeline.connect("speaker_splitter", "speaker_embedder")
indexing_pipeline.connect("speaker_embedder", "speaker_writer")

The connection that matters is the first one. AssemblyAITranscriber exposes three outputs — transcription, summarization, and speaker_labels — and this pipeline wires transcriber.speaker_labels into the splitter rather than transcriber.transcription. That single choice is the difference between the speaker-aware application and the plain one. The speaker letter rides along in each document's meta dictionary.

Run it against a file:

audio_file_path = "/content/Panel_Discussion.mp3" #@param ["/content/Netflix_Q4_2023_Earnings_Interview.mp3", "/content/Working_From_Home_Debate.mp3", "/content/Panel_Discussion.mp3"]
indexing_pipeline.run(
    {
        "transcriber": {
            "file_path": audio_file_path,
            "summarization": None,
            "speaker_labels": True
        },
    }
)

Note that indexing_pipeline.run() blocks until transcription finishes. speaker_labels is what we want; summarization is left off, and we'll come back to why you should leave it off permanently.

Now the RAG pipeline. An embedder for the query, a retriever, a prompt builder, and a generator:

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.huggingface_api import HuggingFaceAPIChatGenerator
from haystack_integrations.components.embedders.sentence_transformers import SentenceTransformersTextEmbedder
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
from haystack.utils import ComponentDevice

prompt_template = """
You will be provided with a transcription of a recording with each sentence or group of sentences attributed to a Speaker by the word "Speaker" followed by a letter representing the person uttering that sentence. Answer the given question based on the given context.
If you think that given transcription is not enough to answer the question, say so.

Transcription:
{% for doc in documents %}
  {% if doc.meta["speaker"] %} Speaker {{doc.meta["speaker"]}}: {% endif %}{{doc.content}}
{% endfor %}
Question: {{ question }}
Answer:
"""

retriever = InMemoryEmbeddingRetriever(speaker_document_store)
text_embedder = SentenceTransformersTextEmbedder(device=ComponentDevice.from_str("cuda:0"))
answer_generator = HuggingFaceAPIChatGenerator(
    api_type="serverless_inference_api",
    api_params={"model": "Qwen/Qwen2.5-7B-Instruct"},
    generation_kwargs={"max_tokens": 500})
prompt_builder = ChatPromptBuilder(
    template=[ChatMessage.from_user(prompt_template)],
    required_variables="*")

speaker_rag_pipe = Pipeline()
speaker_rag_pipe.add_component("text_embedder", text_embedder)
speaker_rag_pipe.add_component("retriever", retriever)
speaker_rag_pipe.add_component("prompt_builder", prompt_builder)
speaker_rag_pipe.add_component("llm", answer_generator)

speaker_rag_pipe.connect("text_embedder.embedding", "retriever.query_embedding")
speaker_rag_pipe.connect("retriever.documents", "prompt_builder.documents")
speaker_rag_pipe.connect("prompt_builder.prompt", "llm.messages")

Read the prompt template again, because it's doing the real work. It explains the transcript format in plain language, then loops over the retrieved documents and prefixes each one with Speaker {{doc.meta["speaker"]}}. Without that prefix the speaker metadata stays in the document store and never reaches the model — you'd have paid for diarization and thrown the result away.

When you wire components together, match the input and output names deliberately. The prompt builder has two inputs, documents and question, so the retriever's documents output has to connect to the prompt builder's documents input. Mismatched names are the most common way these pipelines break.

Finally, ask it something:

question = "What are each speakers' opinions on building in-house or using third parties?" # @param ["What are the two opposing opinions and how many people are on each side?", "What are each speakers' opinions on building in-house or using third parties?", "How many people are speaking in this recording?" ,"How many speakers and moderators are in this call?"]

result = speaker_rag_pipe.run({
    "prompt_builder":{"question": question},
    "text_embedder":{"text": question},
    "retriever":{"top_k": 10}
})
result["llm"]["replies"][0].text

On the panel discussion, the answer comes back broken out per person: Speaker A is interested in understanding how companies decide between building in-house or using third parties — that's the moderator, identifiable purely from the shape of their contributions. Speaker B believes the decision depends on whether the task is part of the company's core IP. Speaker C gets their own position too.

Same recording, same model, same question. The only difference from the flat-transcript version is that the speaker labels made it into the prompt.

What's changed since this video was recorded

The video is from March 2024, and both the AssemblyAI API and the Haystack side have moved. If you're following along, here's what's different in 2026.

Two consequences worth spelling out. First, the summarization option on AssemblyAITranscriber sits on top of a deprecated transcript parameter, so treat that output as legacy and route summaries through LLM Gateway instead. Second, the community-maintained Haystack component doesn't currently expose speech_models, speakers_expected, speaker_options, or speech_understanding — so if you need speaker count constraints, a pinned model, or Speaker Identification, call the transcription API directly and feed the utterances into Haystack yourself.

Our original LeMUR announcement is still up for historical context, and deepset published their own walkthrough of this cookbook if you want the Haystack team's framing of the same pattern.

Why teams building on this care about the transcript underneath

Everything above assumes the labels are right. When they aren't, the failure is invisible — the LLM confidently attributes a sentence to the wrong person and you have no signal that it happened. That's why teams shipping conversation products treat transcript quality as a product problem rather than a vendor checkbox.

Since moving to AssemblyAI, we've seen a meaningful improvement in the confidence tail of our production transcripts....What stands out is not just the model quality, but the way [they] let us bring real meeting context into transcription, from calendar titles to organizations, domains, and participant names, so recruiting conversations come through with the nuance our customers depend on.

— Shahriar Tajbakhsh, Co-founder and CTO, Metaview

Metaview runs AI interview notes for hiring teams, which is about as unforgiving a diarization problem as exists: panel interviews, remote audio, people interrupting each other, and a customer who will notice immediately if a candidate's answer gets attributed to the interviewer.

Where to take this next

The pattern generalizes further than panel discussions. Pipe the same diarized transcript into Speech Understanding and you get per-speaker sentiment and entity detection rather than document-level averages. Do it live instead of after the fact with streaming speaker diarization, which labels speakers as the audio arrives and sends a single correction once the stream ends.

The genuinely useful takeaway from this build isn't the Haystack pipeline. It's that "who said it" is a feature you can add with one boolean, and that almost every question worth asking about a conversation depends on it. The reason so many audio-plus-LLM projects produce mush isn't the model choice. It's that nobody told the model there was more than one person in the room.

Frequently asked questions

How can I transcribe audio recordings with multiple speakers?

Transcribe the file with speaker diarization enabled by setting speaker_labels to true in the transcription request. Instead of one block of text, the response returns an utterances array where each entry carries a speaker label, the transcribed text, start and end times in milliseconds, a confidence score, and a word-level breakdown. From there you can print, store, or prompt against a transcript that knows who said what.

How do you apply an LLM to a multi-speaker recording?

Diarize first, then prompt. Transcribe the audio with speaker_labels: true, format the utterances so each line is prefixed with its speaker, and tell the model in the prompt that the transcript is attributed by speaker. For one-off questions, send it in a single LLM Gateway request; for long or numerous recordings, chunk and embed the utterances and retrieve the relevant ones per question.

What's the difference between speaker diarization and Speaker Identification?

Speaker diarization separates the audio by speaker and assigns generic sequential labels — Speaker A, Speaker B, Speaker C. Speaker Identification is a layer on top that replaces those labels with real names or roles, inferred from the content of the conversation rather than from voice enrollment. It requires speaker_labels: true, and names can only be applied when they appear somewhere in the transcript.

How many speakers can speaker diarization detect?

The default cap depends on audio length: 10 speaker labels for 2–10 minute files and 30 for files over 10 minutes. You can constrain it with speakers_expected for an exact count, or speaker_options.min_speakers_expected and speaker_options.max_speakers_expected for a range. These are hard boundaries rather than hints, so a too-low maximum merges speakers together and a too-high one splits one person across several labels.

Do I need a RAG pipeline, or can I just send the whole transcript to the LLM?

For a single recording and a handful of questions, send the whole transcript — LLM Gateway even accepts a transcript_id and injects the text for you, so you don't have to store it. Retrieval earns its complexity when you're querying hours of audio or many files at once, where resending everything on every question is wasteful and precision drops. Start with the direct call and add retrieval when the token math stops making sense.

Can Whisper do speaker diarization?

Whisper transcribes but does not diarize, so speaker labels have to come from a separate model such as pyannote, stitched onto the Whisper output. That two-model approach means transcript boundaries and speaker boundaries are decided independently, which tends to show up as errors on short turns and overlapping speech. Universal-3.5 Pro produces the transcript and the speaker changes in a single pass, which is why it's optimized for cpWER rather than DER.

‍