Multilingual transcription: How to detect, diarize, and transcribe audio across languages

In short: AssemblyAI transcribes 99+ languages from one API. Universal-3.5 Pro code-switches natively across 18 languages in pre-recorded audio, and Universal-3.6 Pro Realtime code-switches across 32 languages in real time. Both detect the spoken language automatically, label speakers, and accept prompts and key terms. Languages outside the native sets fall back to Universal-2.

Most speech-to-text APIs were built for one language at a time. Real audio doesn't work that way. A support call switches between English and Spanish mid-sentence. A conference panel has speakers from six countries. A doctor dictates in German and drops in English drug names. If your pipeline can't handle that, you're either picking the wrong model per file or losing accuracy exactly where it matters most.

This guide covers what multilingual transcription means, which languages each model supports, and how to detect, diarize, and transcribe mixed-language audio in code.

What is multilingual transcription?

Multilingual transcription is automatic speech recognition that works across more than one language, including when several languages appear in the same recording. Three separate capabilities get bundled under that one term:

• Multi-language coverage. The model transcribes audio in any language it supports.
• Automatic language detection. You don't have to tell the API what language the audio is in.
• Code-switching. A speaker moves between languages within the same utterance, and the model keeps up without dropping words.

Coverage is table stakes now. Detection is common. Code-switching is where models separate, and it's the capability teams underestimate until their real-world accuracy comes back lower than the demo.

Supported languages: the full picture

Here's how coverage breaks down across AssemblyAI's models as of October 2026.

Automatic language detection identifies 130+ languages and works best with at least 15 seconds of speech. Translation covers 100+ target languages for pre-recorded audio.

Universal-3.5 Pro (pre-recorded): 18 languages with native code-switching

English, Spanish, French, German, Italian, Portuguese, Arabic, Danish, Dutch, Finnish, Hebrew, Hindi, Japanese, Mandarin, Norwegian, Swedish, Turkish, and Vietnamese. That's three times the six languages of the prior generation, at the same $0.21/hr.

Universal-3.6 Pro Realtime (streaming): 32 languages with native code-switching

Universal-3.6 Pro Realtime launched September 29, 2026, and added Afrikaans, Cantonese, Estonian, Galician, Korean, Marathi, Norwegian Nynorsk, Persian, Romanian, Russian, Urdu, Xhosa, and Zulu. The full list lives in the streaming multilingual docs.

Universal-2: 99+ languages

For anything outside the native code-switching sets, pre-recorded requests fall back to Universal-2, which covers 99+ languages including Polish, Ukrainian, Greek, Croatian, Czech, Indonesian, Tamil, and Thai. We're also moving some Universal-2 traffic onto Universal-3.5 Pro: since October 2026, Chinese and Japanese Universal-2 requests run on Universal-3.5 Pro at Universal-2 rates, with Hebrew and Swedish rolling out.

How to transcribe multilingual pre-recorded audio

Universal-3.5 Pro is the default for pre-recorded audio, with Universal-2 as the automatic fallback. Turn on language detection and the model follows speakers as they switch languages, transcribing every word in the language it was spoken.

from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

config = TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],  # ordered fallback list
    language_detection=True,
    speaker_labels=True,
)

transcript = Transcriber(api_key="YOUR_API_KEY", config=config).transcribe(
    "https://assembly.ai/code-switching-3"
)

if transcript.status == "error":
    raise RuntimeError(f"Transcription failed: {transcript.error}")

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

Two things to note. The parameter is speech_models (plural), because it's an ordered fallback list. And if you call the REST API directly, pass your raw API key in the authorization header with no Bearer prefix.

Choose your languages in code

When you already know which languages a file contains, pass language_codes to constrain the model to that set. Since September 2026, Universal-3.5 Pro accepts any number of languages here, and English isn't required.

config = TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_codes=["es", "fr"],  # Spanish and French only
)

• A one-element list such as ["es"] pins the file to that language.
• language_codes can't be combined with language_code or language_detection. Set one of the three.
• If the detected language falls outside Universal-3.5 Pro's 18, the job moves to the next model in speech_models.

See the code-switching docs for the full behavior, and this walkthrough of Universal-3.5 Pro code-switching on real audio.

How to transcribe multilingual audio in real time

Universal-3.6 Pro Realtime is multilingual by default. Open a streaming session with no language parameters and it code-switches across all 32 languages. Add language_detection and each turn reports which language was spoken, with a confidence score.

from assemblyai.streaming.v3 import (
    RealTimeEvents, RealTimeParameters, RealTimeTranscriber, RealTimeTranscriberOptions,
)

def on_turn(client, event):
    if event.end_of_turn:
        print(f"[{event.language_code}] {event.transcript}")

client = RealTimeTranscriber(RealTimeTranscriberOptions(), api_key="YOUR_API_KEY")
client.on(RealTimeEvents.Turn, on_turn)

client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-6-pro",
        language_detection=True,
        # language_codes=["en", "es"],  # optional: steer toward known languages
    )
)
# stream 16kHz mono PCM with client.stream(chunk), then:
# client.disconnect(terminate=True)

You can change the language set mid-stream without reconnecting. That's useful when a voice agent detects the caller's language or hands off to a speaker of another language:

client.update_configuration(language_codes=["en", "fr"])  # steer
client.update_configuration(language_codes=[])            # back to full code-switching

On AssemblyAI's code-switching set of five language pairs, Universal-3.6 Pro Realtime posts 7.20% WER, versus 8.65% for Universal-3.5 Pro Realtime and 17.13% for Deepgram Nova-3 Multilingual (September 29, 2026). The research write-up has the methodology.

Code-switching accuracy: pre-recorded benchmark

We measured normalized WER on code-switched pre-recorded audio across five language pairs. Lower is better.

That's a 15% relative reduction over Universal-3 Pro. GPT-4o Transcribe falls apart on code-switched speech, which is a useful reminder that general-purpose transcription and purpose-built multilingual ASR aren't the same product. Full methodology is on our benchmarks page.

Comparing providers for your own audio? Our companion piece, multilingual speech-to-text in 2026, covers how to run that evaluation, and how to evaluate speech recognition models covers the method.

Diarization across languages

Multilingual audio is usually multi-speaker audio. Turn on speaker_labels and the model separates who said what. Universal-3.5 Pro ships the most accurate diarization we've released, optimized for concatenated minimum-permutation WER (cpWER). It averages 30.17 cpWER, versus 37.92 for Deepgram Nova-3 and 35.26 for ElevenLabs Scribe v2. That accuracy holds when speakers switch languages, which is exactly when naive diarization breaks. For live audio, see streaming speaker diarization.

Sharpening accuracy with prompting and key terms

Two tools push multilingual accuracy further, and they work together:

• prompt is a short natural-language description of the audio, such as "Bilingual support call in English and Spanish about a billing dispute." It sets context so the model makes better guesses on ambiguous audio. You can name the expected languages here, though language_codes is the better-supported way to select them.
• keyterms_prompt is a list of up to 1,000 words or phrases you want spelled correctly: product names, people, jargon.

Use prompt to frame the conversation and keyterms_prompt to lock down spellings. One caution: key terms don't help equally in every language, so test them per language. We covered the failure modes in when keyterm prompting backfires.

Multilingual audio with PHI: Medical Mode and BAA

Healthcare audio is often multilingual, and it comes with protected health information (PHI).

Medical Mode. Add domain: "medical-v1" (+$0.15/hr) and the model tunes for clinical vocabulary: drug names, procedures, and conditions. It produces about 20% fewer missed medical entities than the base model. It's available in English, Spanish, German, and French, for pre-recorded audio on Universal-3.5 Pro and for streaming on Universal-3.6 Pro Realtime.

Compliance. AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI. You can also redact PHI across audio and transcripts. See the medical solution page for details.

Who's using it

Teams with multilingual users run on this stack every day. Fireflies evaluated real-time models for its voice agent pipeline:

"We were searching for the best realtime ASR model for our voice agent pipeline in Fireflies. The new Universal 3.5 Pro speech model from Assembly is best so far in terms of accuracy, latency and language switching."

— Foysal Osmany, Software Engineer at Fireflies

Granola, the AI notepad for meetings, also runs on AssemblyAI. For teams like these, multilingual audio isn't an edge case. It's the default condition.

Where multilingual transcription goes next

Language count used to be the headline. It isn't anymore. The question that separates models now is what happens at the switch: the half-second where a speaker leaves one language and enters another. Native code-switching across 32 real-time languages means you no longer route audio to a different model per caller. You stop designing around language and start designing around the conversation.

Multilingual transcription is available today through the Speech-to-Text API for pre-recorded audio and Streaming Speech-to-Text for real time. Pricing is on the pricing page.

Frequently asked questions

What is multilingual transcription and how does it work?

Multilingual transcription converts speech to text across more than one language, including recordings where speakers switch languages. A model detects the spoken language, transcribes each word in the language it was spoken, and keeps going when the language changes. AssemblyAI does this natively in 18 languages for pre-recorded audio and 32 in real time.

Which languages does AssemblyAI support in real time?

Universal-3.6 Pro Realtime supports 32 languages with native code-switching, including English, Spanish, French, German, Hindi, Mandarin, Cantonese, Japanese, Korean, Arabic, Russian, and Persian. Universal-Streaming Multilingual covers six languages with per-turn switching.

Can AssemblyAI handle code-switching?

Yes. Universal-3.5 Pro code-switches natively across 18 languages in pre-recorded audio and posts 7.69 normalized WER on AssemblyAI's code-switching benchmark. Universal-3.6 Pro Realtime code-switches across 32 languages in streaming at 7.20% WER. No extra configuration is needed.

How do I restrict transcription to specific languages?

Pass language_codes with the languages you expect, for example ["en", "es"]. On Universal-3.5 Pro and Universal-3.6 Pro Realtime it accepts any number of languages and English isn't required. A single-element list pins one language. In streaming you can update the list mid-session.

How do I transcribe multilingual conference calls?

Use Universal-3.5 Pro for recordings or Universal-3.6 Pro Realtime for live calls, with speaker labels on. Both code-switch natively, so you don't split the audio by language. If you know the languages in advance, pass language_codes to improve accuracy.

Can I transcribe multilingual medical audio?

Yes. Add domain: "medical-v1" for clinical vocabulary in English, Spanish, German, and French, on pre-recorded or streaming audio. AssemblyAI offers a BAA for customers processing PHI and can redact PHI in audio and transcripts.