Multilingual transcription: How to detect, diarize, and transcribe audio across languages

Most transcription pipelines quietly assume everyone in the audio speaks one language and never switches. Real audio doesn't work that way. A support call bounces between English and Spanish mid-sentence. A board meeting has speakers in three languages. A podcast guest drops a French phrase and keeps going.

If your pipeline can't detect the language automatically, separate speakers across languages, and handle a sentence that flips from one language to another, you're going to ship transcripts full of gibberish—and you won't know until a customer complains.

So this is the practical guide to getting multilingual transcription right: automatic language detection, speaker diarization across languages, code-switching, the accuracy picture across models, and the API params to make it all work. Developer to developer.

What multilingual transcription actually involves

Multilingual transcription is the ability to convert speech to text across more than one language—ideally without you telling the system which language to expect.

That breaks down into a few distinct problems, and it's worth keeping them separate because different audio hits different ones:

Language detection. The system identifies the spoken language on its own, so you don't hardcode an assumption. You flip language_detection=true and the model figures it out from the audio.

‍Per-language accuracy. Recognizing Japanese well is a different engineering problem than recognizing Portuguese well. A model that's great at English can be mediocre at German. The languages a model was trained and tuned on matter enormously, and a headline "supports 100+ languages" number tells you nothing about how well any individual one performs. The honest way to read coverage is two numbers: how many languages a model handles at all, and which subset it handles at top accuracy.

Speaker diarization across languages. Telling who said what gets harder when speakers are using different languages in the same recording. Good speaker diarization labels speakers consistently regardless of which language each one is speaking.

Code-switching. This is the hard one—a single speaker mixing languages within one utterance. "Necesito the report by Friday, ¿vale?" A model that locks to one language per file will butcher that line.

Get all four right and you have transcripts that hold up on messy, real-world, multilingual audio. Miss one and the failure shows up exactly where it's most embarrassing.

Automatic language detection

You shouldn't have to know the language of a file before you transcribe it. With AssemblyAI's async speech-to-text API, you don't—set language_detection=true on your POST to https://api.assemblyai.com/v2/transcript and the model identifies the spoken language for you.

curl https://api.assemblyai.com/v2/transcript \
  --header "authorization: $ASSEMBLYAI_API_KEY" \
  --header "content-type: application/json" \
  --data '{
    "audio_url": "https://example.com/multilingual-call.mp3",
    "language_detection": true,
    "speech_models": ["universal-3-pro", "universal-2"],
    "speaker_labels": true
  }'

That's the core of it. The system reads the audio, picks the language, and transcribes—no per-file configuration, no guessing.

This matters most at scale. If you're processing thousands of files a day from users around the world, you can't manually tag each one's language. Auto-detection lets one pipeline handle everything that comes in.

The model picture: 99 languages at flagship accuracy

Here's the detail that used to determine which model did the work—and it just got a lot simpler.

The old tradeoff was coverage versus accuracy. You picked AssemblyAI's highest-accuracy model for a short list of core languages, then fell back to a broader, lower-tier model for everything else. That's no longer the choice you have to make. AssemblyAI's flagship async model, Universal-3 Pro, now supports 99 languages—so the model with the best entity accuracy, the lowest hallucination rate, and native code-switching is also the one with the widest language coverage. Whether the audio is English, Spanish, Japanese, Hindi, Arabic, or Polish, the same model handles it. It's priced at $0.21/hr currently.

So for multilingual async transcription, the recommendation collapses to one line:

"speech_models": ["universal-3-pro"]

No routing logic, no fallback chain, no per-language model selection. One request, one model, 99 languages.

For real-time multilingual audio, the model is Universal-3.5 Pro Realtime, AssemblyAI's flagship streaming model. It runs in 18 languages at flagship accuracy with mid-sentence code-switching (Hinglish included), priced at $0.45/hr base. More on that in the voice-agents section below.

For pre-recorded multilingual work, Universal-3 Pro is the default. For anything live, Universal-3.5 Pro Realtime.

Speaker diarization across languages

Knowing what was said is half the job. Knowing who said it is the other half, and in multilingual audio it's where a lot of systems quietly fall apart.

Set speaker_labels: true and the API segments the transcript by speaker, attaching a label to each utterance. The thing that matters for multilingual audio: the diarization stays consistent even when speakers are using different languages. Speaker A talking in English and Speaker B answering in German both get tracked correctly, so a bilingual meeting transcript reads cleanly instead of collapsing every voice into one undifferentiated block.

That's what makes diarized multilingual transcripts useful for downstream work—per-speaker summaries, sentiment by participant, compliance review—instead of a wall of text you can't attribute.

Code-switching: the part most APIs get wrong

Code-switching is when a speaker mixes languages inside a single sentence or utterance, and it's genuinely common—bilingual households, international teams, customer support across regions.

A lot of speech-to-text systems handle this badly because they commit to one language per file or per segment. The moment the speaker switches, the model keeps transcribing as if they hadn't, and you get phonetic nonsense.

AssemblyAI's models support native code-switching, so a sentence that moves from English to Spanish and back gets transcribed correctly across the switch rather than mangled at the boundary. That holds for pre-recorded audio on Universal-3 Pro and, in real time, on Universal-3.5 Pro Realtime—which handles the switch mid-sentence without pausing to catch up. If your audience is genuinely bilingual—and a lot of real audiences are—this is the difference between transcripts you can use and transcripts you have to throw out.

Where multilingual transcription earns its keep

A few use cases where this stops being a nice-to-have and becomes the whole point.

International meetings

Distributed teams and global orgs run meetings in mixed languages constantly. Auto language detection plus cross-language diarization turns a chaotic multilingual call into a clean, attributed transcript you can summarize, search, and act on.

The payoff isn't just the raw transcript—it's everything downstream. Once you have a diarized, multilingual record, you can generate per-speaker action items, feed it to a summarization model, or make a year of global meetings searchable in one index regardless of the language each was held in. Without auto-detection, every one of those meetings would need a human to tag its language first, which doesn't scale past a handful of recordings.

Media and entertainment

Subtitling and captioning for international content, podcasts with multilingual guests, film and broadcast localization. Broad language coverage matters here—you don't get to pick what language the next file is in, and now you don't have to, because Universal-3 Pro covers all 99 in one model. This is one of the ways streaming speech-to-text is being used when the captions need to appear live.

Multilingual customer support

Support orgs serve customers in dozens of languages, and any given call might code-switch. Transcribing those calls accurately feeds QA, compliance, and analytics—and it only works if the system handles whatever language the customer actually uses, mid-sentence switches included.

This is also where bad multilingual handling costs real money. If a quality-monitoring pipeline silently mistranscribes every Spanish-English call, your sentiment scores and compliance flags are built on garbage—and nobody notices until an audit. Getting detection and code-switching right at the transcription layer is what keeps the analytics on top of it trustworthy.

Multilingual voice agents

This is the real-time frontier. If you want an agent that talks back in the caller's language, you need transcription that detects and handles multiple languages live. That's exactly what Universal-3.5 Pro Realtime is built for: 18 languages at flagship accuracy, mid-sentence code-switching, and a language_code parameter that commits the model to a single language when you already know it—the cleanest way to head off wrong-language slips on short or ambiguous audio.

‍

You can build on it two ways. The Voice Agent API bundles speech-to-text, an LLM, and text-to-speech on one connection, so you can ship a multilingual conversational agent without stitching the layers together yourself. If you'd rather own the stack, the streaming speech-to-text model gives you the live transcription layer to build on—our guide to building AI voice agents covers the architecture.

Implementation: security, integration, and cost

Security and PHI

If your multilingual audio includes protected health information—say, a clinic transcribing patient calls in multiple languages—the data-handling story matters. AssemblyAI is a business associate under HIPAA and offers a Business Associate Addendum (BAA) for customers processing PHI. A BAA is available. For clinical audio specifically, Medical Mode significantly improves accuracy on medical terminology, which compounds with multilingual support when you're transcribing healthcare conversations across languages.

Integration

Integration is a single REST call. You POST your audio URL and parameters to https://api.assemblyai.com/v2/transcript, then poll or use webhooks for the result. The same request carries language_detection, speech_models, and speaker_labels—you're not calling three different services to get multilingual, diarized, code-switching-aware transcripts. The full parameter reference lives in the speech-to-text docs.

Cost

Cost is straightforward. For pre-recorded audio, Universal-3 Pro is $0.21/hr across all 99 languages—the same rate whether the file is English or Tamil, so you're not doing per-language cost math anymore. Universal-2 remains available at $0.15/hr for lower-cost coverage. For real-time, Universal-3.5 Pro Realtime is $0.45/hr base. No upfront commitments and unlimited concurrency, so a sudden spike in multilingual volume doesn't throttle you.

Best practices

A few things worth doing if you want this to hold up in production.

Default to Universal-3 Pro for pre-recorded multilingual audio—["universal-3-pro"]—and Universal-3.5 Pro Realtime for anything live. You get the highest accuracy and full language coverage from a single model in each mode, with nothing to route.

Turn on language_detection rather than hardcoding languages, even if you think you know what's coming. Real-world audio surprises you, and a wrong language assumption produces confident nonsense. In real time, when you do know the language up front, set language_code to commit the model to it.

Pair diarization with detection when speakers differ—they almost always do in meetings and support calls. The combination is what makes the transcript actually attributable.

And test on your messiest real audio, not clean samples. Code-switching, background noise, and accents are where multilingual systems separate. If your audio is live rather than recorded, the same detection and diarization logic applies—you're just running it over a real-time stream instead of a file.

AI vs. human transcription for multilingual audio

The old assumption was that multilingual work needed human transcriptionists—people fluent in each language. For a lot of cases, that's no longer the tradeoff worth making.

Human transcription is slow and expensive, and finding transcriptionists fluent in every language and comfortable with code-switching is genuinely hard. AI models handle 99 languages, run in near real time, scale to unlimited concurrency, and cost cents per hour. For high-volume, fast-turnaround work—support analytics, meeting notes, media at scale—AI models win clearly on speed, cost, and consistency.

Humans still have an edge on certain low-resource languages, heavy dialect, or audio so degraded that even a good model struggles. The pragmatic move is AI-first, with human review reserved for the genuinely hard edge cases—not as the default for every file. That's a very different cost structure than reviewing everything by hand.

Getting started

The fastest path: grab an API key, POST a multilingual file to /v2/transcript with language_detection=true, speech_models=["universal-3-pro"], and speaker_labels=true, and read the result. That single request gives you detection, diarization, and code-switching support across 99 languages in one shot.

For real-time multilingual transcription, start with the streaming getting-started guide on Universal-3.5 Pro Realtime. And if you're routing to an LLM for translation or summarization after transcription, the LLM gateway gives you 25+ models behind an OpenAI-compatible interface, so you can detect, transcribe, and translate in one pipeline.

The shift worth internalizing: multilingual transcription isn't a separate, harder product you bolt on later. It's a few parameters on the same request you're already making—which means the real question isn't whether you can handle multilingual audio, but whether you've turned it on before the audio that needs it shows up.

Frequently asked questions

What is multilingual transcription and how does it work?

Multilingual transcription is converting speech to text across more than one language, ideally with the system detecting the language automatically. It works by combining automatic language detection, per-language acoustic models, speaker diarization that holds up across languages, and code-switching support for sentences that mix languages. With AssemblyAI you enable it by setting language_detection=true and speech_models=["universal-3-pro"] on a single API request.

What speech-to-text API works best for multiple languages?

The best speech-to-text API for multiple languages combines broad language coverage with high per-language accuracy and automatic detection. AssemblyAI's Universal-3 Pro now covers 99 languages at its highest accuracy tier, so you no longer have to trade coverage for quality or route between models—one request handles everything from English to the long tail. For real-time audio, Universal-3.5 Pro Realtime delivers flagship accuracy across 18 languages with mid-sentence code-switching.

How do I transcribe audio in languages like Spanish, French, or German?

POST your audio to https://api.assemblyai.com/v2/transcript with speech_models=["universal-3-pro"], and either set the language explicitly or use language_detection=true to let the model identify it. Universal-3 Pro handles Spanish, French, German, and 96 other languages in the same request at the same rate, so there's no special configuration per language.

Can the API detect the spoken language and translate transcripts?

Yes—set language_detection=true and the API identifies the spoken language automatically, no per-file configuration needed. For translation, you transcribe first and then route the text through the LLM gateway, which gives you 25+ models behind an OpenAI-compatible interface to translate or summarize the transcript. That lets you detect, transcribe, and translate in a single pipeline.

Can AssemblyAI handle multilingual audio with code-switching?

Yes. AssemblyAI's models support native code-switching, so a single utterance that mixes languages—like an English sentence with a Spanish phrase dropped in—gets transcribed correctly across the switch instead of being mangled at the boundary. This works for pre-recorded audio on Universal-3 Pro and in real time on Universal-3.5 Pro Realtime, alongside automatic language detection and speaker diarization, so bilingual conversations stay accurate and properly attributed.