A team meeting with people in Madrid, Lyon and Munich has one audio stream and three audiences. The Spanish speakers want captions in Spanish, the French team wants French, and everyone wants to know who said what. This tutorial builds that: live captions for a multi-speaker meeting, labeled by speaker, translated into several languages from a single stream, with a translated subtitle file per language when the meeting ends.
If you only need captions for yourself in a Teams or Zoom call, use the built-in feature. If you're building a meeting platform, an AI notetaker or an events product and want translated captions inside your own product, you need an API, and that's what this guide covers.
Built-in meeting captions vs building your own
Microsoft Teams offers live translated captions in meetings as part of Teams Premium and Microsoft 365 Copilot licenses. In town halls and live events, organizers preselect up to six caption languages (ten with Premium), and each attendee sees captions in one of them (Microsoft Support). Zoom has a translated captions feature as well.
Those features live inside the meeting client. You can't put them in your own product, style them, store them with your own transcripts or export them in your format. Teams that need that build their own pipeline:
- Meeting and webinar platforms that want captions in their own player.
- AI notetakers that already capture meeting audio and want a translated view for multilingual teams (see AssemblyAI for captioning).
- Event tools where one stage feed serves an audience reading in many languages.
If that's you, the rest of this post walks through the architecture and the code.
The architecture: one stream in, many languages out
The whole system has four parts:
- Meeting audio. A recorded meeting replayed at real-time pace for development, or live audio from a virtual audio device in production.
- One streaming session per meeting on AssemblyAI-STT-3.7-Realtime with
speaker_labels: trueand anllm_gatewaytranslation prompt. Each final turn comes back as aTurnmessage with a speaker label, and its translation follows on the same WebSocket as anLLMGatewayResponsemessage (Real-time translation docs). - A caption server that joins each translation to its turn by
turn_orderand keeps the speaker label. - N viewers, each connected over a WebSocket and each picking a language.
meeting audio ──► STT-3.7-Realtime session (speaker_labels + llm_gateway)
│ Turn (source text, speaker_label)
│ LLMGatewayResponse (translations, same turn_order)
▼
caption server (FastAPI)
├──► viewer A ?lang=es
├──► viewer B ?lang=fr
└──► viewer C ?lang=de + meeting.<lang>.vtt at the endAssemblyAI-STT-3.7-Realtime is the model the docs recommend for notetakers and ambient transcription. It's $0.30/hr, covers 32 languages with native code switching, and includes real-time speaker labels (pricing). If your product talks back to the speaker, like a voice agent that interprets for a caller, use AssemblyAI-STT-3.7-Realtime-Agent ($0.45/hr) instead. The translation setup is the same.
There's no second translation API in this design. The server sends each final transcript to LLM Gateway with your prompt and returns the result on the connection you already have open.
Get speaker-labeled live captions and per-turn translation on one WebSocket. Create a free account and start with your own meeting recording.
Captions in several languages from one stream
There are two ways to serve three caption languages.
Option A: one session per target language. Stream the same meeting audio into three sessions, each with a single-language prompt. Each prompt is short and simple, but streaming is billed on session duration, so three sessions bill three times the meeting length. You also get three independent sets of turns and speaker labels to reconcile.
Option B: one session, one prompt that returns every language. Ask the gateway model for a JSON object keyed by language code, then parse it on your server. One session, one set of turns and speaker labels, and one gateway call per turn.
| A: one session per language | B: one session, JSON keyed by language | |
|---|---|---|
| Streaming sessions | One per language, each billed for its open time | One per meeting |
| Gateway calls per turn | One per language | One, with a longer output |
| Speaker labels and turns | Separate per session, can disagree | One shared set |
| Prompt | Plain text reply, nothing to parse | JSON reply, parsed server-side with a fallback |
| Best for | One target language, or one that needs its own model | Two to four caption languages |
This tutorial uses option B. Here's the prompt builder from the companion repo. The server fills in {{turn}} (the final transcript) and {{history_3}} (up to the last three turns whose translations have completed) for every turn, so those must stay exactly as written:
import json
LANGUAGE_NAMES = {"es": "Spanish", "fr": "French", "de": "German", "ja": "Japanese"}
_TEMPLATE = (
"You translate live meeting captions. Translate the current turn from whatever "
"language is spoken into each of these languages: __NAMES__.\n"
"Reply with one JSON object and nothing else. Use exactly these keys: __KEYS__. "
"Each value is the translation as a plain string. Keep people's names, product "
"names and numbers as spoken. If the turn is already in a target language, copy it.\n\n"
"Earlier in the meeting, for context only: {{history_3}}\n\n"
"Current turn: {{turn}}"
)
def build_prompt(targets):
names = ", ".join(f"{LANGUAGE_NAMES.get(c, c)} ({c})" for c in targets)
keys = ", ".join(json.dumps(c) for c in targets)
return _TEMPLATE.replace("__NAMES__", names).replace("__KEYS__", keys)
def max_tokens_for(targets):
# A few hundred tokens covers one turn in one language; scale per language.
return max(400, 250 * len(targets))History matters more in meetings than almost anywhere else. "Yes, that one." means nothing on its own and translates well with the question before it, which is the example the docs use. Three turns is the docs' suggested default for conversations.
A small model can wrap the JSON in a code fence or skip a key, so the parser is tolerant. Any language it can't read falls back to the source text, which means a caption line never goes blank:
import json
def parse_translations(content, targets, source):
"""Return ({lang: text}, ok). Missing languages fall back to the source text."""
text = (content or "").strip()
if text.startswith("```"):
text = text.strip("`")
if text.lower().startswith("json"):
text = text[4:]
start, end = text.find("{"), text.rfind("}")
data = {}
if start != -1 and end > start:
try:
data = json.loads(text[start:end + 1])
except json.JSONDecodeError:
data = {}
if not isinstance(data, dict):
data = {}
out, ok = {}, True
for lang in targets:
value = data.get(lang)
if isinstance(value, str) and value.strip():
out[lang] = value.strip()
else:
out[lang] = source
ok = False
return out, okSpeaker labels on translated lines
Set speaker_labels: true and every Turn message carries a speaker_label (A, B and so on) and a speaker_confidence score (streaming diarization docs). The LLMGatewayResponse doesn't repeat the label. It carries the same turn_order as its turn, so you store each final turn by turn_order and attach the translation when it arrives.
These are the messages the caption server handles:
| Message | Fields used | What the server does |
|---|---|---|
Turn, end_of_turn: false |
transcript, turn_order |
Shows the source-language partial as a dim line. Partials are never translated. |
Turn, end_of_turn: true |
transcript, turn_order, speaker_label, words |
Stores the caption with its speaker and timings, and starts a fallback timer. |
LLMGatewayResponse |
turn_order, data.choices[0].message.content |
Parses the JSON, attaches translations to the stored turn, sends each viewer its language. |
Three details from the docs shape the code:
- Short turns can be
PENDING. The model needs about a second of speech to place a speaker, so a quick "yeah" may arrive withspeaker_label: "PENDING". The overlay hides the label for those turns. Labels get more accurate as the meeting goes on and the model hears more from each voice. max_speakersis a hard cap. If more people speak than you allow, extra voices merge into the closest existing label. Leave it unset unless you know the headcount, and if you set it, add a little headroom.- Translations can arrive out of order. They're produced in the background, so two quick turns can come back swapped. Keying everything by
turn_orderhandles that.
Build it in Python
The companion repo, real-time-translated-meeting-captions, has five files: server.py, translate_prompt.py, captions.py, audio_source.py and overlay.html. It uses the assemblyai Python SDK 1.10.1, FastAPI and Uvicorn.
pip install assemblyai==1.10.1 fastapi uvicorn
export ASSEMBLYAI_API_KEY=your_api_key_hereOpen the session
The connection parameters turn on speaker labels and translation. LLMGatewayConfig takes the same model, messages and max_tokens fields as the LLM Gateway chat completions API, and qwen3.5-4b-32k-fast is the small, fast model the docs use for translation:
from assemblyai.streaming.v3 import RealTimeParameters
from assemblyai.streaming.v3.models import LLMGatewayConfig, LLMGatewayMessage
from translate_prompt import build_prompt, max_tokens_for
targets = ["es", "fr", "de"]
params = RealTimeParameters(
sample_rate=16000,
speech_model="assemblyai-stt-3-7-realtime", # AssemblyAI-STT-3.7-Realtime
speaker_labels=True,
llm_gateway=LLMGatewayConfig(
model="qwen3.5-4b-32k-fast",
messages=[LLMGatewayMessage(role="user", content=build_prompt(targets))],
max_tokens=max_tokens_for(targets),
),
)Don't set language_codes for a multilingual meeting. With no language parameters, the model code-switches across all 32 supported languages, and the prompt tells the gateway model to translate "from whatever language is spoken".
Handle turns and translations
The SDK calls your handlers on its read thread. on_turn stores final turns and publishes partials; on_llm_response attaches translations. If the gateway call fails or times out, the server sends no LLMGatewayResponse for that turn, so a timer shows the source text after three seconds:
import threading
class CaptionPipeline:
def __init__(self, store, hub, fallback_after=3.0):
self.store, self.hub, self.fallback_after = store, hub, fallback_after
def on_turn(self, client, event):
if not event.transcript:
return
if not event.end_of_turn:
msg = {"type": "partial", "turn_order": event.turn_order,
"speaker": event.speaker_label, "text": event.transcript}
self.hub.publish(lambda lang: msg)
return
cap = self.store.add_final_turn(event.turn_order, event.transcript,
event.speaker_label, event.words or [])
msg = {"type": "final", "turn_order": cap.turn_order,
"speaker": cap.speaker, "text": cap.source}
self.hub.publish(lambda lang: msg)
timer = threading.Timer(self.fallback_after, self._fallback, args=(cap.turn_order,))
timer.daemon = True
timer.start()
def on_llm_response(self, client, event):
try:
content = event.data["choices"][0]["message"]["content"]
except (KeyError, IndexError, TypeError):
content = ""
cap = self.store.add_translation(event.turn_order, content)
if cap is not None:
self._publish_translation(cap)
def _fallback(self, turn_order):
cap = self.store.fall_back(turn_order)
if cap is not None:
self._publish_translation(cap)
def _publish_translation(self, cap):
def build(lang):
return {"type": "translation", "turn_order": cap.turn_order,
"speaker": cap.speaker, "status": cap.status,
"text": cap.translations.get(lang, cap.source)}
self.hub.publish(build)hub.publish takes a function of the viewer's language, so one translation event becomes a Spanish message for Spanish viewers and a German one for German viewers. If a translation lands after the fallback fired, the store updates the caption and the overlay replaces the line.
Fan captions out to viewers
The hub is a dictionary of WebSocket connections and their languages. Because SDK callbacks run on a separate thread, publish hands the send to the server's event loop with asyncio.run_coroutine_threadsafe:
import asyncio
from fastapi import WebSocket
class Hub:
def __init__(self):
self.clients = {} # WebSocket -> language code
self.loop = None # set to the running loop at startup
def publish(self, build):
"""Thread-safe: called from the SDK's read thread."""
if self.loop is not None:
asyncio.run_coroutine_threadsafe(self._send_all(build), self.loop)
async def _send_all(self, build):
for ws, lang in list(self.clients.items()):
try:
await ws.send_json(build(lang))
except Exception:
self.clients.pop(ws, None)The /ws?lang=fr endpoint adds a viewer to the hub and sends the last five captions so a late joiner isn't staring at a blank screen. The streaming session runs in a background thread started from the FastAPI lifespan:
from assemblyai.streaming.v3 import (
RealTimeEvents,
RealTimeTranscriber,
RealTimeTranscriberOptions,
)
from audio_source import open_source
def run_transcriber(pipeline, params, audio, api_key, out_dir):
client = RealTimeTranscriber(
RealTimeTranscriberOptions(api_host="streaming.assemblyai.com"), api_key=api_key)
client.on(RealTimeEvents.Turn, pipeline.on_turn)
client.on(RealTimeEvents.LLMGatewayResponse, pipeline.on_llm_response)
client.on(RealTimeEvents.Error, lambda c, err: print(f"Streaming error: {err}"))
client.on(RealTimeEvents.Termination, lambda c, e: print(
f"Session ended: {e.audio_duration_seconds}s audio, "
f"{e.session_duration_seconds}s billed session time"))
client.connect(params)
try:
client.stream(open_source(audio))
finally:
client.disconnect(terminate=True)
write_vtt(pipeline.store, out_dir)Get meeting audio in
audio_source.py yields 50 ms chunks of 16 kHz, 16-bit mono PCM. It reads three kinds of input:
- A 16 kHz mono WAV, paced to the wall clock so the server sees live audio. This is the easiest way to develop against a recorded meeting.
- Any file or URL ffmpeg can decode, replayed with
-reat real-time pace. - A live capture device through ffmpeg, such as a virtual audio cable that receives the meeting's output, for example
ffmpeg:-f avfoundation -i :1on macOS.
Joining Zoom or Teams as a bot is out of scope here. Whatever gets you the meeting's mixed audio as PCM plugs into the same client.stream() call.
Run it and open the overlay in as many tabs as you want:
python server.py --audio standup.wav --langs es,fr,de
# then open http://127.0.0.1:8000/?lang=frThe repo's offline tests replay the documented Turn and LLMGatewayResponse shapes through the pipeline and check that a French viewer and a German viewer each get their own language from one event.
See every llm_gateway field, the prompt template variables and the message format in the real-time translation docs, then run the companion repo on your own meeting audio.
Caption UX for meetings
Translated captions trail the speaker by design: only final turns are translated, so a line appears in the source language while the person is talking, and the translation replaces it once the turn ends. A few choices make that feel deliberate:
- Show the source partial as a dim line. Viewers see that someone is speaking and roughly what they're saying, even in a language they don't read.
- Key every line by
turn_order. Insert translations in turn order, not arrival order, so swapped arrivals don't reshuffle the screen. - Mark fallbacks. The overlay italicizes lines that show source text because no translation arrived. Viewers learn that the line wasn't translated, and nothing goes missing.
- Keep the speaker label short. "A:" or a participant name you map from your own roster works better than a confidence score.
- Cap the visible lines. Four lines is plenty for a meeting overlay. For tiny screens like glasses, see the layout notes in our live translation for smart glasses and wearables post.
Turn length also affects the experience. Longer turns give the model more context per request and fewer, more complete translations. Shorter turns give faster but choppier captions.
Export a translated WebVTT file when the meeting ends
The caption store already has everything a subtitle file needs: the start of the first word, the end of the last word, the speaker and each translation. WebVTT's voice tag carries the speaker:
def to_webvtt(captions, lang):
"""captions: objects with start_ms, end_ms, speaker, source and translations."""
lines = ["WEBVTT", ""]
for i, cap in enumerate(captions, start=1):
text = cap.translations.get(lang, cap.source)
end = max(cap.end_ms, cap.start_ms + 500)
lines += [str(i), f"{_ts(cap.start_ms)} --> {_ts(end)}",
f"<v Speaker {cap.speaker}>{text}", ""]
return "\n".join(lines)
def _ts(ms):
h, rem = divmod(int(ms), 3_600_000)
m, rem = divmod(rem, 60_000)
s, ms = divmod(rem, 1000)
return f"{h:02d}:{m:02d}:{s:02d}.{ms:03d}"When the stream ends, the server writes captions_out/meeting.es.vtt, meeting.fr.vtt and meeting.de.vtt, and /captions/<lang>.vtt serves the current file during the meeting. Word timings are relative to the start of the session, so they line up with a recording that starts when streaming starts. For the differences between WebVTT, SRT and other formats, see our subtitle file format guide.
Cost, latency and scale
Cost has two parts. Streaming is billed on session duration: AssemblyAI-STT-3.7-Realtime is $0.30/hr, and the Termination message reports session_duration_seconds so you can log it. Translation runs on LLM Gateway and is billed on the model's input and output tokens. History variables and extra languages both add tokens. Per-model rates are in the LLM Gateway model list. This is one more reason to prefer option B: three caption languages cost one session, not three.
Latency. The docs say that with a small model such as qwen3.5-4b-32k-fast, a translation typically lands within about 100 ms of the final turn. The translation runs in the background and doesn't delay the final Turn message.
Scale. Each meeting is one session. Paid accounts start at 200 new streaming sessions per minute, the limit rises 10% whenever you use 80% or more of it, and there's no cap on how many sessions are open at once (rate limits).
Data. Translation requests go to LLM Gateway with zero data retention, so the gateway doesn't store the prompt or the translated text.
Where to go from here
The pattern here is one session per meeting, translations joined by turn_order, and a fan-out keyed by language. It extends cleanly. Map speaker labels to participant names from your meeting roster. Add a summary prompt alongside translation in a separate process. Or narrow it to one language pair: our Spanish to English speech translation guide covers a single ES→EN stream with regional-Spanish prompt rules and spoken English output.
For the theory behind cascaded speech translation, read the real-time speech translation pillar post, compare vendors in best real-time speech translation APIs, and see the translation feature page for batch translation of recorded meetings.
Start free with AssemblyAI-STT-3.7-Realtime, speaker labels and llm_gateway translation, all on one streaming connection.
Frequently Asked Questions (FAQs)
Can Zoom or Teams translate live captions, and when should I build my own?
Yes. Microsoft Teams offers live translated captions with Teams Premium or Microsoft 365 Copilot licenses, and Zoom has a translated captions feature too. Build your own when the captions need to live in your product: a meeting platform, an AI notetaker or an events tool where you control the display, storage and export.
How do I show who said what in translated captions?
Turn on speaker_labels in the streaming session. Each Turn message carries a speaker_label, and the translation arrives as an LLMGatewayResponse with the same turn_order, so you join the two on turn_order and print the label next to the translated text.
Can one meeting stream produce captions in several languages?
Yes. Use one session and a prompt that asks the gateway model for a JSON object keyed by language code, then parse it server-side and send each viewer their language. You pay for one streaming session per meeting instead of one per language, and every language shares the same turns and speaker labels.
Why do translated captions lag the spoken words?
Only final turns are translated, so the translation can't start until the speaker finishes the turn. After that, the docs put translation with a small model like qwen3.5-4b-32k-fast at about 100 ms. Showing the source-language partial while someone is talking covers most of the gap.
How much does real-time caption translation cost?
Streaming on AssemblyAI-STT-3.7-Realtime is $0.30 per hour of session time, and translation is billed on LLM Gateway tokens for the model you choose. Prompt history and extra caption languages add tokens, so check the per-model rates in the LLM Gateway docs.
Can I save translated captions as subtitles after the meeting?
Yes. Store each final turn with the start time of its first word, the end time of its last word, its speaker and its translations. When the meeting ends, write one WebVTT file per language, using the voice tag for the speaker.