Real-time speech translation: how it works and how to build it

Real-time speech translation turns spoken words in one language into text, or speech, in another while the speaker is still talking. To a user it looks like one feature. Underneath, it's a chain of decisions: when a sentence is finished, how much context the translator sees, and how many milliseconds each step is allowed to spend.

This guide is for developers building that chain. It covers the two architectures, a latency budget per stage, why you should translate final turns rather than partials, how to write the translation prompt, and a working Python build on Universal-3.6 Pro Realtime. If you need a refresher on streaming transcription itself, start with our guide to real-time speech-to-text. For supported languages and the batch translation feature, see the translation feature page.

How real-time speech translation works

Every live audio translation system does three jobs, in order:

1. Recognize the speech in the source language.
2. Translate the recognized words into the target language.
3. Deliver the result as captions, or as synthesized speech in the target language.

The hard part isn't any single job. It's deciding when to hand off from one to the next. Translate too early and you translate half a sentence that the speaker is about to change. Translate too late and the listener waits. Every design choice in this post comes back to that trade.

There are two ways to build the chain.

Cascaded pipelines: STT, then MT or an LLM, then optional TTS

A cascaded pipeline runs separate models for each job. A streaming speech-to-text model produces a transcript. A machine translation model or an LLM translates it. If you want spoken output, a text-to-speech model voices the translation.

The cost is extra hand-offs. The benefits are control and visibility:

• You get text at every stage. The source transcript and the translation are both strings you can log, display, search, redact or audit.
• You can swap or tune each stage. Change the translation model, add a glossary to the prompt, or switch the output voice without touching recognition.
• You can stop at captions. Many products, such as live captions, meeting subtitles and wearables with a display but no speaker, never need the TTS stage at all.

End-to-end speech-to-speech models

An end-to-end model takes audio in and returns translated audio out, sometimes with transcripts alongside. There are fewer hops, and some models try to carry over the speaker's voice or prosody.

What you give up is the seams. You can't inspect or correct the intermediate text before it's spoken, you can't change the translation model independently of the voice, and your language coverage is whatever the one model supports in both directions.

The rest of this guide builds a cascaded pipeline. The latency section shows where each hop spends its time, and why the hand-off point matters more than the number of hops.

A latency budget for each stage

Users judge live translation by one gap: the time between the speaker finishing a thought and the translation appearing or starting to play. Break that gap into stages and give each one a budget.

The figures below are the documented numbers for each stage of an AssemblyAI pipeline. Each comes from a different measurement, so don't add them up and quote the sum as a measured end-to-end figure. Use them to see where the time goes, then measure the full path in your own app.

The stage-2 figure comes from Pipecat's open STT benchmark, which measures from when the speaker stops to the final transcription segment. Pipecat's numbers use its own audio and aren't comparable to the other suites on the AssemblyAI benchmarks page.

Three things stand out.

Turn detection is the biggest lever. Stage 2 is mostly the system deciding the speaker is done. Shorter silence thresholds mean earlier translations but more fragments. The docs note that "longer turns give the model more context per request and fewer, more complete translations." The turn detection settings control this trade, including mode presets for latency or accuracy.

Model size sets the translation step. The docs advise: "Pick a small, fast model so the translation keeps pace with speech." The about-100 ms figure is for a small model. A large reasoning model will be slower.

Translation doesn't block transcription. The translation is produced in the background, so it doesn't delay the final transcript. You can show the source-language caption the moment the turn ends and swap in or add the translation a beat later.

Partials vs finals: when to translate

A streaming STT model sends two kinds of results for each turn. Partial transcripts arrive while the person is speaking. A final transcript arrives once the turn ends.

Partials are tempting for translation because they arrive first. They're also unstable. In AssemblyAI streaming, each Turn message re-transcribes the full turn audio, so every partial supersedes the last. The docs say to "always replace what you have rendered rather than appending." If you translate each partial, you pay for several translations per sentence, and the translated caption rewrites itself in front of the reader.

Language structure makes it worse. Word order differs between languages, so a translation of the first half of a sentence can be wrong once the second half arrives. A German verb at the end of the clause or a Japanese sentence-final negation can flip the meaning of everything before it.

That's why AssemblyAI's built-in real-time translation translates only final turns. From the real-time translation docs: "Only final turns are translated. Partial Turn messages stay in the spoken language."

A pattern that works well in practice:

• Show partials as live source-language captions. They tell the listener that someone is talking and roughly what about.
• Translate the final turn once. Display it under the source line, or replace the source line with it.
• Match translations to turns by turn_order. Translations for consecutive short turns can arrive out of order, so key everything on the turn number rather than arrival order.
• Plan for a missing translation. The docs say that "if the gateway call fails or times out, no LLMGatewayResponse is sent for that turn." Keep showing the source text if no translation arrives.

How to write a translation prompt

With an LLM doing the translation step, the prompt is your main quality control. AssemblyAI's real-time translation runs the prompt as a template, rendered on the server for each final turn, with two kinds of variables:

• {{turn}} is replaced with the final transcript of the turn being translated. Leave it exactly as written.
• {{history_1}} to {{history_10}} are replaced with up to the last ten final transcripts whose translation has already completed, oldest first. The current turn isn't included.

The docs' guidance on prompts is short: state the source language, the target language, and that the model must reply with the translation only. That last instruction matters more than it looks. Without it, many models add "Here's the translation:" in front of every caption.

History is where quality comes from. In the docs' words, "History is the main lever for translation quality. A single sentence like 'Yes, that one.' translates badly on its own and well with the question that preceded it." Their recommended starting point is three turns of history for conversations. Raise it for long monologues. Lower it for a voice agent where each turn stands alone, since every turn of history adds prompt tokens and so adds cost and a little latency.

One caveat: a turn joins the history only once its own translation returns. When turns come in quick succession, the turn just before the current one may not be in the history yet.

Here's the prompt the build below uses:

You are translating a live conversation from English into Spanish.
Earlier in the conversation: {{history_3}}

Translate the next turn into natural Spanish. Reply with only the Spanish translation of this turn: {{turn}}

To change the target language, change both mentions of Spanish. To handle speakers who switch languages, Universal-3.6 Pro Realtime already follows code-switching across its 32 languages with no configuration (see multilingual transcription and the full language list). The docs suggest telling the model to translate "from whatever language is spoken."

Build real-time speech translation in Python

The build uses the llm_gateway connection parameter on the Universal-3.6 Pro Realtime WebSocket. You open one streaming connection with a translation prompt attached. The server transcribes, sends each final turn to LLM Gateway, and returns the translation on the same WebSocket as an LLMGatewayResponse message. There's no second API to call. The full code for this build, including the TTS step and the JavaScript version, is in the companion GitHub repo.

Step 1: Install the SDK and set your key

You need Python 3, a microphone and an AssemblyAI API key. Install the SDK and PyAudio, and export your key:

pip install -U assemblyai pyaudio
export ASSEMBLYAI_API_KEY=<your-key>

On macOS, PyAudio needs PortAudio first (brew install portaudio).

Step 2: Stream the microphone and translate each final turn

Save this as translate_live.py. It streams English from your microphone, prints partials as a live caption, and prints the Spanish translation of each final turn.

import os

import pyaudio
from assemblyai.streaming.v3 import (
    BeginEvent,
    LLMGatewayResponseEvent,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    TerminationEvent,
    TurnEvent,
)
from assemblyai.streaming.v3.models import LLMGatewayConfig, LLMGatewayMessage

API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
SAMPLE_RATE = 16000
FRAMES_PER_BUFFER = 800  # 50 ms of audio at 16 kHz

# {{history_3}} and {{turn}} are filled in by the server for every final turn.
# Leave them exactly as written.
PROMPT = (
    "You are translating a live conversation from English into Spanish.\n"
    "Earlier in the conversation: {{history_3}}\n\n"
    "Translate the next turn into natural Spanish. "
    "Reply with only the Spanish translation of this turn: {{turn}}"
)

# Final transcripts, keyed by turn_order, so each translation can be
# joined back to the words it came from.
source_turns = {}


def microphone_stream():
    """Yield 16-bit PCM chunks from the default microphone."""
    audio = pyaudio.PyAudio()
    stream = audio.open(
        input=True,
        format=pyaudio.paInt16,
        channels=1,
        rate=SAMPLE_RATE,
        frames_per_buffer=FRAMES_PER_BUFFER,
    )
    try:
        while True:
            yield stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
    finally:
        stream.stop_stream()
        stream.close()
        audio.terminate()


def on_begin(self, event: BeginEvent):
    print(f"Session started: {event.id}. Start speaking English.\n")


def on_turn(self, event: TurnEvent):
    if not event.end_of_turn:
        # Partial: show it as a live caption in the spoken language.
        print(f"\r  ... {event.transcript}", end="", flush=True)
        return
    source_turns[event.turn_order] = event.transcript
    print(f"\r[{event.turn_order}] EN: {event.transcript}")


def on_llm_response(self, event: LLMGatewayResponseEvent):
    translation = event.data["choices"][0]["message"]["content"].strip()
    print(f"[{event.turn_order}] ES: {translation}\n")


def on_terminated(self, event: TerminationEvent):
    print(f"Session ended: {event.audio_duration_seconds} s of audio processed")


def on_error(self, error: RealTimeError):
    print(f"Error: {error}")


def main():
    client = RealTimeTranscriber(
        RealTimeTranscriberOptions(api_host="streaming.assemblyai.com"),
        api_key=API_KEY,
    )

    client.on(RealTimeEvents.Begin, on_begin)
    client.on(RealTimeEvents.Turn, on_turn)
    client.on(RealTimeEvents.LLMGatewayResponse, on_llm_response)
    client.on(RealTimeEvents.Termination, on_terminated)
    client.on(RealTimeEvents.Error, on_error)

    client.connect(
        RealTimeParameters(
            sample_rate=SAMPLE_RATE,
            speech_model="universal-3-6-pro",
            llm_gateway=LLMGatewayConfig(
                model="qwen3.5-4b-32k-fast",
                messages=[LLMGatewayMessage(role="user", content=PROMPT)],
                max_tokens=400,
            ),
        )
    )

    try:
        client.stream(microphone_stream())
    except KeyboardInterrupt:
        pass
    finally:
        client.disconnect(terminate=True)


if __name__ == "__main__":
    main()

Run it with python translate_live.py, speak a few sentences, and press Ctrl+C to stop. The llm_gateway object takes the same fields as the LLM Gateway chat completions API. The three that matter here:

• model: any model from the LLM Gateway models list. The build uses qwen3.5-4b-32k-fast, the small model the docs use for translation. If you leave model out, it defaults to gemini-2.5-flash-lite.
• messages: the prompt template. On the raw WebSocket you can send a single prompt string instead. The SDK config types take the messages form.
• max_tokens: a cap on the translation length. The docs note that a few hundred tokens covers any single turn.

Step 3: Read the output

Each final turn prints its source line first, and its translation lands a moment later. For a restaurant booking, the docs' example run looks like this:

Turn 0   Hi, I'd like to make a reservation at a restaurant.
         ¡Hola! Me gustaría hacer una reserva en un restaurante.
Turn 1   Saturday around 7 PM.
         Sábado a las 7 de la tarde.
Turn 2   4 people total.
         4 personas en total.

Each translation arrives as an LLMGatewayResponse message. Its data field is the LLM Gateway chat completion for that turn, so the translated text is at data.choices[0].message.content, and turn_order ties it to its source:

{
  "type": "LLMGatewayResponse",
  "turn_order": 2,
  "transcript": "4 people total.",
  "data": {
    "model": "qwen3.5-4b-32k-fast",
    "choices": [
      {
        "index": 0,
        "message": { "role": "assistant", "content": "4 personas en total." },
        "finish_reason": "stop"
      }
    ],
    "usage": { "prompt_tokens": 38, "completion_tokens": 7, "total_tokens": 45 }
  }
}

The usage block gives you prompt and completion tokens for every turn, so you can see what history length is costing you.

The script keeps final transcripts in source_turns by turn_order. A caption UI uses the same dictionary to render each translation under the right source line, even when translations arrive out of order. If you turn on speaker labels, the speaker_label is on the Turn message, so join it to the translation the same way.

Translation requests go to LLM Gateway with zero data retention, so the gateway doesn't store the prompt or the translated text.

The same build in JavaScript

The JavaScript SDK uses the same connection parameter, with camelCase option names and an llmGatewayResponse event. Install with npm install assemblyai node-record-lpcm16 and save as translate-live.mjs:

import { AssemblyAI } from "assemblyai";
import recorder from "node-record-lpcm16";
import { Readable } from "stream";

const PROMPT =
  "You are translating a live conversation from English into Spanish.\n" +
  "Earlier in the conversation: {{history_3}}\n\n" +
  "Translate the next turn into natural Spanish. " +
  "Reply with only the Spanish translation of this turn: {{turn}}";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-6-pro",
  llmGateway: {
    model: "qwen3.5-4b-32k-fast",
    messages: [{ role: "user", content: PROMPT }],
    max_tokens: 400,
  },
});

transcriber.on("turn", (turn) => {
  if (turn.end_of_turn) {
    console.log(`[${turn.turn_order}] EN: ${turn.transcript}`);
  }
});

transcriber.on("llmGatewayResponse", (response) => {
  const translation = response.data.choices[0].message.content.trim();
  console.log(`[${response.turn_order}] ES: ${translation}`);
});

transcriber.on("error", (error) => console.error(`Error: ${error}`));

await transcriber.connect();

const recording = recorder.record({ channels: 1, sampleRate: 16000, audioType: "wav" });
Readable.toWeb(recording.stream()).pipeTo(transcriber.stream());

process.on("SIGINT", async () => {
  recording.stop();
  await transcriber.close();
  process.exit();
});

If you'd rather call LLM Gateway from your own code, with full control over batching and retries, the docs keep a separate guide to translating streaming transcripts client-side.

Optional: speak the translation with AssemblyAI TTS

Captions are enough for many products. If you need spoken output, you can add the third stage of the cascade with AssemblyAI text-to-speech. The docs don't have a single speech-to-speech translation recipe. This section uses two documented APIs together: the real-time translation above, and a streaming TTS session.

One limit decides whether this works for you. AssemblyAI-TTS-1 speaks six languages: English, Spanish, French, German, Italian and Portuguese. Every voice speaks all six, though the TTS docs note that a voice usually sounds best in its own language. Your translation target has to be one of these six for AssemblyAI TTS to voice it.

The pattern is a queue and one worker thread. The streaming callbacks put each translation on the queue, and the worker speaks them over one TTS session that stays open for the whole run. The TTS docs recommend one long-lived session over a connection per sentence, which pays the handshake every time and runs into the per-account limit on new sessions per minute. TTS in the Python SDK needs release 1.8.2 or later.

Add this to translate_live.py, below the imports:

import os
import queue
import threading

import assemblyai as aai

to_speak = queue.Queue()


def speak_translations():
    """Speak each Spanish translation over one TTS session for the whole run."""
    config = aai.TTSConfig(voice_id="aai-antonioluis", language="spanish")
    with aai.TTSSession(config, api_key=os.environ["ASSEMBLYAI_API_KEY"]) as session:
        with open("translated.pcm", "wb") as out:
            while True:
                text = to_speak.get()
                if text is None:  # sentinel: the stream has ended
                    break
                for chunk in session.speak(text):  # raw 24 kHz 16-bit mono PCM
                    out.write(chunk)
                out.flush()


speaker = threading.Thread(target=speak_translations, daemon=True)

Then queue each translation in on_llm_response:

def on_llm_response(self, event: LLMGatewayResponseEvent):
    translation = event.data["choices"][0]["message"]["content"].strip()
    print(f"[{event.turn_order}] ES: {translation}\n")
    to_speak.put(translation)

And start and stop the worker in main():

def main():
    speaker.start()
    client = RealTimeTranscriber(
        RealTimeTranscriberOptions(api_host="streaming.assemblyai.com"),
        api_key=API_KEY,
    )
    # ... register the same handlers and call client.connect(...) as before ...

    try:
        client.stream(microphone_stream())
    except KeyboardInterrupt:
        pass
    finally:
        client.disconnect(terminate=True)
        to_speak.put(None)
        speaker.join(timeout=30)

Note that TTS takes the language as a full English name (spanish), while streaming STT uses short codes such as es. aai-antonioluis is the default Spanish voice. The TTS voices docs list the others.

The session writes raw PCM with no header. Play it with ffmpeg:

ffplay -f s16le -ar 24000 -ac 1 translated.pcm

In a live product you'd play each chunk as it arrives instead of writing a file. Two more things to handle in production. First, this worker speaks translations in arrival order, which can differ from turn_order for short back-to-back turns, so buffer by turn_order if strict ordering matters. Second, the TTS session closes after two minutes with no frames in either direction, so send a keep-alive when the speaker goes quiet.

Where real-time speech translation fits

The same pipeline shows up across very different products:

• Live captions and subtitles for events, lectures and broadcasts, where translated text appears under the speaker. See captioning use cases.
• Meetings and calls between people who don't share a language, with each line labeled by speaker.
• Wearables and smart glasses, where captions appear on a display that has no speaker.
• Multilingual voice agents and contact centers, where translation lets one team serve callers in several languages.

The architecture is the same in each. The settings change: how long a turn is, how much history the prompt carries, and whether you need the TTS stage. Start with captions, measure the gap between end of speech and translated text in your own app, and only add spoken output once the text path is fast and accurate enough.

Frequently Asked Questions (FAQs)

What is real-time speech translation?

Real-time speech translation converts spoken language into text or speech in another language while the speaker is still talking. Most systems recognize the speech with a streaming speech-to-text model, translate each finished sentence with a machine translation model or an LLM, and optionally voice the result with text-to-speech.

What's the difference between cascaded and end-to-end speech translation?

A cascaded system chains separate models: speech-to-text, then translation, then optional text-to-speech. You get the source transcript and translation as text at every step, and you can tune or swap each stage. An end-to-end model goes straight from audio to translated audio in one model. It has fewer hops but gives you less control over the intermediate text and language coverage.

How fast is real-time speech translation with AssemblyAI?

The documented figures are per stage. On Pipecat's STT benchmark, Universal-3.6 Pro Realtime returns the final transcript a median 307 ms after the speaker stops, and the docs say a small model's translation typically lands within about 100 ms of the final turn. These come from different measurements, so measure the full path in your own app rather than adding them up.

Should I translate partial transcripts or final transcripts?

Translate final transcripts. Partials are revised as the speaker continues, so translating them multiplies cost and makes translated captions rewrite themselves, and word order differences between languages can change the meaning of a half-finished sentence. AssemblyAI's built-in real-time translation translates only final turns and leaves partials in the spoken language, which you can show as live captions.

Which languages can I translate in real time?

The source language can be any of the 32 languages Universal-3.6 Pro Realtime transcribes, including speakers who switch languages mid-sentence. You set the target language in the prompt, so it depends on the LLM you choose. For spoken output with AssemblyAI TTS, the target must be English, Spanish, French, German, Italian or Portuguese. The translation feature page covers batch translation languages.

How is real-time translation billed?

There's no separate real-time translation price. The streaming session is billed on how long the WebSocket is open, and each translation is a separate LLM Gateway request billed on its input and output tokens at the chosen model's rate. LLM Gateway isn't covered by the free tier. Current rates are on the AssemblyAI pricing page and the LLM Gateway models list.