New AssemblyAI-TTS-1: Text-to-speech for voice agents. Learn more
Insights & Use Cases

How to build a multilingual voice agent with AssemblyAI

A hands-on build guide: stream speech-to-text across languages, steer and detect them, reply in the caller's language with AssemblyAI TTS, add tools, and test.

Abstract green sphere illustration

Written by

Kelsey Foster

Published on

28 October 2026

This is a build guide. By the end you'll have a voice agent that answers in more than one language, keeps up when a caller switches mid-sentence, calls your backend when it needs a real answer, and survives a dropped connection without making the caller start over.

The whole thing runs on three AssemblyAI endpoints and one API key: Universal-3.6 Pro Realtime streaming speech-to-text over a WebSocket, the LLM Gateway over HTTPS, and the AssemblyAI Text-to-Speech API over a second WebSocket. AssemblyAI Text-to-Speech delivers frontier-level quality and runs faster than other production voice agent TTS models. No SDK and no orchestration framework. If your language can open a socket, make an HTTP request, and parse JSON, you can build this.

If you're still deciding whether to build one — the architecture, the benchmarks, how to evaluate a vendor — the concept guide covers that ground. This post assumes you've decided.

What you'll build

A voice agent for a delivery company's support line. It greets callers, understands them in English, Spanish, or French without asking which they'd prefer, looks up an order when asked, and handles a caller who says "mi pedido es one-four-seven-two" without falling over at the language boundary.

Six steps:

  1. Open the sessions and configure the agent
  2. Steer the languages
  3. Tune transcription for the room and the vocabulary
  4. Give the agent a tool
  5. Pass conversation context and handle reconnects
  6. Test across languages

The code blocks in this guide, in order, make up one file: agent.py.

Before you start

You need three things:

  • An API key. Free to create, no card required. The same key works for speech-to-text, the LLM Gateway, and text-to-speech. LLM Gateway access may require a paid account; if yours doesn't have it yet, point LLM_URL at OpenAI's Chat Completions endpoint with your own OpenAI key, since the request and response shapes are the same.
  • Python 3.9+ with two packages. websockets (version 14 or newer) for the two sockets and requests for the LLM call.
  • A source of 16 kHz mono PCM16 audio. A microphone, a phone stream, or a WAV file you read off disk all work, as long as the audio matches the sample_rate you declare when you connect. Start with the file — it makes everything reproducible.
pip install -U websockets requests
export ASSEMBLYAI_API_KEY=your_assemblyai_key

The single most common setup mistake is the sample rate, and this stack has two of them. Audio goes up at the rate you declare on the speech-to-text connection (16 kHz here), and the agent's voice comes back at 24 kHz, the text-to-speech model's native rate. Declare one rate and send another, and you get garbage transcripts with no error to tell you why. Play 24 kHz audio as if it were 16 kHz, and the agent sounds slowed down and deep.

Budget-wise, each piece bills on its own meter. Universal-3.6 Pro Realtime streaming speech-to-text is $0.45 per hour, the LLM Gateway bills per token at each model's rate, and text-to-speech is $15 per 1M characters, billed on the characters of text you send. See pricing for current rates.

How the stack is wired

Three connections, one per job. Your audio goes up the speech-to-text socket, each finished caller turn goes to the LLM, and each reply goes down the text-to-speech socket as audio:

Connection You send You receive
Speech-to-text
wss://streaming.assemblyai.com/v3/ws
Raw PCM16 audio as binary messages; UpdateConfiguration to change settings mid-call; Terminate to end Begin; SpeechStarted when the caller starts talking; Turn with end_of_turn and language_code; Termination
LLM
https://llm-gateway.assemblyai.com/v1/chat/completions
The conversation so far, plus your tool definitions The reply text, or tool_calls for your backend to run
Text-to-speech
wss://tts.us.assemblyai.com/v1/ws
Generate and Flush per reply; Cancel on barge-in; KeepAlive; Terminate Begin; Audio with base64 PCM in audio; FlushDone; Cancelled; Termination

That's the whole protocol surface you need for this build. The multilingual streaming docs and the TTS message frames reference have the exhaustive lists.

Get a key and follow along

Create a free account, grab your API key, and open your first speech-to-text and text-to-speech sessions.

Sign up free

Step 1: Open the sessions

Start with the configuration and the two helpers that everything else plugs into: the system prompt for the LLM and the connection URL for text-to-speech.

import asyncio
import base64
import json
import os
import sys
import wave
from urllib.parse import urlencode

import requests
import websockets

API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
# For the EU data zone: wss://streaming.eu.assemblyai.com and wss://tts.eu.assemblyai.com
STT_HOST = os.environ.get("STT_HOST", "wss://streaming.assemblyai.com")
TTS_HOST = os.environ.get("TTS_HOST", "wss://tts.us.assemblyai.com")
LLM_URL = "https://llm-gateway.assemblyai.com/v1/chat/completions"
LLM_MODEL = "gemma-4-31b"

STT_SAMPLE_RATE = 16_000  # what you send: 16 kHz mono PCM16
TTS_SAMPLE_RATE = 24_000  # what you get back: 24 kHz mono PCM16

# assemblyai-tts-1 speaks every voice in all six of its languages, so one
# voice covers every reply language. Only the session's `language` changes.
VOICE_ID = "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3"  # Audrey

REPLY_LANGUAGES = {
    "en": ("english", "English", "Hi, thanks for calling. How can I help?"),
    "es": ("spanish", "Spanish", "Hola, gracias por llamar. ¿En qué puedo ayudarle?"),
    "fr": ("french", "French", "Bonjour, merci de votre appel. Comment puis-je vous aider ?"),
}
REPLY_LANGUAGE = "es"
# Reopen the TTS session when the caller settles into another reply language
FOLLOW_CALLER_LANGUAGE = False


def system_prompt(language_name):
    return (
        "You are a support agent for Parcelo, a delivery company. "
        f"Always reply in {language_name}, whatever language the caller uses. "
        "Keep every reply to one or two short sentences, with no lists or markdown. "
        "Use lookup_order whenever the caller asks about an order."
    )


def tts_url(language):
    params = {
        "voice_id": VOICE_ID,
        "language": language,
        "sample_rate": TTS_SAMPLE_RATE,
        "encoding": "pcm_s16le",
    }
    return f"{TTS_HOST}/v1/ws?" + urlencode(params)

The text-to-speech session gets its voice and its language in the URL. language takes the language's full English name ("spanish"), and the API doesn't detect language: the session speaks the language you set, whatever language the text is in.

One text-to-speech session per reply language

A text-to-speech session's voice, language, and model are fixed when it opens. You can change the speech-to-text languages mid-call, the transcription prompt, the keyterms, and the system prompt — but the reply language belongs to the session. That single constraint shapes the architecture of any genuinely multilingual line.

So don't open an English session and ask the system prompt to "reply in the caller's language": an English session treats Spanish text as English. Pick the reply language up front — from the number dialed, the caller's account record, or a brief language-selection turn — and open the session in that language. If the caller turns out to want a different reply language, close the session and open a new one.

The voice doesn't have to change. assemblyai-tts-1 speaks every voice in all six of its languages — English, Spanish, French, German, Italian, and Portuguese — so one voice ID covers every reply language here, and the agent keeps the same voice when it switches.

Here's the class that owns that session:

class Speaker:
    """One TTS WebSocket session per reply language: one request per agent reply."""

    def __init__(self, language):
        self.language = language
        self.ws = None
        self.speaking = False
        self.dropping = False  # between a Cancel and its Cancelled
        self.out = open("agent_audio.pcm", "ab")  # replace with your audio output

    async def open(self):
        self.ws = await websockets.connect(
            tts_url(self.language), additional_headers={"Authorization": API_KEY}
        )
        begin = json.loads(await self.ws.recv())
        if begin["type"] == "Error":
            raise RuntimeError(f"{begin['error_code']}: {begin['error']}")
        print("tts session:", begin["id"], begin["configuration"]["language"])
        self.reader = asyncio.create_task(self.read())

    async def say(self, text):
        self.speaking = True
        # Keep the trailing space: consecutive Generate texts are joined as sent
        await self.ws.send(json.dumps({"type": "Generate", "text": text.strip() + " "}))
        await self.ws.send(json.dumps({"type": "Flush"}))

    async def cancel(self):
        # Barge-in: stop the request in flight and drop audio already on the way
        if self.speaking:
            self.dropping = True
            self.speaking = False
            await self.ws.send(json.dumps({"type": "Cancel"}))

    async def switch(self, language):
        # A session's voice, language, and model are fixed when it opens,
        # so a new reply language means a new session
        await self.ws.send(json.dumps({"type": "Terminate"}))
        await self.reader
        self.language = language
        self.speaking = self.dropping = False
        await self.open()

    async def keep_alive(self):
        # A session closes after two minutes with no frame either way
        while True:
            await asyncio.sleep(60)
            await self.ws.send(json.dumps({"type": "KeepAlive"}))

    async def read(self):
        async for raw in self.ws:
            frame = json.loads(raw)
            kind = frame.get("type")
            if kind == "Audio" and not self.dropping:
                self.out.write(base64.b64decode(frame["audio"]))  # 24 kHz PCM16
            elif kind == "FlushDone":
                self.speaking = False
            elif kind == "Cancelled":
                self.dropping = False
            elif kind == "Error":
                print("tts error:", frame["error_code"], frame["error"])
            # Other frames, such as WordBoundaries, are ignored here

    async def close(self):
        await self.ws.send(json.dumps({"type": "Terminate"}))
        await self.reader  # ends at Termination, when the server closes
        self.out.close()

Each agent reply is one request: Generate carries the text, and Flush concludes the request so the server speaks all of it. Audio comes back as Audio frames with base64 PCM in the audio field, followed by a FlushDone. Consecutive Generate texts are joined exactly as sent, which is why say() keeps a trailing space.

When the caller talks over the agent, cancel() sends Cancel, which stops the request in flight and drops anything queued. A short tail of audio can still arrive between your Cancel and the server's Cancelled, so the reader drops audio until Cancelled comes back. A session closes after two minutes with no frames either way, which is easy to hit while a caller reads out a long address, so keep_alive() sends a KeepAlive every minute. switch() is the "new reply language means a new session" rule in code.

Audio goes up the speech-to-text socket as raw binary messages, about 50 ms at a time:

async def send_audio(stt_ws, path):
    """Stream a 16 kHz mono PCM16 WAV file at real time, 50 ms per message."""
    with wave.open(path, "rb") as wav:
        if wav.getframerate() != STT_SAMPLE_RATE or wav.getnchannels() != 1:
            raise ValueError("resample to 16 kHz mono before streaming")
        frames_per_chunk = STT_SAMPLE_RATE // 20
        while True:
            chunk = wav.readframes(frames_per_chunk)
            if not chunk:
                break
            await stt_ws.send(chunk)  # raw PCM16 bytes as a binary message
            await asyncio.sleep(0.05)
    await asyncio.sleep(3)  # let the last reply play out
    await stt_ws.send(json.dumps({"type": "Terminate"}))

Two rules that bite people. First, stream at real time. A file fired at the socket as fast as it reads arrives as one burst that no live caller could produce, so pace it the way a microphone would. Second, watch the encoding in each direction: speech-to-text takes raw PCM16 bytes as binary frames, while text-to-speech returns base64 inside JSON. That asymmetry is a trap, and decoding the wrong thing fails silently in the middle of a live call.

Step 2: Steer the languages

Universal-3.6 Pro Realtime is multilingual by default: open a session with no language parameters and it code-switches natively across all 32 of its languages, mid-sentence included. The language_codes connection parameter steers it toward the languages you expect. For a multilingual agent, declaring a short list is usually better than leaving it wide open — you narrow the search space without committing to one language.

# Step 2: the languages callers are expected to speak
LANGUAGE_CODES = ["en", "es", "fr"]

The model still code-switches between the languages you list, so a caller moving from Spanish to English mid-sentence is transcribed as spoken. You can list up to 10 codes. The stt_url() helper in the next step sends the list as JSON, and also sets language_detection, which adds a language_code and a language_confidence to each final turn so you know which language the caller actually used.

You can change the list mid-call. Send an UpdateConfiguration on the open socket and the new list applies from the next turn, with no reconnect — worth knowing if you're switching in response to something the caller just said:

# Narrow to Spanish from the next turn on
await stt_ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "language_codes": ["es"],
}))

# Clear the steering: back to native code-switching across every language
await stt_ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "language_codes": [],
}))

Where the six-language voice stops

Be precise about which direction you're configuring. Universal-3.6 Pro Realtime transcribes 32 languages. The text-to-speech model speaks six: English, Spanish, French, German, Italian, and Portuguese. Any language outside those six needs a different text-to-speech provider for the reply side.

So language_codes can name any of the 32, and recognition is rarely the constraint. What constrains you is which language the agent can answer in, and, because of the one-session-per-language rule above, how many text-to-speech sessions you need to cover those answers. The three in this build, English, Spanish, and French, sit inside both lists. A line that serves, say, Japanese or Hindi callers can still understand them with this setup, and needs another text-to-speech provider to answer in their language.

Step 3: Tune transcription for the room and the vocabulary

Three connection parameters do most of the work here. The stt_url() helper puts them on the URL along with the languages from step 2:

# Step 3: context and vocabulary for the speech model
STT_PROMPT = (
    "Customer support calls about parcel delivery. Callers give order "
    "numbers as digits and may switch between English, Spanish, and "
    "French mid-sentence."
)
KEYTERMS = ["Parcelo", "Express Saver", "depot"]


def stt_url(agent_context):
    params = {
        "sample_rate": STT_SAMPLE_RATE,
        "speech_model": "universal-3-6-pro",
        "language_codes": json.dumps(LANGUAGE_CODES),
        "language_detection": "true",  # each final turn reports its language
        "prompt": STT_PROMPT,
        "keyterms_prompt": json.dumps(KEYTERMS),
        "voice_focus": "far-field",
        "agent_context": agent_context,
    }
    return f"{STT_HOST}/v3/ws?" + urlencode(params)

prompt is free-form context about the conversation, and it's the most underused lever on this list. Telling the model that callers switch languages and read out digits measurably changes what it produces. The prompt carries context; transcription behavior such as punctuation and turn detection is built in.

How much detail is worth writing? The streaming docs benchmark three prompt levels against sending nothing at all, across 20,000 real voice-agent calls. A bare domain label of two to five words ("Parcel delivery support call.") cuts word error rate about 5%. A scenario sentence of five to fifteen words — what the conversation is actually about — cuts WER about 10% and takes name and place entity errors down 16% and 21%. A detailed prompt of twenty to fifty words, naming the people, products, and identifiers in play, nearly halves name errors at −49%, with overall entity errors down 29% and place names down 44%. Scenario context is the sensible default, because it asks for nothing your application doesn't already know; detailed context is what you reach for when you have the caller's record in hand. Two results worth knowing before you worry about overdoing it: hallucinated words go down as you add context, and turn detection is unaffected. The full table is in the streaming prompting documentation.

keyterms_prompt boosts exact strings: brand names, product tiers, internal jargon. It takes up to 100 terms per session, each 50 characters or shorter; a request with more than 100 returns an error, and longer terms are ignored. Multilingual audio makes this more valuable, because your product names are the tokens least likely to exist in the model's training data for any given language.

voice_focus isolates the primary speaker and suppresses everything behind them. Use near-field for headsets and phones, far-field for rooms and kiosks. Background chatter in a second language is a genuinely nasty failure mode, and this is what handles it.

prompt, keyterms_prompt, and language_codes can all be updated mid-call with UpdateConfiguration, and an empty list clears keyterms or language steering. If you need to shift the overall speed-versus-accuracy balance, the mode connection parameter takes min_latency, balanced, or max_accuracy.

Step 4: Give the agent a tool

Tools live in the LLM request. The LLM Gateway uses the OpenAI tool format: declare each function as JSON Schema in tools, and when the model wants one, its message carries tool_calls in place of text. Run the tool, append the result as a tool message with the same tool_call_id, and ask again.

# Step 4: tools are part of the LLM request
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "lookup_order",
            "description": "Look up a delivery order by its order number.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_number": {
                        "type": "string",
                        "description": "The order number, digits only",
                    },
                },
                "required": ["order_number"],
            },
        },
    }
]

ORDERS = {"1472": {"status": "out for delivery", "eta": "today before 6 pm"}}


def lookup_order(order_number):
    # Stand-in for your order system
    return ORDERS.get(order_number, {"status": "not found"})


def call_llm(messages):
    response = requests.post(
        LLM_URL,
        headers={"authorization": API_KEY},
        json={"model": LLM_MODEL, "messages": messages, "tools": TOOLS, "max_tokens": 300},
        timeout=30,
    )
    response.raise_for_status()
    return response.json()["choices"][0]["message"]


def run_llm_turn(messages):
    """Run one agent turn, executing tool calls until the model answers in words."""
    while True:
        message = call_llm(messages)
        tool_calls = message.get("tool_calls")
        if not tool_calls:
            reply = message.get("content") or ""
            messages.append({"role": "assistant", "content": reply})
            return reply
        messages.append({"role": "assistant", "content": None, "tool_calls": tool_calls})
        for call in tool_calls:
            args = json.loads(call["function"]["arguments"] or "{}")
            if call["function"]["name"] == "lookup_order":
                result = lookup_order(args.get("order_number", ""))
            else:
                result = {"error": "unknown tool"}
            messages.append({
                "role": "tool",
                "tool_call_id": call["id"],
                "content": json.dumps(result),
            })

run_llm_turn() loops until the model answers in words, so a turn that needs two lookups still ends in one spoken reply. arguments arrives as a JSON string, so parse it before you use it. Write tool names and descriptions in English regardless of the languages you support — the model reasons over the schema, and the system prompt is what sets the language of the reply.

LLM_MODEL can be any model on the LLM Gateway's available models list that supports tools. The request here waits for the whole reply, which keeps the code short. For lower latency, stream the reply from a model that supports streaming and send each sentence to the text-to-speech session as its own Generate as soon as it's complete.

Step 5: Pass conversation context and handle reconnects

This is the loop that ties the three connections together, and it's where conversation context comes in.

async def connect_stt(messages):
    # Seed agent_context with the agent's last line, so a fresh connection
    # knows what the caller is answering
    last_reply = next(m["content"] for m in reversed(messages) if m["role"] == "assistant")
    return await websockets.connect(
        stt_url(agent_context=last_reply[-1750:]),
        additional_headers={"Authorization": API_KEY},
    )


async def converse(stt_ws, speaker, messages):
    async for raw in stt_ws:
        event = json.loads(raw)
        kind = event.get("type")

        if kind == "Begin":
            print("stt session:", event["id"])

        elif kind == "SpeechStarted":
            await speaker.cancel()  # the caller is talking over the agent

        elif kind == "Turn" and event.get("end_of_turn"):
            text = event["transcript"]
            if not text:
                continue
            detected = event.get("language_code")
            print(f"caller [{detected}]:", text)

            if FOLLOW_CALLER_LANGUAGE and detected in REPLY_LANGUAGES:
                tts_language, language_name, _ = REPLY_LANGUAGES[detected]
                if tts_language != speaker.language:
                    await speaker.switch(tts_language)
                    messages[0] = {"role": "system", "content": system_prompt(language_name)}

            messages.append({"role": "user", "content": text})
            reply = await asyncio.to_thread(run_llm_turn, messages)
            print("agent:", reply)
            await speaker.say(reply)
            # Tell the speech model what the agent just asked
            await stt_ws.send(json.dumps({
                "type": "UpdateConfiguration",
                "agent_context": reply[-1750:],
            }))

        elif kind == "Termination":
            return

Every finished caller turn (end_of_turn set) goes to the LLM, and the reply goes to the text-to-speech session. SpeechStarted fires when the caller starts talking, which is the cue to cancel the agent mid-reply. In production you'll want to ignore backchannels like "mhm" while the agent talks; the streaming voice agent guide has a filter for that. With FOLLOW_CALLER_LANGUAGE on, a turn detected in another of your reply languages reopens the text-to-speech session in that language and updates the system prompt to match.

The UpdateConfiguration after each reply is the single highest-value thing you can do for accuracy. agent_context tells the speech model what the agent just said, up to 1,750 characters. Now when the caller answers "uno cuatro siete dos," the model knows an order number is coming rather than guessing at four loose words. This is the same lever as the transcription prompt in step 3, applied per turn, and short replies are exactly where the gain shows up. The code keeps the end of each reply, because the question at the end does the work.

Reconnects come out of the same design. The conversation lives in messages, in your process, so a dropped speech-to-text socket costs a reconnect, and the caller carries on without re-explaining themselves:

async def run_call(audio_path, speaker, messages):
    # The conversation lives in `messages`, in this process, so a dropped
    # speech-to-text connection costs a reconnect and the call carries on
    while True:
        stt_ws = await connect_stt(messages)
        sender = asyncio.create_task(send_audio(stt_ws, audio_path))
        try:
            await converse(stt_ws, speaker, messages)
            await sender
            return
        except websockets.exceptions.ConnectionClosedError:
            sender.cancel()
            print("speech-to-text connection dropped, reconnecting")
            await asyncio.sleep(1)


async def main(audio_path):
    tts_language, language_name, greeting = REPLY_LANGUAGES[REPLY_LANGUAGE]
    messages = [
        {"role": "system", "content": system_prompt(language_name)},
        {"role": "assistant", "content": greeting},
    ]

    speaker = Speaker(tts_language)
    await speaker.open()
    keep_alive = asyncio.create_task(speaker.keep_alive())
    await speaker.say(greeting)

    await run_call(audio_path, speaker, messages)

    keep_alive.cancel()
    await speaker.close()


if __name__ == "__main__":
    asyncio.run(main(sys.argv[1]))

connect_stt() seeds agent_context on the new connection with the agent's last line, so the first turn after a reconnect still knows what the caller is answering. In a live call, resume streaming from your audio source on the new socket; with a file, the demo simply starts it again. The text-to-speech session is independent of the speech-to-text socket and stays open through a reconnect. It lasts at most an hour, so a longer call opens a new one; see Limits.

When the call genuinely ends, send Terminate on both sockets, as send_audio() and Speaker.close() do. Each server answers with a Termination summary and closes the connection.

Run it against a 16 kHz mono WAV of a caller. The agent's audio lands in agent_audio.pcm as raw 24 kHz PCM16; swap the file write in Speaker for your audio output when you wire up a real line.

python agent.py caller-16k.wav

Step 6: Test across languages

Here's where multilingual agents get shipped broken, because the obvious test — one clean recording per language — passes easily and predicts nothing.

Test these five, in this order:

  • Clean single-language calls. Your baseline. If these fail, nothing else matters.
  • Code-switched calls. Record a bilingual speaker doing what they naturally do — an English product name inside a Spanish sentence, a French caller spelling a German surname.
  • Short replies. One-word and two-word turns in every language: "sí," "that's right," "quatre." Short utterances are where agents fail and where nothing in your aggregate metrics will warn you.
  • Entities under noise. Order numbers, addresses, and names read aloud over background audio. Flip voice_focus and mode and measure the difference rather than guessing.
  • Turn-taking per language. Count how often the agent interrupts and how long it waits after the caller stops, separately for each language.

Log each final Turn transcript, with its language_code, against your reference text and score entities separately from words. A 6% word error rate that misses every order number is worse than an 8% one that gets them all. Listen to the replies too: one voice across three reply languages is only convincing if the language on each session matches the text the LLM wrote.

Fireflies went through this evaluation for their own agent pipeline:

"We were searching for the best realtime ASR model for our voice agent pipeline in Fireflies. The new Universal 3.5 Pro speech model from Assembly is best so far in terms of accuracy, latency and language switching." — Foysal Osmany, Software Engineer at Fireflies

Try it before you wire up audio

Stream audio in the playground, switch languages mid-sentence, and see how the transcripts and turn detection respond.

Try playground

Where to go from here

You've got a working multilingual agent: language steering, contextual transcription, noise handling, tool calls, conversation context, and reconnects. What's left is specific to your product.

Three directions worth taking next. Put it on a phone number: telephony audio is 8 kHz μ-law, which both sockets accept as an encoding. Stream the LLM reply into the text-to-speech session sentence by sentence, so the agent starts talking sooner. And instrument the transcripts — you now have a per-language record of every turn, which is the raw material for finding out where your agent loses people.

The thing that surprises most teams at this point is how much of the perceived quality came from three lines of configuration — the transcription prompt, the keyterms list, and voice focus — compared with the model choice they spent two weeks agonizing over.

For the architectural reasoning behind these choices and how to benchmark them against alternatives, the multilingual voice agent guide is the companion to this one. The Voices reference covers the text-to-speech languages and voice library, the Python streaming walkthrough goes deeper on the speech-to-text socket, and the changelog is where new parameters show up first.

Ship your multilingual agent

One API key covers the speech-to-text, the LLM Gateway, and the voice. Create a key and open your first sessions.

Sign up free

Frequently asked questions

How do I build a multilingual voice agent?

Stream the caller's audio to Universal-3.6 Pro Realtime with a language_codes list of the languages you expect, send each finished turn to an LLM through the LLM Gateway with a system prompt that sets the reply language, and speak the reply through an AssemblyAI text-to-speech session opened in that language. Code-switching on the input side is handled by the speech model, so you don't route between per-language agents — but a text-to-speech session's language is fixed when it opens, so plan one session per language you reply in. The configuration that matters most for multilingual quality is the transcription prompt, the keyterms list, and voice focus.

Which languages can the agent understand and speak?

Universal-3.6 Pro Realtime transcribes 32 languages with native mid-sentence code-switching. AssemblyAI text-to-speech speaks six: English, Spanish, French, German, Italian, and Portuguese, and every voice speaks all six on the assemblyai-tts-1 model. For a reply language outside those six, you need a different text-to-speech provider for that language.

How would I use streaming speech-to-text in my voice agent?

Open a WebSocket to wss://streaming.assemblyai.com/v3/ws with speech_model=universal-3-6-pro, stream raw PCM16 audio as binary messages, and act on Turn events where end_of_turn is true: those are finished caller turns, ready for your LLM. Turn detection is built in, and SpeechStarted tells you when the caller starts talking so you can stop the agent's audio. After each agent reply, send agent_context in an UpdateConfiguration message so short caller responses resolve correctly.

How do I build a voice agent with function calling?

Put the tools in the LLM request. Declare each function as JSON Schema in the tools array of your LLM Gateway request, with a name, a description, and typed parameters. When the model wants one, its message carries tool_calls; run each call, append the result as a tool message with the matching tool_call_id, and send the conversation back until the model replies in text. Write tool names and descriptions in English even for a multilingual agent.

Can I build a language learning voice agent that handles code-switching?

Yes, and it's one of the few cases where code-switching is the product itself. List both the target language and the learner's native language in language_codes so the transcript follows when a learner drops back mid-sentence, and use the transcription prompt to describe the exercise, which helps the model expect hesitant and non-native pronunciation. For the agent's voice, the target language needs to be one of the six text-to-speech languages.

What happens if the connection drops during a call?

Keep the conversation in your own process and reconnect. The LLM history lives in your code, so a dropped speech-to-text socket only means opening a new one, seeded with the agent's last line as agent_context so the next caller turn is still understood in context. The text-to-speech session is a separate socket and stays open; if it does close, open a new one in the same language before the next reply. Send Terminate on both sockets when a call is genuinely over.