Build a production voice agent with LiveKit and AssemblyAI

Here's the honest version of what building a voice agent used to cost you: a WebRTC stack you didn't want to maintain, plus a speech-to-text vendor, plus an LLM provider, plus a text-to-speech engine, plus the glue code that keeps all four in sync while a human is mid-sentence. Four integrations, three of which have nothing to do with your product.

This tutorial cuts that to two. LiveKit — our partner, and the easiest way to ship production-grade real-time audio — handles transport and orchestration through the LiveKit Agents framework. AssemblyAI runs all three models behind it: Universal-3.6 Pro Realtime for speech-to-text, the LLM Gateway for the language model, and the AssemblyAI Text-to-Speech API for the voice. One AssemblyAI API key covers all three, and the whole agent is about 100 lines of Python.

You'll end up with a multi-user, browser-ready voice agent you can talk to in a real LiveKit room, with barge-in, tool calling, and turn detection working out of the box. Let's get you there fast, then tune it and deploy it.

"We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms." — David Zhao, Co-founder at LiveKit

Want the shortest path first? The LiveKit quickstart gets a minimal agent talking in a few minutes. This post is the production build: tools, tuning, multiple participants, data zones, and deployment.

The idea: LiveKit for transport, AssemblyAI for the models

WebRTC and voice AI are different problems, and they have different best-in-class answers.

• LiveKit is the transport layer. SDKs for Web, iOS, Android, React Native, Flutter, and Unity. Built-in recording, simulcast, adaptive bitrate, and end-to-end encryption. A managed cloud and a self-hostable open-source server. Its Agents framework runs your agent as a participant in the room and handles the orchestration: who's speaking, when to reply, and what happens when the user interrupts.
• AssemblyAI is the model layer. The livekit-plugins-assemblyai package gives you assemblyai.STT (Universal-3.6 Pro Realtime, with turn detection built in) and assemblyai.TTS (the assemblyai-tts-1 model, built for conversational voice agents). The LLM Gateway speaks the OpenAI Chat Completions API, so LiveKit's OpenAI plugin points straight at it. Same key for all three.

Universal-3.6 Pro Realtime is the piece I'd point to first. On AssemblyAI's English voice-agent benchmark it posts a 5.19% mean word error rate, with a 14.4% entity error rate on the names, numbers, and addresses your agent has to get right. It also takes the agent's own last line as input (agent_context), which is a bigger deal than it sounds: short replies like "yeah, the second one" resolve far better when the model knows what was just asked. On livekit-agents 1.6.6 and later, AgentSession forwards each agent reply into agent_context for you.

The voice side is built for the same job. assemblyai-tts-1 was trained on millions of hours of audio, including real conversational data, and the LiveKit plugin is shaped around a conversation: it streams the LLM's reply one sentence at a time, keeps one WebSocket session open for every reply in the call, cancels in-flight audio on barge-in without reconnecting, and publishes per-word timings so LiveKit knows exactly how much of an interrupted reply the user heard. The model itself delivers frontier-level quality, runs faster and costs less than other production voice agent TTS models, and has unlimited concurrency.

This is the "invisible infrastructure" framing, and it's deliberate. You don't want your users to notice the speech pipeline any more than they notice TCP. The speech model, the LLM, and the voice are meant to disappear into a conversation that just works. Your job is the product on top; our job is to make the layer underneath boring in the best way.

Swapping a piece out

Every model in this agent is its own plugin, so each one can change without touching the others. Already pay for OpenAI directly? Drop the base_url and api_key arguments from openai.LLM and set OPENAI_API_KEY. Want a different model on the Gateway? Change the model string to anything on the available models list. The rest of the agent stays the same.

Who does what

Architecture

Browser / iOS / Android client (any LiveKit SDK)
        │
        │  WebRTC into a LiveKit room
        ▼
LiveKit Cloud (or self-hosted livekit-server)
        │
        │  dispatches a job to your agent worker for each new room
        ▼
agent.py — LiveKit Agents AgentSession, joined to the room
        │
        ├── assemblyai.STT  → wss://streaming.assemblyai.com
        │                     Universal-3.6 Pro Realtime: transcripts + end of turn
        │
        ├── openai.LLM      → https://llm-gateway.assemblyai.com/v1
        │                     LLM Gateway: the reply, plus tool calls
        │
        └── assemblyai.TTS  → wss://tts.us.assemblyai.com
                              assemblyai-tts-1: one session, one request per sentence
        │
        │  agent audio published as a track in the same room
        ▼
LiveKit Cloud → WebRTC out → the client hears the agent

Your code never touches WebRTC, and it never touches audio frames either. LiveKit moves the media and runs the turn-taking loop; the three plugins turn audio into text, text into a reply, and the reply back into audio.

Prerequisites

• Python 3.10+ (the LiveKit Agents framework requires it).
• An AssemblyAI API key. The same key works for speech-to-text, text-to-speech, and the LLM Gateway. LLM Gateway access may require a paid account; if yours doesn't have it yet, use your own OpenAI key for the LLM (see "Swapping a piece out" above).
• A LiveKit server — either a free LiveKit Cloud project or a self-hosted livekit-server. Console mode, the first test below, doesn't need one.
• A LiveKit client to talk to the agent. The fastest path is the hosted LiveKit Agents Playground.

Want a complete project to fork instead of building from scratch? AssemblyAI's reference voice agent for LiveKit runs every model on AssemblyAI and ships with tests and a Dockerfile for LiveKit Cloud.

Quick start

1. Install

mkdir livekit-voice-agent && cd livekit-voice-agent
python -m venv .venv && source .venv/bin/activate

pip install "livekit-agents[openai,silero]~=1.8" \
    "livekit-plugins-assemblyai>=1.8.9" \
    python-dotenv

The openai extra installs LiveKit's OpenAI plugin, which is how the agent talks to the LLM Gateway. silero is the voice activity detector LiveKit uses for barge-in.

2. Configure environment

Create a .env file:

ASSEMBLYAI_API_KEY=           # https://www.assemblyai.com/dashboard/signup

# Not needed for console mode, only to connect to LiveKit
LIVEKIT_URL=wss://<project>.livekit.cloud
LIVEKIT_API_KEY=              # LiveKit Cloud → Settings → Keys
LIVEKIT_API_SECRET=

For self-hosted LiveKit, run livekit-server --dev and use LIVEKIT_URL=ws://localhost:7880. The dev server prints an API key and secret on startup.

3. Write the agent

Save this as agent.py. It's the whole agent: speech-to-text, the LLM, text-to-speech, a greeting, and one tool.

import os

from dotenv import load_dotenv
from livekit.agents import (
    Agent,
    AgentServer,
    AgentSession,
    JobContext,
    JobProcess,
    RunContext,
    TurnHandlingOptions,
    cli,
    function_tool,
)
from livekit.plugins import assemblyai, openai, silero

load_dotenv()

# Silero (used for barge-in) should agree with AssemblyAI's own VAD on what
# counts as speech, so both use the STT's default threshold.
VAD_THRESHOLD = 0.2

# Stand-in for your order system
ORDERS = {
    "1472": {"status": "out for delivery", "eta": "today before 6 pm"},
    "2210": {"status": "delivered", "eta": "delivered yesterday"},
}


class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions=(
                "You are a customer support agent for Acme. Speak in one or two "
                "short sentences, with no lists, markdown, or emojis. Confirm the "
                "caller's question before answering. Use lookup_order whenever "
                "the caller asks about an order."
            ),
        )

    async def on_enter(self) -> None:
        # A fixed greeting is spoken right away, without waiting on the LLM
        self.session.say("Hi, this is Acme support. What's going on?")

    @function_tool
    async def lookup_order(self, context: RunContext, order_number: str) -> dict:
        """Look up a customer's order by its order number.

        Args:
            order_number: The order number, digits only.
        """
        return ORDERS.get(order_number, {"status": "not found"})


server = AgentServer()


def prewarm(proc: JobProcess) -> None:
    proc.userdata["vad"] = silero.VAD.load(activation_threshold=VAD_THRESHOLD)


server.setup_fnc = prewarm


@server.rtc_session()
async def entrypoint(ctx: JobContext) -> None:
    tts = assemblyai.TTS(
        model="assemblyai-tts-1",
        voice="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3",  # Audrey
    )
    # Open the TTS session now, so the first reply doesn't wait on the handshake
    tts.prewarm()

    session = AgentSession(
        stt=assemblyai.STT(
            model="universal-3-6-pro",
            mode="balanced",
            vad_threshold=VAD_THRESHOLD,
            keyterms_prompt=["Acme", "LiveKit", "AssemblyAI"],
        ),
        # The LLM Gateway, through its OpenAI-compatible API
        llm=openai.LLM(
            model="gemma-4-31b",
            base_url="https://llm-gateway.assemblyai.com/v1",
            api_key=os.environ["ASSEMBLYAI_API_KEY"],
        ),
        tts=tts,
        vad=ctx.proc.userdata["vad"],
        turn_handling=TurnHandlingOptions(
            # AssemblyAI decides when the user has finished speaking...
            turn_detection="stt",
            # ...so LiveKit shouldn't add its own delay on top
            endpointing={"min_delay": 0},
        ),
        tts_text_transforms=["filter_emoji", "filter_markdown"],
        # When the user interrupts, keep only the words they heard in the chat history
        use_tts_aligned_transcript=True,
    )

    await session.start(agent=Assistant(), room=ctx.room)


if __name__ == "__main__":
    cli.run_app(server)

4. Talk to it in your terminal

# One-time: download the Silero VAD model files
python agent.py download-files

# Talk to the agent with your own microphone and speakers
python agent.py console

Console mode uses your computer's microphone and speakers and needs only your AssemblyAI API key. Use headphones, so the agent doesn't hear its own voice. Recent livekit-agents releases print a deprecation notice for console and point at lk agent console, which runs the same mode through the LiveKit CLI. On a machine without a microphone, python agent.py console --text takes typed turns.

5. Connect it to a LiveKit room

With your LiveKit credentials in .env, run the worker in development mode:

python agent.py dev

The worker registers with your LiveKit project and waits. Because @server.rtc_session() sets no agent_name, LiveKit dispatches the agent into every new room automatically. Then connect a client — the fastest way is the LiveKit Agents Playground:

• Open the playground and connect it to your LiveKit Cloud project, or paste your LIVEKIT_URL and a token generated in the LiveKit Cloud dashboard (Settings → Keys → Create token).
• Click Connect, allow microphone access, and start talking. The agent joins the room, greets you, and answers through your browser.

You can also connect from your own LiveKit Web, iOS, Android, or Flutter app pointed at the same project.

How it works

The agent is one file. Five pieces of it do the real work.

1. Speech-to-text and end of turn

stt=assemblyai.STT(
    model="universal-3-6-pro",
    mode="balanced",
    vad_threshold=VAD_THRESHOLD,
    keyterms_prompt=["Acme", "LiveKit", "AssemblyAI"],
),

assemblyai.STT streams the room's audio to Universal-3.6 Pro Realtime and gets back transcripts plus an end-of-turn decision. mode="balanced" applies the turn-silence settings that match streaming directly to the API, which is what you want when the STT decides when the user is done. vad_threshold keeps AssemblyAI's voice activity detection in step with Silero's, so both agree on what counts as speech. keyterms_prompt boosts the names your agent has to hear right.

2. The LLM, through the LLM Gateway

# The LLM Gateway, through its OpenAI-compatible API
llm=openai.LLM(
    model="gemma-4-31b",
    base_url="https://llm-gateway.assemblyai.com/v1",
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
),

The LLM Gateway speaks the OpenAI Chat Completions API, so LiveKit's stock openai.LLM works once you point base_url at it and pass your AssemblyAI key. model takes any model on the available models list that supports tools and streaming.

3. The voice

tts = assemblyai.TTS(
    model="assemblyai-tts-1",
    voice="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3",  # Audrey
)
# Open the TTS session now, so the first reply doesn't wait on the handshake
tts.prewarm()

assemblyai.TTS keeps one WebSocket session open for the whole call and sends each sentence of the LLM's reply as its own request, so the agent starts speaking as soon as the first sentence is ready. Set voice explicitly: the plugin's default voice differs from the API's own default, and naming the voice keeps your agent's sound from changing under you. tts.prewarm() opens the session while the agent is still joining the room, so the greeting doesn't wait on the handshake.

4. Turn handling and barge-in

turn_handling=TurnHandlingOptions(
    # AssemblyAI decides when the user has finished speaking...
    turn_detection="stt",
    # ...so LiveKit shouldn't add its own delay on top
    endpointing={"min_delay": 0},
),
tts_text_transforms=["filter_emoji", "filter_markdown"],
# When the user interrupts, keep only the words they heard in the chat history
use_tts_aligned_transcript=True,

turn_detection="stt" hands the end-of-turn decision to Universal-3.6 Pro Realtime, and min_delay: 0 stops LiveKit from adding its own wait on top. When the user talks over the agent, LiveKit stops playback and the TTS plugin sends Cancel on the open session, dropping the audio still in flight while keeping the session for the next reply. use_tts_aligned_transcript=True uses the per-word timings from the TTS so the chat history keeps the interrupted reply only up to the word the user heard — the LLM won't think it said things it didn't.

5. Tool calling

@function_tool
async def lookup_order(self, context: RunContext, order_number: str) -> dict:
    """Look up a customer's order by its order number.

    Args:
        order_number: The order number, digits only.
    """
    return ORDERS.get(order_number, {"status": "not found"})

Tools live in the LLM request. Any method on your Agent decorated with @function_tool is sent to the LLM as an OpenAI-style tool, with its schema built from the type hints and its description from the docstring. When the model calls it, LiveKit runs the method and feeds the return value back to the model, which then answers out loud. Swap the ORDERS dictionary for a call to your real backend.

Tuning the agent

Pick a voice and a language

# Any voice ID from the dashboard voice library, or a voice you cloned
tts = assemblyai.TTS(
    voice="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3",  # Audrey
    language="es",  # a code or an English name such as "spanish"
)

# Later in the call: the next reply opens a new session with these options
tts.update_options(language="french")

Copy a voice ID from the voice library in the dashboard, which holds thousands of voices, or clone one from a short recording of a speaker who consents to it. assemblyai-tts-1 speaks every voice in all six of its languages: English, Spanish, French, German, Italian, and Portuguese. The API doesn't detect language, so set language whenever the LLM writes in a language other than the voice's own. A session's voice, language, and model are fixed when it opens, so update_options takes effect on the next reply, which opens a new session. See Voices for the full reference.

Adjust the instructions and greeting

These are the two highest-leverage settings for how the agent feels, and both live in the Assistant class: instructions is the system prompt, and the line passed to self.session.say() in on_enter is the greeting. A fixed greeting is spoken the moment the user joins, without waiting on the LLM. Keep the instructions asking for short, plain sentences: the agent speaks after the first sentence, so a short first sentence gets audio out sooner.

Tune turn detection

The defaults work for most apps. If you need to shift the balance, change mode before you reach for individual silence settings:

stt=assemblyai.STT(
    model="universal-3-6-pro",
    mode="max_accuracy",  # "min_latency" · "balanced" · "max_accuracy"
    vad_threshold=VAD_THRESHOLD,
),

min_latency hands turns off fastest, balanced is the right default for most conversations, and max_accuracy favors quality when callers read out long entities such as account numbers or addresses. The Universal-3.6 Pro on LiveKit guide covers the individual turn-silence settings and when they're worth changing.

Boost domain terms and tune for the room

stt=assemblyai.STT(
    model="universal-3-6-pro",
    mode="balanced",
    vad_threshold=VAD_THRESHOLD,
    # What the conversation is about, as context for the model
    prompt="Customer support call about a delayed delivery.",
    # Exact names and terms to boost
    keyterms_prompt=["Acme", "Express Saver", "LiveKit"],
    # Laptop mics and open offices: isolate the speaker's voice
    voice_focus="far-field",
),

• prompt describes what the conversation is about. Start with none, and add context only if domain vocabulary is being misrecognized.
• keyterms_prompt boosts exact strings: product names, brand names, rare proper nouns.
• voice_focus isolates the speaker's voice before it reaches the recognizer. LiveKit rooms often carry laptop-mic or open-office audio, so use "far-field" for that and "near-field" for headsets and handsets.

You don't need to wire up conversation context yourself. On livekit-agents 1.6.6 and later, AgentSession forwards each agent reply into the STT's agent_context, so a terse "uh, the blue one" is transcribed against the question that prompted it. prompt, keyterms_prompt, agent_context, and language_codes can all change mid-call with update_options on the STT, without reconnecting.

Multiple participants in one room

An AgentSession listens to one participant: the first one to join the room, unless you name another with room_options=room_io.RoomOptions(participant_identity=...) in session.start(). Additional participants can hear the agent, but it doesn't respond to them. There are two ways to scale this:

• One agent per room. LiveKit dispatches a job to your worker for every new room, and each job runs its own AgentSession. Best for one-on-one use cases like phone-style support agents.
• Hand the floor between participants. If you genuinely want a meeting-style agent, switch the linked participant as the conversation moves; see LiveKit's Agents documentation for the room input options.

For most real-world apps, option 1 is the right call.

Deploy it

For production, run the worker with start, which runs it in production mode and gives active calls time to finish before the worker shuts down:

python agent.py start

Each worker process registers with LiveKit and takes jobs as rooms are created, so you scale out by running more of them; there's no shared coordination point. You can host the workers yourself, in a container, or on LiveKit Cloud — see LiveKit's deployment guide, and the reference agent for a ready-made Dockerfile.

Keep data in the EU

Each plugin takes a base_url for its data zone. To keep audio and text in the EU, point all three at their EU hosts:

assemblyai.STT(..., base_url="wss://streaming.eu.assemblyai.com")
assemblyai.TTS(..., base_url="wss://tts.eu.assemblyai.com")
openai.LLM(..., base_url="https://llm-gateway.eu.assemblyai.com/v1")

The EU LLM Gateway serves a different set of models, so check the available models before you pick one. Keep the worker near the data zone it uses, too: every new session pays a few network round trips for its handshake.

Troubleshooting

The agent never joins the room. Check that the worker printed its registration with your LiveKit project and that LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET all come from the same project. If you set an agent_name on @server.rtc_session(), automatic dispatch is off and the agent only joins rooms that request it by name.

Authentication errors from AssemblyAI. Your ASSEMBLYAI_API_KEY is missing, expired, or pasted with whitespace. Grab a fresh key from the AssemblyAI dashboard and confirm .env is loaded. If only the LLM fails, your account may not have LLM Gateway access yet; switch openai.LLM to your own OpenAI key while you sort it out.

LiveKit ConnectError: invalid token. The JWT signature didn't validate against the LIVEKIT_API_SECRET. Check that the URL, key, and secret all come from the same LiveKit project. For self-hosted, regenerate keys with livekit-server generate-keys.

Silero model files are missing. Run python agent.py download-files once on every machine or container image that runs the worker.

The first reply is slow after a long silence. A TTS session closes after two minutes with no frames either way, and the next reply opens a new one, which waits on the handshake. The plugin logs the close as a warning. That's expected in calls with long pauses; see Limits for session lifetimes.

Agent keeps interrupting itself. Browser clients with echo cancellation (the default for getUserMedia) handle this automatically. If you see it on a custom mobile client, make sure acoustic echo cancellation is enabled on the capture side. In console mode, use headphones.

The TTS on LiveKit and Universal-3.6 Pro on LiveKit docs cover every plugin parameter.

Try it, then build the real thing

You've got a working voice agent in a real LiveKit room. Barge-in works, turn detection works, tools work, and the only orchestration you wrote was a constructor call.

Here's the insight worth leaving with: this collapses to two integrations because "transport" and "the models" are the two boundaries in a voice agent that actually deserve to be separate systems. Everything else — one key per model, one bill per model, one data-residency review per model — only got split across vendors for historical reasons, and every seam between them is a place latency, bugs, and procurement delays hide. When you're deciding how to architect your next agent, that's the question to ask about any component: is this a real boundary, or an accident of who sold it to you?

Related reading

• LiveKit voice agent quickstart
• Build and deploy real-time AI voice agents using LiveKit and AssemblyAI
• Universal-3.6 Pro Realtime
• AssemblyAI Text-to-Speech API
• LLM Gateway
• Streaming TTS on LiveKit (docs)
• What are AI voice agents?
• Pricing

Frequently asked questions

How do I build a voice agent with LiveKit?

Use the LiveKit Agents framework with the AssemblyAI plugin. Install livekit-agents[openai,silero] and livekit-plugins-assemblyai 1.8.9 or later, then build an AgentSession with assemblyai.STT(model="universal-3-6-pro") for speech-to-text, openai.LLM pointed at the LLM Gateway for the reply, and assemblyai.TTS for the voice. Run it with python agent.py console to test locally, or python agent.py dev to join LiveKit rooms. The full agent above is about 100 lines.

Does AssemblyAI integrate with LiveKit?

Yes. LiveKit is an active AssemblyAI partner, and the livekit-plugins-assemblyai package provides both assemblyai.STT (Universal-3.6 Pro Realtime, with turn detection) and assemblyai.TTS (AssemblyAI's streaming text-to-speech). Add the LLM Gateway through LiveKit's OpenAI plugin and every model in the agent runs on one AssemblyAI API key. Universal-3.6 Pro is also available through LiveKit Inference.

How is AssemblyAI different from a voice agent platform like Retell?

They sit at different layers of the stack. Retell is a hosted voice agent platform with its own dashboard, agent builder, and workflow. AssemblyAI is infrastructure: the speech-to-text, LLM access, and text-to-speech models you compose into your own agent, in your own code, with LiveKit, Pipecat, or your own orchestration carrying the audio.

Do I have to use the LLM Gateway?

No. The LLM is a separate plugin, so you can point openai.LLM at OpenAI directly with your own OPENAI_API_KEY, or use any other LiveKit LLM plugin. The Gateway's advantage is one key and one bill for every model in the agent, plus access to models from several providers through one OpenAI-compatible API. LLM Gateway access may require a paid account.

What speech-to-text model does this agent use?

Universal-3.6 Pro Realtime (universal-3-6-pro), AssemblyAI's streaming model for voice agents. It posts a 5.19% mean word error rate on AssemblyAI's English voice-agent benchmark, transcribes 32 languages with native code-switching, decides end of turn itself, takes the agent's last reply as agent_context, and includes voice_focus for isolating a speaker's voice in noisy rooms. See the Universal-3.6 Pro Realtime announcement for the full breakdown.

What audio format does the agent need?

None you have to handle. LiveKit receives WebRTC audio from the client and hands it to the STT plugin, and the TTS plugin returns 16-bit PCM at 24 kHz by default, the model's native rate, which LiveKit publishes back into the room. If you need a different output rate, assemblyai.TTS takes sample_rate values of 8000, 16000, 22050, 24000, 44100, or 48000. For phone calls, LiveKit's SIP integration brings callers into the same rooms, so the agent code doesn't change.

Can the agent call tools from inside a LiveKit room?

Yes. Decorate a method on your Agent with @function_tool, give it type hints and a docstring, and LiveKit sends it to the LLM as an OpenAI-style tool. When the model calls it, LiveKit runs your method and passes the result back to the model before it answers. Tool calls work the same whether the audio comes from a browser, a mobile app, or a phone line.

How do I scale to many concurrent rooms?

Run more workers. LiveKit dispatches a job to an available worker for every new room, each job runs its own AgentSession, and workers scale horizontally with no shared coordination point. On the TTS side, AssemblyAI's rate limits count new sessions per minute, and the plugin reuses one session per call, so each call opens few sessions.

How much does it cost?

Universal-3.6 Pro Realtime streaming speech-to-text is $0.45 per hour. The LLM Gateway bills per token at each model's rate, and text-to-speech is $15 per 1M characters, billed on the characters of text you send. See the AssemblyAI pricing page for current rates, and the LiveKit Cloud site for LiveKit's pricing. You can start on a free AssemblyAI account.