Agent Context Carryover: more accurate voice agent transcription on LiveKit

If you've built a voice agent, you know the failure mode. The agent asks "What's your email?" and the caller says "j-dot-oconnor at gmail" — and the transcript comes back "j dot o connor at gmail dot com," or worse. Short replies, spelled-out entities, proper nouns, menu items, street names: these are exactly the moments where a caller expects the agent to just get it, and exactly where streaming speech-to-text tends to slip.

The usual fix is to feed the model hints — key terms, custom vocabulary, a running list of nouns you expect to hear. It works, but it's plumbing you have to build and maintain. With Agent Context Carryover on Universal-3.5 Pro Realtime, running on LiveKit, you skip the plumbing entirely. Flip one setting and the model transcribes each user turn while aware of what your agent just said. In this guide we'll build a working voice agent that puts it to the test on some deliberately hard-to-transcribe input.

What Agent Context Carryover actually does

Context carryover lets the speech-to-text model stay aware of the conversation while it transcribes new input. Universal-3.5 Pro Realtime preserves both sides of the dialogue:

• Agent context — your assistant's most recent spoken reply.
• User context — prior finalized user turns from the session.

That bidirectional awareness is what moves the needle on accuracy, especially for short answers and spelled-out entities. When your agent asks "What's your email address?", the model can anticipate the shape of the answer and format "user@assemblyai.com" correctly instead of producing "user at assemblyai dot com."

Here's the part that matters for how you build: on LiveKit, the AssemblyAI plugin feeds that agent context in automatically. There's no extra wiring to capture the agent's last message and shuttle it back to the transcriber. You enable one parameter and the plugin does the rest — no key-term lists to predefine, no context relay to maintain. As of this release, AssemblyAI is the only speech-to-text provider offering this kind of drag-and-drop context carryover.

What we'll build

We'll stand up a LiveKit voice agent from the official Python starter, wire in AssemblyAI for speech-to-text, enable Agent Context Carryover, and then stress-test it as a restaurant takeout line — a fictional spot called Le Périgord Fusion whose menu is a minefield of proper nouns: escargots à la bourguignonne, khachapuri adjaruli, bánh xèo, gnocchi alla sorrentina, mille-feuille, kouign-amann, yuzu kombucha. If the model can order all of that without a single custom vocabulary entry, it can handle your product names too.

Prerequisites

• Python 3.10+ and uv installed
• A free AssemblyAI API key
• A LiveKit Cloud project (for LIVEKIT_URL, API key, and secret)
• livekit-agents 1.6.6 or newer (needed for automatic agent-context forwarding)

Step 1 — Spin up the LiveKit Python starter

LiveKit publishes a starter project that scaffolds a complete Python agent — a voice pipeline on LiveKit Inference (Gemma for the LLM, Cartesia for TTS), the LiveKit turn detector, background noise cancellation, and an eval suite. The quickest way to create one is with the LiveKit CLI:

lk cloud auth
lk agent init my-agent --template agent-starter-python

The CLI clones the template and writes your LiveKit credentials into .env.local. Prefer to do it by hand? Clone the repo, install dependencies with uv, and copy the env template yourself:

git clone https://github.com/livekit-examples/agent-starter-python
cd agent-starter-python
uv sync
cp .env.example .env.local

Step 2 — Install the AssemblyAI plugin for LiveKit

Add the AssemblyAI plugin to the project:

uv add "livekit-agents[assemblyai]"

This gives you the assemblyai plugin namespace you'll use to configure speech-to-text in the agent.

Step 3 — Configure Universal-3.5 Pro and enable context carryover

Open the agent entrypoint at src/agent.py. Out of the box, the starter builds its AgentSession with Deepgram as the speech-to-text provider through LiveKit Inference:

# Before — the starter's default STT line
session = AgentSession(
    stt=inference.STT(model="deepgram/nova-3", language="multi"),
    # ... llm, tts, turn detection
)

Swap that one line for the AssemblyAI plugin and point it at the universal-3-5-pro model. Add the import at the top of the file — everything else in the session (LLM, TTS, turn detection) stays exactly as it was:

from livekit.plugins import assemblyai

# After — AssemblyAI STT with context carryover
session = AgentSession(
    stt=assemblyai.STT(model="universal-3-5-pro"),
    # ... llm, tts, turn detection unchanged
)

The agent_context and previous_context_n_turns parameters below apply only to universal-3-5-pro. That's the whole integration. On livekit-agents 1.6.6 and newer, AgentSession automatically forwards each assistant reply into the model's agent_context — the carryover happens without another line of code. (If a reply runs past the 1,750-character limit, the plugin truncates it silently and keeps the trailing content, which is usually the question you care about.)

Want the model primed before the caller says anything? Seed it with your agent's opening line:

stt = assemblyai.STT(
    model="universal-3-5-pro",
    agent_context="Hello, how can I help you today?",
)

Automatic forwarding takes over after the first real assistant reply.

Step 4 — Add your API key and run

Drop your AssemblyAI key into .env.local alongside your LiveKit credentials:

ASSEMBLYAI_API_KEY=your_api_key_here
LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

Then start the agent in dev mode:

uv run python src/agent.py dev

Connect to your agent, say hello, and confirm it responds. Now the fun part.

Putting it to the test

With the agent live as our Le Périgord Fusion takeout line, we ordered straight down the menu — appetizer, drink, dessert — using the hardest names on it. The exchange went like this:

Agent: Welcome to Le Périgord Fusion. Would you like to place an order, or may I read through our menu for you?

Caller: Yeah, what do you have for appetizers?

Agent: We have escargots à la bourguignonne for $14.50, khachapuri adjaruli for $13, bánh xèo for $11.75, and gnocchi alla sorrentina for $12.25…

Caller: Let's do escargots à la bourguignonne… and a yuzu kombucha… and a kouign-amann.

Every item — French, Georgian, Vietnamese, Italian — came back transcribed correctly, and the order summary matched exactly: one escargots à la bourguignonne, one yuzu kombucha, one kouign-amann, subtotal $29.25. No custom vocabulary. No key-term list. The only thing we turned on was Agent Context Carryover, and because the model could see what the agent had just recited, it knew which unusual words to expect when the caller echoed them back.

The build actually splits the call across two agents — an order-taker that runs the menu, and a checkout host it hands off to for delivery and payment. That handoff is where the next hard case showed up: the caller read out a full delivery address, "1214 Madison Avenue, New York, 10012," and a payment method, and the checkout host read both back verbatim in its confirmation. Street addresses and spoken numbers are exactly the kind of input that trips up streaming transcription, and here the model's awareness of what the agent had just asked ("Please provide your full delivery address") is what kept the digits and the street name intact.

Tuning it: the parameters worth knowing

Automatic forwarding covers most builds, but two parameters give you finer control when you need it.

A couple of rules to keep in mind: an explicit agent_context over 1,750 characters raises a ValueError (only automatic forwarding truncates silently), and previous_context_n_turns is fixed at connection time.

If you're on a version older than 1.6.6, or you want to manage the agent context yourself, disable automatic forwarding and push updates on each assistant turn:

from livekit.agents import AgentSession, ConversationItemAddedEvent
from livekit.plugins import assemblyai

AGENT_CONTEXT_MAX_CHARS = 1750

session = AgentSession(
    stt=assemblyai.STT(model="universal-3-5-pro"),
    stt_context_options={"forward_chat_context": False},
)

@session.on("conversation_item_added")
def _on_conversation_item_added(ev: ConversationItemAddedEvent) -> None:
    if ev.item.type != "message" or ev.item.role != "assistant":
        return

    agent_stt = session.stt
    if not isinstance(agent_stt, assemblyai.STT):
        return

    spoken = ev.item.text_content
    if not spoken:
        return

    agent_stt.update_options(agent_context=spoken[-AGENT_CONTEXT_MAX_CHARS:])

Why this matters for production voice agents

Transcription accuracy is the foundation every downstream step in a voice agent stands on — intent detection, tool calls, order summaries, CRM writes. A misheard proper noun early in a call cascades into the wrong action later. Agent Context Carryover attacks that at the source, and it does it without the maintenance burden of key-term prompting or a hand-rolled context relay. You get the accuracy gain on proper nouns and key terms, and you get it by flipping a switch.

LiveKit sees the same thing from the infrastructure side:

"We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms."

— David Zhao, Co-founder at LiveKit

Try it on your own agent

If you're already building on LiveKit, Agent Context Carryover is available today — turn it on and hear the difference on your hardest inputs. Grab a free API key, follow the four steps above, and point it at the proper nouns your callers actually say. For the full parameter reference, see the Context Carryover docs, and if you're new to this stack, start with our guide to building a voice agent with LiveKit.

Frequently asked questions

What is Agent Context Carryover?

Agent Context Carryover is a feature of AssemblyAI's Universal-3.5 Pro Realtime model that lets the speech-to-text model transcribe each user turn while aware of the conversation so far — specifically the agent's most recent reply and prior finalized user turns. That context improves accuracy on short replies, proper nouns, and spelled-out entities like emails.

How is it different from key-term prompting or custom vocabulary?

Key-term prompting requires you to predefine the words you expect and maintain that list. Context carryover works from the live conversation instead, so on LiveKit it needs no predefined terms and no extra plumbing — the plugin forwards the agent's context automatically.

Do I need to write code to forward the agent's context on LiveKit?

No. On livekit-agents 1.6.6 and newer, AgentSession forwards each assistant reply into the model's agent_context automatically. You only write forwarding code if you're on an older version or want to manage the context manually.

Which model supports it?

The agent_context and previous_context_n_turns parameters apply only to universal-3-5-pro (Universal-3.5 Pro Realtime).

Is there a limit on how much context I can pass?

Yes. agent_context accepts up to 1,750 characters. Setting an explicit value over that limit raises a ValueError; automatic forwarding instead truncates silently, keeping the trailing content. previous_context_n_turns ranges from 0 to 100 (default 5) and is set at connection time only.

Does it work with speech-to-text providers other than AssemblyAI?

As of this release, AssemblyAI is the only speech-to-text provider offering drag-and-drop context carryover on LiveKit.

‍