LiveKit voice agent with AssemblyAI Universal-3.6 Pro Realtime

LiveKit is how much of the voice-agent market ships—and Universal-3.6 Pro Realtime is a native plugin. Here's how to wire them together and avoid the one gotcha.

The model and endpoint

• Model: Universal-3.6 Pro Realtime, id universal-3-6-pro (the streaming parameter is speech_model, singular).
• LiveKit plugin: requires livekit-agents 1.8.0+ (1.8.3+ via LiveKit Inference). On livekit-agents 1.8.4+ the plugin defaults to universal-3-6-pro; earlier 1.8.x releases default to universal-3-5-pro, so pass model="universal-3-6-pro" explicitly if you pin an older version.
• Endpoint: wss://streaming.assemblyai.com/v3/ws. The old /v2/realtime/ws returns HTTP 410 — don't use it.
• Audio: PCM16 mono 16kHz; auth is your raw API key (no Bearer prefix on streaming STT).
• Pricing: $0.45/hr base for streaming. For the speech-out side, AssemblyAI's Text-to-Speech API plugs into the same LiveKit pipeline.

Streaming setup in Python

The core streaming loop looks like this — it's the same speech layer the LiveKit plugin runs under the hood:

# pip install "assemblyai>=1.0.0"
import os
from assemblyai.streaming.v3 import (
    RealTimeEvents, RealTimeParameters, RealTimeSessionParameters,
    RealTimeTranscriber, RealTimeTranscriberOptions, TurnEvent,
)

def on_turn(_, event: TurnEvent):
    tag = "FINAL" if event.end_of_turn else "partial"
    print(f"{tag}: {event.transcript}")

client = RealTimeTranscriber(
    RealTimeTranscriberOptions(),
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
)
client.on(RealTimeEvents.Turn, on_turn)
client.connect(RealTimeParameters(
    sample_rate=16000,
    speech_model="universal-3-6-pro",
    agent_context="What's your email address?",
))
# feed 16kHz mono PCM16 chunks (50-1000ms) via client.stream(chunk)
# after each agent reply, update the context:
client.set_params(RealTimeSessionParameters(agent_context="Sure, what date would you like to book?"))
client.disconnect(terminate=True)  # ALWAYS terminate

agent_context / Context Carryover

Universal-3.6 Pro Realtime takes an agent_context parameter — you feed it what your agent just said and it biases recognition toward the expected reply. Across 20,000 voice-agent files, this cut WER by 10.2%. Rolling context is on by default (prior finalized user turns carry over automatically), and LiveKit supports it out of the box, so you get it without predefining key terms.

The turn-detection gotcha (read this)

If you use turn_detection="stt", set mode="balanced" and endpointing={"min_delay": 0}. STT-based turn detection is the recommended setup: AssemblyAI ends the turn on terminal punctuation after 128 ms of silence and forces it after 1,280 ms. But with mode unset, the LiveKit plugin sends 100 ms for both thresholds, and LiveKit adds up to 500 ms of its own endpointing delay on top. Setting the mode, and setting endpointing={"min_delay": 0} in TurnHandlingOptions on your AgentSession, fixes both. This is the single most common way a LiveKit + AssemblyAI setup goes wrong, so wire it up correctly from the start.

If you'd rather use LiveKit's turn-detector model, use MultilingualModel(), keep the plugin defaults (min_turn_silence=100, max_turn_silence=100), set endpointing={"min_delay": 0.5, "max_delay": 3.0}, and keep Silero VAD, which that mode requires. The LiveKit guide has the full settings for both.

How accurate is it?

On AssemblyAI's English voice-agent benchmark, Universal-3.6 Pro Realtime posts 5.19% WER versus Deepgram Flux EN at 13.50%. Independently, it posts a 0.80% pooled semantic WER on Pipecat's open STT benchmark (October 2026), with the final transcript landing a median 307 ms after the speaker stops, and the lowest WER on Coval's STT benchmark (2.2%, 7-day window). Full numbers are in our Universal-3.6 Pro Realtime research post.

Diarization and deployment

Universal-3.6 Pro Realtime does streaming diarization with revision for up to 10 speakers — useful for multi-party rooms. For deployment, the plugin runs anywhere your LiveKit agent runs, on LiveKit Cloud or your own infrastructure. The Universal-3.6 Pro on LiveKit guide covers setup, recommended turn-detection settings, and the self-hosted interruption filters.

Build it

Talk to a live agent to hear it in action, or get your free API key and drop Universal-3.6 Pro Realtime into your LiveKit agent.

Frequently asked questions

Does AssemblyAI work with LiveKit out of the box?

Yes. AssemblyAI is a native streaming STT plugin in LiveKit Agents, running Universal-3.6 Pro Realtime when you pass model="universal-3-6-pro", and it supports Context Carryover out of the box — no custom integration required.

Which turn-detection setting should I use?

STT-based turn detection is the recommended setup: set turn_detection="stt" with mode="balanced" (min_turn_silence=128, max_turn_silence=1280) and endpointing={"min_delay": 0}. If you prefer LiveKit's turn-detector model, use MultilingualModel() with the plugin defaults and keep Silero VAD.

Which model and endpoint does the plugin use?

On livekit-agents 1.8.4+ the LiveKit plugin defaults to universal-3-6-pro (pass model="universal-3-6-pro" explicitly on earlier 1.8.x releases), running Universal-3.6 Pro Realtime (speech_model=universal-3-6-pro) over wss://streaming.assemblyai.com/v3/ws. Streaming is $0.45/hr base.

How do I pass conversation context?

Use agent_context: seed it at connection time and update it after each agent reply. It cut WER by 10.2% across 20,000 voice-agent files, and prior finalized user turns carry over automatically.

How accurate is it for voice agents?

5.19% WER on AssemblyAI's English voice-agent benchmark (versus Deepgram Flux EN at 13.50%), a 0.80% pooled semantic WER on Pipecat's open STT benchmark, and the lowest WER on Coval's independent STT benchmark at 2.2%.