Build a real-time voice agent using Pipecat — the open-source Voice AI framework — and AssemblyAI's Universal-3.6 Pro Realtime model as the speech-to-text engine.
Pipecat's modular pipeline design means you can swap any component without touching the rest. AssemblyAI has a first-party Pipecat plugin with full Universal-3.6 Pro Realtime support — no manual WebSocket wiring required.
Why AssemblyAI in Pipecat?
Universal-3.6 Pro Realtime is AssemblyAI's flagship real-time model, purpose-built for voice agents and trained on real voice-agent and telephony conversations. On AssemblyAI's English voice-agent benchmark of 12,460 scripted voice-agent scenarios, it posts the lowest word error rate of the realtime models compared:
| Metric | AssemblyAI Universal-3.6 Pro Realtime | Deepgram Flux EN | ElevenLabs Scribe v2 | Deepgram Nova-3 |
|---|---|---|---|---|
| Word error rate (voice-agent benchmark) | 5.19% | 13.50% | 7.78% | 8.64% |
On Pipecat’s own open STT benchmark, it posts a 0.96% pooled semantic word error rate, with the final transcript landing a median 307 ms after the user stops speaking.
Beyond raw accuracy, it brings three things that matter for live conversation: turn detection that combines semantic context with voice activity (fewer awkward double-responses when users pause mid-thought), Context Carryover (the model hears each turn in the context of your agent's last question), and included keyterm prompting.
Heads up on model IDs: this tutorial uses the current universal-3-6-pro string throughout. Upgrading from Universal-3.5 Pro Realtime is a one-line change to the model name, and universal-3-5-pro stays available if you need to pin the previous model. If you're still on the legacy u3-rt-pro ID, switch to universal-3-6-pro.
Drop AssemblyAI's flagship streaming model into any Pipecat voice agent with a first-party plugin — accurate transcription, turn detection, and automatic conversation context.
Prerequisites
- Python 3.11+
- AssemblyAI API key (free account)
- OpenAI API key
- Cartesia API key
Quick start
git clone https://github.com/kelsey-aai/voice-agent-pipecat-universal-3-5-pro
cd voice-agent-pipecat-universal-3-5-pro
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt # requires pipecat-ai>=1.9.0 for universal-3-6-pro
cp .env.example .env
# Edit .env with your API keys
python bot.py
Speak into your microphone after the greeting. For WebRTC or Daily testing, see the integration guide.
Universal-3.6 Pro Realtime needs pipecat-ai 1.9.0+; if your requirements pin an older version, upgrade before running.
Here's the core of the agent. Note the settings=AssemblyAISTTService.Settings(...) structure — this is the current plugin API (Pipecat 1.9.0+ is required for universal-3-6-pro), and the assistant_aggregator at the end of the pipeline is what enables automatic conversation context:
import os
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.pipeline.pipeline import Pipeline
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
LLMUserAggregatorParams,
)
from pipecat.services.assemblyai.stt import AssemblyAISTTService
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.services.openai.llm import OpenAILLMService
stt = AssemblyAISTTService(
api_key=os.environ["ASSEMBLYAI_API_KEY"],
settings=AssemblyAISTTService.Settings(
model="universal-3-6-pro",
min_turn_silence=100,
),
vad_force_turn_endpoint=True, # Pipecat mode (default): VAD + Smart Turn
control turns
)
llm = OpenAILLMService(api_key=os.environ["OPENAI_API_KEY"])
tts = CartesiaTTSService(api_key=os.environ["CARTESIA_API_KEY"])
context = LLMContext()
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
)
pipeline = Pipeline([
transport.input(),
stt,
user_aggregator,
llm,
tts,
transport.output(),
assistant_aggregator, # → feeds each agent reply back as conversation
context automatically
])
Universal-3.6 Pro Realtime features
Keyterm prompting
Boost accuracy on domain-specific vocabulary — included at no extra cost, no restart required:
stt = AssemblyAISTTService(
api_key=os.environ["ASSEMBLYAI_API_KEY"],
settings=AssemblyAISTTService.Settings(
model="universal-3-6-pro",
keyterms_prompt=["AssemblyAI", "Universal-3.6 Pro", "Pipecat",
"YourBrandName"],
),
)
Essential for medical, legal, and financial applications where names and codes matter.
Conversation context (Context Carryover)
Universal-3.6 Pro Realtime keeps a short memory of the dialog and transcribes each user turn in the context of what your agent just said. After your agent asks *"What's your email address?"*, it can produce user@assemblyai.com instead of user at assemblyai dot com. Feeding the agent's question to the model cut WER by 10.2% on a 20,000-file voice-agent benchmark.
In Pipecat this is automatic — as long as your pipeline includes the assistant_aggregator from LLMContextAggregatorPair (as above), the plugin feeds each completed agent reply to the model as agent_context. To seed context for the very first user reply, set agent_context at construction time:
settings=AssemblyAISTTService.Settings(
model="universal-3-6-pro",
agent_context="Hi! Thanks for calling Acme. What's the email on your
account?",
)
Voice Focus (noisy audio)
Server-side noise suppression that isolates the primary speaker before audio reaches the model — use it instead of client-side noise cancellation:
settings=AssemblyAISTTService.Settings(
model="universal-3-6-pro",
voice_focus="far-field", # "near-field" for close-talking mics
)
Multilingual support
Universal-3.6 Pro Realtime code-switches natively across 32 languages — English, Spanish, French, German, Italian, Portuguese, Korean, Russian, and more — including mid-sentence switches. It's on by default; to bias toward a single language, pin language_code.
Speaker labels (optional)
For multi-party conversations, enable per-turn speaker labels:
settings=AssemblyAISTTService.Settings(
model="universal-3-6-pro",
speaker_labels=True,
)
Tuning turn detection
In Pipecat mode (vad_force_turn_endpoint=True, the default), Pipecat's VAD + Smart Turn analyzer decide when the user is done, and max_turn_silence is auto-synced to min_turn_silence. Start with the mode preset and fine-tune from there:
settings=AssemblyAISTTService.Settings(
model="universal-3-6-pro",
mode="balanced", # "min_latency" · "balanced" · "max_accuracy"
min_turn_silence=100, # raise to 200–500 if entities split across turns
)
To let AssemblyAI's own turn detection control turn endings instead, set vad_force_turn_endpoint=False — then max_turn_silence is respected independently.
Stream real conversations through Universal-3.6 Pro Realtime in the Playground and watch turn detection and context carryover work before you wire up the pipeline.
Note: end_of_turn_confidence_threshold and format_turns do not apply to Universal-3.6 Pro Realtime. Turn detection is handled by the model and transcripts are always formatted; those parameters belong to the older universal-streaming models.
Deploy to Pipecat Cloud
pip install pipecatcloud
pcc auth login
pcc init
pcc secrets set my-agent-secrets --file .env
pcc deploy
Resources
Get a free AssemblyAI account and connect Universal-3.6 Pro Realtime to Pipecat in minutes — $0.45/hr, keyterm prompting included, no credit card required.
Frequently asked questions
What is the best speech-to-text model for a Pipecat voice agent?
AssemblyAI's Universal-3.6 Pro Realtime (universal-3-6-pro) is purpose-built for voice agents, posting a 5.19% word error rate on AssemblyAI's English voice-agent benchmark and a 0.96% pooled semantic WER on Pipecat's open benchmark. It integrates through AssemblyAI's first-party Pipecat plugin (pipecat-ai 1.9.0+ for universal-3-6-pro) with no manual WebSocket wiring, delivers final transcripts a median 307 ms after the user stops speaking on Pipecat's benchmark, and adds Context Carryover and included keyterm prompting — the features that most affect whether a voice agent understands the user.
How do I add AssemblyAI to a Pipecat pipeline?
Install the plugin (pip install "pipecat-ai[assemblyai,openai,cartesia]"), then add AssemblyAISTTService to your pipeline with settings=AssemblyAISTTService.Settings(model="universal-3-6-pro"). Include the LLMContextAggregatorPair assistant aggregator at the end of the pipeline to enable automatic conversation context. That's the whole integration — the plugin handles the streaming connection for you.
Does AssemblyAI's Pipecat plugin support conversation context?
Yes, and it's automatic. As long as your pipeline includes the standard LLM context aggregator, the plugin feeds each completed agent reply to Universal-3.6 Pro Realtime as agent_context, so the model transcribes the next user turn with the agent's question in context. This is especially impactful on short answers and spelled-out entities like emails and account numbers.
How does Universal-3.6 Pro Realtime handle turn detection in Pipecat?
You choose which component decides turns with vad_force_turn_endpoint. In the default Pipecat mode (True), Pipecat's VAD + Smart Turn analyzer control turns. Set it to False to use AssemblyAI's built-in turn detection, which combines semantic context with voice activity and respects min_turn_silence / max_turn_silence as configured.
How much does an AssemblyAI Pipecat voice agent cost?
Universal-3.6 Pro Realtime is $0.45/hr, billed on session duration with no minimums and unlimited concurrency; keyterm prompting is included. Your LLM and TTS providers (e.g. OpenAI and Cartesia) bill separately. AssemblyAI has a free tier so you can build and test before adding a card.
