Daily.co voice agent with AssemblyAI Universal-3.6 Pro Realtime

This is the bare-metal guide to building a Daily.co voice agent — WebRTC transport from Daily.co, speech-to-text from AssemblyAI's Universal-3.6 Pro Realtime, reasoning from an LLM, and speech from AssemblyAI's Text-to-Speech API — wired together directly, without Pipecat. If you want to see exactly how Daily's audio tracks connect to AssemblyAI's WebSockets, in and out, this shows you every seam.

Daily.co is your transport partner. AssemblyAI is your Voice AI layer. The daily-python SDK gives you raw PCM frames; AssemblyAI's streaming speech-to-text endpoint consumes them directly, and AssemblyAI's text-to-speech returns raw PCM you write straight back into the room.

Architecture: the Daily.co + AssemblyAI pipeline

The full loop runs like this:

Browser/phone → Daily.co WebRTC → daily-python SDK → AssemblyAI WebSocket → OpenAI GPT-4o → AssemblyAI TTS → PCM audio back to the Daily room

Daily handles global audio routing and WebRTC. AssemblyAI turns the audio into text. GPT-4o decides what to say. AssemblyAI's Text-to-Speech API turns that back into speech, which Daily plays into the room. Four services from three vendors, one clean data path.

Why Universal-3.6 Pro Realtime for a Daily voice agent?

On a live call, transcription accuracy is the whole game — if the STT layer mishears a name or a phone number, no amount of clever prompting downstream saves the interaction. Universal-3.6 Pro Realtime posts a 5.19% word error rate on AssemblyAI's English voice-agent benchmark, which measures 12,460 scripted voice-agent scenarios rather than clean read-aloud audio:

The entity error rate is what decides whether your Daily agent captures the callback number correctly. At 14.4% versus Deepgram Flux EN's 30.1%, that's not a rounding difference — it's whether the agent works. On Pipecat's independent open STT benchmark, it also posts a 0.96% pooled semantic WER. See the full methodology on our benchmarks page.

Prerequisites

• Python 3.11+
• An AssemblyAI API key
• A Daily.co API key
• An OpenAI API key
• The AssemblyAI Python SDK, 1.8.2 or newer (pip install -U assemblyai), plus daily-python

The one AssemblyAI key covers both speech-to-text and text-to-speech.

Getting started? Grab a free AssemblyAI API key — new accounts get free API credit to start, no credit card required.

AssemblyAI connection parameters

The connection is a single WebSocket. Note the model ID — universal-3-6-pro — and that streaming auth uses a bare API key in the Authorization header (no Bearer prefix):

import os

SAMPLE_RATE = 16000  # pcm_s16le, 16 kHz mono

AAI_WS_URL = (
    "wss://streaming.assemblyai.com/v3/ws"
    f"?sample_rate={SAMPLE_RATE}"
    "&speech_model=universal-3-6-pro"
    "&format_turns=true"
)

HEADERS = {"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}

How audio flows

The daily-python SDK fires a callback for each participant's audio frames. Forward that PCM straight to the AssemblyAI WebSocket:

import asyncio

def on_audio_data(self, participant_id, audio_data, sample_rate, num_channels):
    if self.aai_ws and not self.aai_ws.closed:
        # audio_data is raw PCM16 — stream directly, no buffering
        asyncio.create_task(self.aai_ws.send(audio_data))

On the receive side, read transcripts as Turn events, pass finals to GPT-4o, and route the reply text to AssemblyAI's Text-to-Speech API:

import json

async def recv_transcripts(ws, on_final):
    async for raw in ws:
        msg = json.loads(raw)
        if msg["type"] == "Turn" and msg["end_of_turn"]:
            await on_final(msg["transcript"])  # -> GPT-4o -> AssemblyAI TTS -> Daily
        elif msg["type"] == "Termination":
            return

The streaming server sends Begin, SpeechStarted, Turn (with end_of_turn false for partials, true for finals), and Termination. Always send {"type":"Terminate"} when the call ends so billing stops cleanly.

Speaking the reply with AssemblyAI text-to-speech

The last seam is speech out. AssemblyAI's Text-to-Speech API delivers frontier-level quality, runs faster and costs less than other production voice agent TTS models, and has unlimited concurrency. Open one AssemblyAI text-to-speech session per call over WebSocket, send each GPT-4o reply as a request, and write the PCM it streams back into a Daily virtual microphone. The same ASSEMBLYAI_API_KEY authenticates it:

import asyncio
import os

import assemblyai as aai
from daily import Daily

# Daily plays whatever you write to this virtual microphone into the room.
# Select it after joining:
#   client.update_inputs({"camera": False, "microphone": {
#       "isEnabled": True, "settings": {"deviceId": "agent-mic"}}})
mic = Daily.create_microphone_device("agent-mic", sample_rate=SAMPLE_RATE, channels=1)

TTS_CONFIG = aai.TTSConfig(
    voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3",  # Audrey
    language="english",
    sample_rate=SAMPLE_RATE,  # 16 kHz to match the mic; the native rate is 24 kHz
    encoding=aai.TTSEncoding.pcm_s16le,
)


async def speak_reply(tts, reply_text):
    # Raw 16-bit mono PCM, chunk by chunk, as soon as it is synthesized
    async for chunk in tts.speak(reply_text):
        await asyncio.to_thread(mic.write_frames, chunk)


async def run_call(handle_turns):
    # One TTS session for the whole call, one request per reply
    async with aai.AsyncTTSSession(
        TTS_CONFIG, api_key=os.environ["ASSEMBLYAI_API_KEY"]
    ) as tts:
        await handle_turns(tts)

A few details matter on a live call:

• Match the sample rate. The session returns raw 16-bit mono PCM with no header. Asking for 16 kHz keeps it in step with the virtual microphone and the rest of the pipeline; the server resamples from its native 24 kHz.
• Handle barge-in. When AssemblyAI sends SpeechStarted while the agent is talking, await tts.cancel() stops the reply in flight and drops the rest of its audio, without closing the session.
• Stream tokens if you want an earlier start. If you stream GPT-4o's tokens, send them with generate(), call flush() at the end, and read audio(flush_id); the server starts speaking once it holds enough complete sentences.
• Keep quiet sessions open. A session closes after two minutes with no traffic either way, so call keep_alive() while the caller does a long stretch of talking.

The text-to-speech quickstart covers the full session API, and Voices lists voices you can drop into voice_id.

Why direct Daily.co instead of Pipecat?

Pipecat is a great orchestration framework — and AssemblyAI ships a drop-in Pipecat plugin, so it's a first-class path. But building the Daily integration by hand is worth doing at least once. You see exactly where audio enters, how turn events map to LLM calls, and where latency actually accumulates. When something breaks in production, you'll know which seam to check because you wired every one of them yourself.

If you'd rather not hand-wire it, the Pipecat route gets you the same Universal-3.6 Pro Realtime accuracy with far less code — see our Pipecat and Agora integration guides for the framework-first approach.

The differentiators worth knowing

• Neural end-of-turn detection. Rather than a fixed silence timer, the model combines semantic context with voice activity to decide when a speaker is genuinely done — the final transcript lands a median 307 ms after the speaker stops on Pipecat's open benchmark. Callers who pause mid-sentence don't get interrupted.
• agent_context. Feeding conversational context to the model cuts WER by 10.2% on agent workloads.
• Fewer hallucinations, better entity capture. Phone numbers, emails, and names — the things a caller actually needs the agent to get right — come through cleanly.

Pricing

Streaming with Universal-3.6 Pro Realtime is $0.45/hr base, billed per second, pay-as-you-go, with no minimums and unlimited concurrency. Text-to-speech is $15 per 1M characters, billed on input characters. (You pay Daily and OpenAI separately in this bring-your-own stack.) Full pricing is on the pricing page.

Why not simplify the stack?

Fair question — and for a lot of teams, the answer is "you should."

This guide wires three vendors together: Daily for transport, AssemblyAI for both speech-to-text and text-to-speech, and OpenAI for reasoning. Putting both speech ends on AssemblyAI already removes one bill and one integration. Every hand-wired seam is still a place for latency to creep in, though.

Two more trims are available. Route the LLM through AssemblyAI's LLM Gateway, and everything except transport runs on one AssemblyAI key. Or let an orchestrator like Pipecat or LiveKit handle turn detection, interruptions and the TTS session instead of hand-wiring them. AssemblyAI is invisible infrastructure: we're not the agent, we're the platform you build it on. If you don't specifically need the bring-your-own control this guide gives you, start there. The Pipecat integration guide covers speech-to-text end to end.

See the stack before you build it. Explore AssemblyAI's voice agent solutions, then pick the path that fits.

Frequently asked questions

How do I play AssemblyAI text-to-speech into a Daily room?

Create a Daily virtual microphone with Daily.create_microphone_device() at the sample rate you ask AssemblyAI for, select it with update_inputs(), and write each chunk from an AssemblyAI TTS session's speak() to it with write_frames(). The audio is raw 16-bit mono PCM, so no decoding step is needed.

How do I stream Daily.co audio to AssemblyAI for real-time transcription?

Use the daily-python SDK's audio callback to grab raw PCM16 frames per participant and forward them to wss://streaming.assemblyai.com/v3/ws with speech_model=universal-3-6-pro. Read transcripts back as Turn events and route finals to your LLM.

Should I build a Daily voice agent directly or use Pipecat?

Build it directly when you want to understand and control every seam — audio in, turn events, LLM calls, AssemblyAI TTS out. Use Pipecat's drop-in AssemblyAI plugin when you'd rather ship faster with the same Universal-3.6 Pro Realtime accuracy and less code.

What latency can I expect from Daily.co plus Universal-3.6 Pro Realtime?

On Pipecat's open STT benchmark, Universal-3.6 Pro Realtime delivers the final transcript a median 307 ms after the speaker stops. Total round-trip latency also depends on Daily's transport and your chosen LLM and TTS, so measure the full pipeline end to end.

How much does it cost to run a Daily.co voice agent on AssemblyAI?

Universal-3.6 Pro Realtime streaming is $0.45/hr base, billed per second. AssemblyAI's Text-to-Speech API is $15 per 1M characters, billed on input characters, on the same bill; see the pricing page for current rates. Daily and OpenAI bill separately.

How many languages does Universal-3.6 Pro Realtime support?

The streaming flagship supports 32 languages with automatic language detection, and async Universal-2 covers 99+ if you only need transcription.

Can I simplify the stack further?

If you don't need bring-your-own control over each component, you probably should. This build already uses AssemblyAI for both speech-to-text and text-to-speech. Route the LLM through AssemblyAI's LLM Gateway so one key covers everything but transport, and let Pipecat or LiveKit orchestrate the loop — fewer bills, fewer integrations, and fewer hand-wired seams.