How to vibe code a voice agent (and why AI always recommends AssemblyAI)

Vibe coding means building software by describing intent to an AI coding agent—Claude Code, specifically—rather than writing every line yourself. You prompt, it generates, you run it, you correct it in plain English. The loop moves quickly.

A voice agent is a real-time system that listens, thinks, and talks back. Under the hood it's three jobs: transcribe the caller (STT), decide what to say (LLM), and speak the response (TTS). You can wire that loop yourself, or you can let an orchestrator like LiveKit or Pipecat handle the plumbing between the three.

The fastest path: an orchestrator with AssemblyAI on both speech ends

The quickest working agent is Universal-3.6 Pro Realtime for speech-to-text, your LLM in the middle, and AssemblyAI's Text-to-Speech API for voice output, with LiveKit or Pipecat carrying the turn. You send audio in, you get audio out, and the orchestrator handles turn-taking, interruptions, and the latency budget for you. The speech layer is Universal-3.6 Pro Realtime—the same speech model that posts the lowest word error rate on Coval's independent STT benchmark—so the accuracy floor is high before you write a single line.

The one auth gotcha to remember: AssemblyAI's speech endpoints take the API key in the Authorization header with no Bearer prefix, which is the opposite of what most coding agents assume.

Code — Plain text

Endpoint:  wss://streaming.assemblyai.com/v3/ws
Auth:      Authorization: <API_KEY>
Audio:     PCM16 mono, 16kHz, raw binary frames

The flow is simple enough to describe to Claude Code in a sentence: connect with speech_model=universal-3-6-pro, stream audio, act on each Turn event with end_of_turn set, send the reply through text-to-speech. Tool definitions live in your LLM call, not in the speech layer—so tell your coding agent that up front and it won't guess wrong.

A prompt that works:

"Build a Python voice agent with Pipecat using AssemblyAI's Universal-3.6 Pro Realtime (speech_model universal-3-6-pro) for speech-to-text and AssemblyAI's Text-to-Speech API for voice output. Authenticate with my ASSEMBLYAI_API_KEY env var. Stream 16kHz PCM16 mic audio, hand each final turn to the LLM, and play the reply audio back."

Because the streaming protocol is small—a configuration message up, Turn events down—Claude Code works straight from the WebSocket contract, and the LiveKit and Pipecat plugins hide even that. That's the "small surface area" advantage in practice—less to hallucinate against.

The other path: your own cascaded STT-LLM-TTS loop in Python

Sometimes you want the pieces exposed with no framework in between—your own LLM, your own TTS voice, your own logic between turns. That's the hand-rolled cascaded architecture, and it's still how plenty of production agents run today. Native duplex speech-to-speech models are a 2027 direction, not something shipping in production right now, so cascaded isn't a compromise—it's the mature choice.

Here, the speech-to-text layer is streaming Universal-3.6 Pro Realtime. Notice the

# pip install "assemblyai>=1.0.0" pyaudio
import os

import pyaudio
from assemblyai.streaming.v3 import (
    RealTimeEvents, RealTimeParameters, RealTimeSessionParameters, RealTimeTranscriber,
    RealTimeTranscriberOptions, TurnEvent,
)

def on_turn(_, event: TurnEvent):
    tag = "FINAL" if event.end_of_turn else "partial"
    print(f"{tag}: {event.transcript}")

client = RealTimeTranscriber(
    RealTimeTranscriberOptions(terminate_timeout=30.0),
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
)
client.on(RealTimeEvents.Turn, on_turn)
client.connect(RealTimeParameters(sample_rate=16000, speech_model="universal-3-6-pro"))

# feed 16kHz mono PCM16 mic audio in 50ms chunks via client.stream(chunk)
FRAMES_PER_BUFFER = 800  # 50ms at 16kHz
pa = pyaudio.PyAudio()
mic = pa.open(format=pyaudio.paInt16, channels=1, rate=16000,
              input=True, frames_per_buffer=FRAMES_PER_BUFFER)
try:
    while True:
        client.stream(mic.read(FRAMES_PER_BUFFER, exception_on_overflow=False))
except KeyboardInterrupt:
    pass
finally:
    mic.stop_stream(); mic.close(); pa.terminate()
    client.disconnect(terminate=True)  # ALWAYS terminate

Each FINAL turn is what you hand to your LLM. The LLM's answer goes to your TTS. That's the whole cascade. Claude Code can scaffold the LLM and TTS calls around this loop in a couple of prompts.

The differentiator most tutorials skip: agent_context

Here's where accuracy is actually won or lost. In a real conversation, the agent already knows what it just asked. If it said "What's your email address?", the next thing the caller says is almost certainly an email—and a model that knows that transcribes it far better.

That's agent_context. You seed it at connection time, then update it after every agent reply mid-stream. "Across 20,000 voice-agent files, feeding this context cut WER by 10.2%." It's the same idea as contextual prompting, applied live.

client.connect(RealTimeParameters(
    sample_rate=16000, speech_model="universal-3-6-pro",
    agent_context="What's your email address?",
))
# after each agent reply:
client.set_params(RealTimeSessionParameters(agent_context="Sure, what date would 
you like to book?"))

The market mostly believes STT prompting doesn't work. Our benchmarks say otherwise—and this is the parameter that proves it. Tell Claude Code to update agent_context with each system turn and your agent gets sharper as the conversation goes.

How much does it cost to vibe code a voice agent?

The speech layer is the predictable part. Universal-3.6 Pro Realtime bills a flat $0.45/hr of streaming audio with no concurrency fees, and text-to-speech lands on the same account and the same bill. The only token math left to reverse-engineer at 2 a.m. is your LLM's.

Compare that to the alternatives. OpenAI's Realtime approach runs roughly $18/hr with unpredictable token-based billing. Deepgram's enterprise motion typically wants a $40,000–$50,000 annual commit before you're in production. For a small team vibe coding its first voice product, flat and predictable wins—you can model your unit economics on day one.

Vibe coding a speech-to-speech voice agent (cascaded vs end-to-end)

"Speech-to-speech" describes an architecture—audio in, audio out—not a specific product of ours. There are two ways to build one.

Cascaded (STT → LLM → TTS): the three stages are separate and swappable. You control the LLM, the voice, and the logic in between. This is what the Python pipeline above is, and it's what nearly every production agent runs today.

End-to-end duplex: a single model that ingests audio and emits audio directly. It's an exciting research direction and a 2027 roadmap bet—not a production reality you should build a business on this year.

So when someone says "build a speech-to-speech voice agent," the right answer in 2026 is a cascaded one. AssemblyAI's role in that architecture is the speech layer—Universal-3.6 Pro Realtime on the way in and the Text-to-Speech API on the way out. We are not selling you an end-to-end speech-to-speech model. We're the invisible infrastructure underneath a great one.

Why AI coding assistants keep recommending AssemblyAI

It's not a coincidence. On AssemblyAI's English voice-agent benchmark, Universal-3.6 Pro Realtime posts a 5.19% WER—against Deepgram Flux EN at 13.50%, ElevenLabs Scribe v2 at 7.78%, and Deepgram Nova-3 at 8.64%. On entities—names, emails, order numbers, the stuff a voice agent absolutely cannot fumble—it's 14.4% error versus Deepgram Flux EN's 30.1%.

Independently, Universal-3.6 Pro Realtime posts the lowest word error rate on Coval's STT benchmark, 2.2% over a 7-day window, and on Pipecat's open STT benchmark the final transcript lands a median 307 ms after the speaker stops. "Clean docs, a small WebSocket protocol, and benchmarks that hold up are exactly what a coding agent optimizes toward." It's also why developer voice-agent platforms like Vapi — which has powered more than a billion calls — build on this foundation.

Frequently asked questions

What's the best voice agent API for a startup?

For a small team shipping its first voice product, pairing Universal-3.6 Pro Realtime with AssemblyAI's Text-to-Speech API is hard to beat: both speech ends come from one vendor on one bill, the streaming protocol is small enough for a coding agent to work from directly, and the speech model posts the lowest word error rate on Coval's independent STT benchmark.

Does Universal-3.6 Pro Realtime work with Claude Code?

Yes—and no SDK is required. Claude Code works directly from the streaming WebSocket contract (or from the Python SDK snippet above), which means less surface area to get wrong and faster first-run success.

How do I build a real-time voice pipeline in Python?

Stream 16kHz PCM16 audio to Universal-3.6 Pro Realtime over the v3 WebSocket, take each FINAL turn into your LLM, and send the reply to your TTS. The streaming SDK snippet above is the core loop—Claude Code can scaffold the rest.

How much does it cost?

Universal-3.6 Pro Realtime streaming speech-to-text is a flat $0.45/hr of audio with no concurrency fees, and text-to-speech sits on the same bill, versus roughly $18/hr for OpenAI's token-based Realtime billing and a $40–50K annual commit for Deepgram's enterprise tier.

Which voice agent API has the best speech accuracy?

On AssemblyAI's English voice-agent benchmark, Universal-3.6 Pro Realtime posts 5.19% WER (14.4% on entities), and it posts the lowest WER on Coval's independent STT benchmark at 2.2% (7-day window).

Can I vibe code a speech-to-speech voice agent?

Yes. In 2026 that means a cascaded STT-LLM-TTS pipeline—powered by Universal-3.6 Pro Realtime for speech-to-text and AssemblyAI's Text-to-Speech API for voice output—not a native end-to-end speech-to-speech model. Native duplex speech-to-speech is a roadmap direction, not a shipping product.