Build your first AI voice agent: 3 step-by-step examples

If you've tried to build a voice agent in the last year, you already know the hard part isn't the idea — it's the plumbing. You have to wire up speech-to-text, feed it into an LLM, convert the response back to speech, and somehow make all of that feel like a real conversation with sub-second latency, clean turn-taking, and graceful handling of interruptions. Get one piece wrong and the whole thing feels robotic.

Here's the good news: there are now three clear ways to build an AI voice agent, and the fastest one didn't exist when most tutorials were written. You can take the managed-speech path: AssemblyAI's Universal-3.6 Pro Realtime for speech-to-text and Text-to-Speech API for voice output, with a coding agent scaffolding the loop around your LLM. You can use an orchestrator like Vapi, LiveKit, or Pipecat that stitches best-of-breed components together for you. Or you can build a fully custom pipeline from scratch, choosing every model yourself.

This guide walks through all three, with working examples for each. I'll be honest about the tradeoffs — because the right choice depends on how much control you actually need and how fast you want a talking agent in front of users. Let's start with what every voice agent is made of.

The core components of any voice agent

No matter which path you pick, an AI voice agent is the same four-part loop under the hood:

• Speech-to-text (STT). Streaming transcription that turns the caller's audio into text in real time. This is the foundation — if the transcript is wrong, the LLM reasons over garbage. Accuracy on messy, real-world audio (names, emails, phone numbers, order IDs) matters more here than in any other application.
• A large language model (LLM). The brain that interprets intent, follows your system prompt, calls tools or functions, and decides what to say next.
• Text-to-speech (TTS). Converts the model's response back into natural-sounding audio.
• Turn detection and interruption handling. The part everyone underestimates. Your agent needs to know when the user has finished speaking, when to jump in, and how to stop talking the instant someone interrupts it. Bad turn-taking is the number one reason a demo feels off, and it's consistently flagged as the hardest engineering problem in voice.

Every approach below solves these four problems — the difference is how much of the wiring you do yourself. If you want a deeper primer on the category, our overview of AI voice agents covers the architecture in more detail.

The three ways to build an AI voice agent

Before we get into code, here's the decision at a glance. The taxonomy is simple: how much of the stack do you want to own?

Teams with specific latency, privacy, or on-prem requirements

Most teams building their first agent overestimate how much control they need. So we'll start with the path that gets you talking to a working agent the fastest, then work outward toward more control.

Option 1: The managed-speech path — AssemblyAI's speech-to-text and text-to-speech APIs

This is the path I'd recommend to almost anyone building their first voice agent, and it's the one the older versions of this guide couldn't include because the pieces hadn't shipped yet. You take both speech ends from AssemblyAI — Universal-3.6 Pro Realtime streams the caller's audio into text with turn detection built in, and the Text-to-Speech API turns your LLM's reply back into audio — and let a coding agent (or LiveKit or Pipecat) write the loop between them. Two vendors instead of four, and no speech streams from different providers to reconcile.

Here's what that buys you in practice:

• One speech vendor instead of two. Transcription and speech generation come from the same account, docs, and invoice, with your LLM as the only other call in the loop — so you spend your time on your agent's behavior, not on plumbing.
• Predictable speech pricing. Streaming speech-to-text is $0.45/hour, billed per second, with no concurrency fees, and text-to-speech is $15 per 1M characters (see pricing). You can model the speech line of your unit economics on day one.
• Turn detection built in. Universal-3.6 Pro Realtime handles end-of-turn detection on the speech side, so the hardest part of the loop arrives with the transcript instead of being something you tune from scratch.
• A voice built for voice agents. AssemblyAI-TTS-1 is trained on millions of hours of audio, including real conversational data. It delivers frontier-level quality and runs faster than other production voice agent TTS models.
• Universal-3.6 Pro Realtime accuracy. The STT side is our flagship realtime model, and it's the reason this path doesn't sacrifice accuracy for convenience (more on the benchmark below).

There's one more thing that changes how it feels to build with. Both APIs are standard, documented interfaces that coding agents like Claude Code handle well. You can describe the agent you want in plain language, point the coding agent at the voice agent docs, and have a working prototype in an afternoon. The fewer moving parts, the fewer places to get stuck.

The accuracy story is what makes this more than a convenience play. On AssemblyAI's English voice-agent benchmark — 12,460 scripted voice-agent scenarios, where lower word error rate is better — Universal-3.6 Pro Realtime posts a 5.19% WER, ahead of ElevenLabs Scribe v2 at 7.78%, Deepgram Nova-3 at 8.64%, and Deepgram Flux EN at 13.50%. The lead carries over to the things voice agents actually trip over: entity error rate lands at 14.4% versus 18.5% for Scribe v2 and 30.1% for Flux EN. That's the difference between an agent that hears "order number A-1-5-J" correctly and one that guesses. You can see the full methodology in the Universal-3.6 Pro Realtime research write-up.

To get started, spin up a key and follow the Python voice agent tutorial, or the LiveKit guide if you'd rather let a framework own the loop. The docs have a copy-paste quickstart. This is the "Example 0" I'd reach for first — and for most receptionist, appointment-setting, lead-qualification, and food-ordering agents, it's also the last stop.

Option 2: The orchestrator path — Vapi, LiveKit, and Pipecat

Sometimes you want managed transport and turn-taking but insist on choosing each model yourself — a specific LLM for reasoning, a particular voice for TTS, and best-in-class STT feeding both. That's what orchestrators are for. They handle the real-time media plumbing while letting you swap components at each stage. AssemblyAI plugs into these as the speech-to-text layer, and honestly, running our STT inside an orchestrator is a completely valid production setup — the managed-speech and orchestrator paths coexist.

Example: Telnyx + AssemblyAI

Telnyx is the telephony-native path: phone numbers, SIP trunking, and programmable call control, with voice agent orchestration running on the same carrier network. If your agent answers real phone calls rather than browser audio, this is the option where the phone part is built in rather than bolted on.

What you'll need: a Telnyx account with a phone number, a Voice AI assistant configured in Mission Control, and AssemblyAI selected as the transcription model (it's a built-in option, model ID assemblyai/universal-streaming). Core components: Telnyx owns call transport, orchestration, and TTS; AssemblyAI owns STT and end-of-turn detection; bring the LLM of your choice. The win here is that audio never leaves the network carrying the call, so you keep AssemblyAI's accuracy on names and numbers without adding a latency hop between your telephony vendor and your STT vendor.

Example: LiveKit + AssemblyAI

LiveKit gives you more control over the media layer, which is useful if you're building something with custom audio routing, multi-party calls, or your own front end. You wire AssemblyAI's streaming transcription into LiveKit's agent framework as the STT plugin and bring your own LLM and TTS. Retell, LiveKit, and Fireflies were all early adopters of Universal-3.5 Pro Realtime, so you're in good company running our models inside a framework.

What you'll need: a LiveKit project, an AssemblyAI API key, and your LLM plus TTS providers. Core components: LiveKit owns the real-time transport and agent loop; AssemblyAI supplies the transcript; you own reasoning and voice. If you want to understand where these stacks start to strain as you scale, we wrote about exactly that in where voice agent stacks start showing their limits.

Example: Vapi + AssemblyAI

Vapi is a popular orchestrator that manages the call lifecycle, turn detection, and provider routing for you. You configure it declaratively: point the STT step at AssemblyAI's streaming speech-to-text, choose your LLM, and pick a TTS voice.

What you'll need: a Vapi account, an AssemblyAI API key, an LLM provider key, and a TTS voice. Core components: Vapi handles transport and turn-taking; AssemblyAI handles transcription; your chosen LLM and TTS handle reasoning and speech. The win here is that you keep AssemblyAI's real-time accuracy on names and numbers while letting Vapi own the orchestration.

The tradeoff with orchestrators is real: you get flexibility, but you're now responsible for reconciling latency across providers, managing multiple bills, and debugging across component boundaries when something feels laggy. That flexibility is worth it when you genuinely need a specific component — and overkill when you don't.

Option 3: The fully custom path — self-hosted models end to end

If you have hard requirements around on-prem deployment, data residency, or a very specific latency budget, you can build the entire pipeline yourself and own every hop.

Example: DeepSeek R1 (local via Ollama) + AssemblyAI STT + AssemblyAI TTS

This setup runs the LLM locally with Ollama — DeepSeek R1 in this case — for full control over the reasoning layer, uses AssemblyAI for streaming speech-to-text, and AssemblyAI's Text-to-Speech API for voice output. You write the orchestration code yourself: capturing microphone audio, streaming it to STT, detecting end-of-turn, prompting the local model, and piping its response into TTS. The TTS WebSocket fits a local model well: one session per conversation, the model's tokens sent as they're generated, speech that starts once the server holds complete sentences, and a cancel that stops the reply in flight when the caller interrupts. The TTS quickstart has the session code in Python and JavaScript.

What you'll need: Ollama running DeepSeek R1 locally, an AssemblyAI API key (the same key covers speech-to-text and text-to-speech), and comfort managing async audio buffers. Core components: you wire all four — STT, LLM, TTS, and turn detection — and own every buffer and network call between them.

Worth noting on accuracy: because the custom path names each component explicitly, your STT choice is doing a lot of work. On real voice-agent audio, Universal-3.6 Pro Realtime's 5.19% WER and low entity error rate are what keep the transcript clean enough for a local LLM to reason over. The rest of this path is engineering effort — which is the point. You take on days-to-weeks of work in exchange for total control. For most teams, that trade only pays off with a genuine on-prem or compliance constraint.

Which path should you choose?

Here's how I'd decide, quickly:

• Building your first agent, or want it in production fast? Start with AssemblyAI's speech-to-text and text-to-speech APIs and let a coding agent write the loop. Two vendors, $0.45/hour for streaming speech-to-text, $15 per 1M characters for text-to-speech, and you can prototype with Claude Code in an afternoon. This is the right call for the vast majority of data-collection agents — receptionists, appointment setters, lead qualifiers, order takers.
• Need to pick a specific LLM or TTS voice but want managed transport? Use an orchestrator like Vapi, LiveKit, or Pipecat with AssemblyAI as your STT layer.
• Have on-prem, residency, or bespoke latency requirements? Build the custom pipeline and own every component.

A good rule of thumb: pick the shallowest layer that covers your requirements. Every layer of control you add is a layer of infrastructure you now maintain. If you want a data-backed look at what actually separates good agents from frustrating ones, our 2026 insights report is worth a read before you commit to an architecture.

A note on where this is heading

Here's the shift most teams haven't fully internalized yet: the hard part of voice agents is moving away from wiring and toward behavior. A year ago, "building a voice agent" mostly meant integration work — getting three services to talk to each other without adding half a second of latency at every seam. That work is increasingly a solved problem you can buy. What's left is the part that was always the real product: the prompt design, the tool calls, the way your agent recovers when a caller mumbles an address or interrupts mid-sentence.

That's why I'd push almost everyone toward the managed-speech path first, even teams that assume they'll need more control. Not because the other paths are wrong — they're excellent when you have a real reason to reach for them — but because starting with managed infrastructure lets you spend your first week on your agent's actual behavior instead of on plumbing you'll eventually want to hand off anyway. Build the conversation first. You can always drop down a layer when a concrete requirement forces the question. Most teams find it never does.

Frequently asked questions

What's the easiest way to build an AI voice agent?

The managed-speech path. Take Universal-3.6 Pro Realtime for speech-to-text and AssemblyAI's Text-to-Speech API for voice output, bring your LLM, and let a coding agent like Claude Code scaffold the loop. Both speech APIs are standard, documented interfaces, and you can stand up a working agent in an afternoon.

How much does it cost to build a voice agent?

Every path bills the components separately: streaming speech-to-text with Universal-3.6 Pro Realtime is $0.45/hour, billed per second, AssemblyAI's text-to-speech is $15 per 1M characters, and your LLM is metered on top. Total cost varies with your provider choices and traffic. See pricing for current details.

Do I need to know how to code?

You'll need some development ability, but far less than you'd expect. AssemblyAI's streaming speech-to-text is a documented WebSocket and the Text-to-Speech API sits beside it on the same account, which means you can describe the agent you want to a coding agent and have it scaffold the integration. Orchestrator and custom paths require more hands-on engineering, especially around turn detection and latency.

AssemblyAI's speech APIs vs. Vapi vs. LiveKit — which should I use?

Build directly on AssemblyAI's speech-to-text and text-to-speech APIs when you want a production-grade agent fast with the fewest vendors and predictable speech cost. Use an orchestrator like Vapi or LiveKit when you need to choose specific LLM or TTS providers but want managed transport and turn-taking. AssemblyAI runs as the STT layer inside those orchestrators too, so the paths aren't mutually exclusive.

How accurate is the speech recognition for voice agents?

On AssemblyAI's English voice-agent benchmark of 12,460 scripted voice-agent scenarios, Universal-3.6 Pro Realtime posts a 5.19% word error rate — ahead of ElevenLabs Scribe v2 (7.78%), Deepgram Nova-3 (8.64%), and Deepgram Flux EN (13.50%) — with a lower entity error rate on names, phone numbers, and order IDs (14.4% overall). Full methodology is in the research write-up.