How to create a phone-based voice agent

A phone call is the hardest place to run a voice agent, and it's also where most of the demand is.

The audio is 8kHz mulaw—a codec designed in the 1970s for human ears, not for speech models. There's no video, no screen, no fallback to a text box. The caller can't see a "thinking" spinner, so every pause reads as a dropped call. And callers on phones do the one thing voice agents are worst at: they read out account numbers, email addresses, and dates of birth, at speed, with pauses in the middle.

Get it right and you have something genuinely useful running 24/7. Get the latency wrong by 400 milliseconds and people hang up.

This guide covers what actually goes into a phone-based voice agent: the components, the latency budget, working code that bridges Twilio to AssemblyAI, and the design decisions that separate a demo from something you'd put a real phone number on.

What is a phone-based voice agent?

A phone-based voice agent is an AI system that conducts a full conversation over the phone network—PSTN or SIP—understanding free-form speech and responding with synthesized audio in real time. No touch-tone menus, no "press 1 for billing," no fixed script tree.

The caller talks the way they'd talk to a person. The agent transcribes it, decides what to do, calls whatever systems it needs, and speaks back.

Common deployments: inbound customer support and triage, appointment scheduling and reminders, order status and reordering, lead qualification, after-hours coverage, and outbound follow-ups. Anything where the conversation is bounded enough to be predictable and frequent enough to be worth automating.

The distinction from an AI voice agent generally is the transport. Everything is harder over a phone line: narrower audio, more packet jitter, higher caller expectations of instant response, and no visual channel to fall back on.

The four components of a phone-based voice agent

Telephony. Twilio, Vonage, a SIP trunk into your own infrastructure, or a managed platform like Vapi that provisions the number and the pipeline for you. This layer owns the phone number, answers the call, and hands you a bidirectional audio stream. Twilio Media Streams is the most common path: it opens a WebSocket to your server and sends base64-encoded audio frames.

Streaming speech-to-text. Converts the caller's audio to text continuously, and—critically—decides when the caller has finished a turn. Native 8kHz mulaw support matters more than it sounds: if your STT provider only takes 16kHz PCM, you're resampling every frame in your bridge, which adds latency and loses information that was already scarce.

An LLM. Decides what to say and which tools to call. Needs to stream tokens and needs to be reliable at function calling, because a phone agent that can't look up an order is a very expensive answering machine.

Text-to-speech. Converts the response to audio and streams it back through the telephony layer in chunks. Must support streaming synthesis and must be interruptible mid-utterance.

You can assemble these yourself, or you can collapse them. More on that below.

Architecture: how the pieces fit together

The shape is a WebSocket bridge with two independent loops.

Twilio opens a WebSocket to your server when the call connects. Audio flows in as JSON messages containing base64 mulaw payloads. Your server decodes those and forwards the raw bytes to AssemblyAI over a second WebSocket, held open for the life of the call.

On the return path, AssemblyAI streams back partial transcripts continuously and a final transcript when it detects end-of-turn. Your server hands that final turn to the LLM, streams the response into TTS, and pushes synthesized audio frames back down the Twilio socket to the caller.

The two loops run concurrently and never block each other. That's the whole architectural requirement, and it's why this is asyncio.gather and not a request/response handler. Audio keeps arriving while you're generating a response, because the caller might interrupt—and handling that interruption correctly is the difference between a natural conversation and a shouting match.

The latency budget

Natural conversation has a turn-taking gap of roughly 200ms between humans. You won't hit that. What you can hit is around 800ms, which most callers experience as attentive rather than laggy. Past 1,500ms, the agent feels broken.

First, every stage has to stream. One non-streaming component blows the whole budget, because a component that waits for complete input before producing output turns your parallel pipeline back into a serial one.

Second, endpointing is the stage people optimize last and should optimize first. It's pure dead air—the caller has stopped talking and nothing is happening yet. A fixed 700ms silence timer, which is what a lot of stacks ship with, spends most of your budget before the LLM has seen a single word.

Universal-3.5 Pro Realtime checks the transcript instead: when the caller pauses, the model re-transcribes the turn and looks for terminal punctuation, and falls back to the max_turn_silence timer only if none appears. The defaults come from the mode preset, at min_turn_silence 128ms and max_turn_silence 1280ms on balanced, and both are tunable per call type.

Choosing the speech-to-text layer

Speech-to-text is the component where phone audio punishes you hardest, and where the errors are unrecoverable. A misheard order number becomes a wrong lookup becomes a confidently wrong answer. The caller doesn't experience that as a transcription problem. They experience it as an agent that isn't listening.

Pooled word error rate is the usual headline number and it undersells the problem, because WER weights every word equally. On a phone call, the words that matter are the ones callers read out digit by digit.

Here's Universal-3.5 Pro Realtime on Pipecat's open STT benchmark, which is built from real voice agent conversations rather than clean read speech. Lower is better.

The phone number row is the one to hold a phone agent to. A 10.41% entity error rate on phone numbers means roughly one in ten callback numbers comes back wrong, and every one of those is a failed callback and an angry follow-up call.

We were searching for the best realtime ASR model for our voice agent pipeline in Fireflies. The new Universal 3.5 Pro speech model from Assembly is best so far in terms of accuracy, latency and language switching.

— Foysal Osmany, Software Engineer at Fireflies

Universal-3.5 Pro Realtime runs at $0.45/hr base, billed on how long the WebSocket is open, with unlimited concurrency and automatic rate-limit scaling. Add-ons stack only if you use them: diarization with revision +$0.12/hr, prompting +$0.05/hr, voice isolation +$0.10/hr. agent_context, rolling conversation memory, and keyterm prompting are included.

Building a phone-based voice agent with Twilio and AssemblyAI

Here's a minimal server. Twilio answers the call, opens a media stream to your endpoint, and you bridge that to AssemblyAI.

First, the TwiML that Twilio requests when a call comes in:

<Response>
  <Connect>
    <Stream url="wss://your-server.ngrok.app/ws" />
  </Connect>
</Response>

<Connect><Stream> is bidirectional—it lets you send audio back to the caller, which is what you need for an agent. <Start><Stream> only forks audio to you one way, which is fine for live transcription but useless here.

Now the bridge:

import asyncio
import base64
import json
import os

import websockets
from fastapi import FastAPI, WebSocket
from fastapi.responses import PlainTextResponse

app = FastAPI()

AAI_API_KEY = os.environ["ASSEMBLYAI_API_KEY"]

# Telephony audio is 8kHz mulaw. Set speech_model explicitly—without it,
# the session rides whatever the account default happens to be.
AAI_WS = (
    "wss://streaming.assemblyai.com/v3/ws"
    "?sample_rate=8000"
    "&encoding=pcm_mulaw"
    "&speech_model=universal-3-5-pro"
    "&mode=balanced"
    "&voice_focus=near-field"
    "&agent_context=Thanks%20for%20calling.%20How%20can%20I%20help%20you%20today%3F"
)

@app.post("/incoming-call")
async def incoming_call():
    twiml = """<Response>
  <Connect>
    <Stream url="wss://your-server.ngrok.app/ws" />
  </Connect>
</Response>"""
    return PlainTextResponse(twiml, media_type="application/xml")


@app.websocket("/ws")
async def media_stream(twilio_ws: WebSocket):
    await twilio_ws.accept()

    async with websockets.connect(
        AAI_WS, additional_headers={"Authorization": AAI_API_KEY}
    ) as aai_ws:

        async def twilio_to_aai():
            """Decode Twilio's base64 mulaw frames, forward raw bytes to
AssemblyAI."""
            async for message in twilio_ws.iter_text():
                event = json.loads(message)
                if event["event"] == "media":
                    await aai_ws.send(base64.b64decode(event["media"]["payload"]))
                elif event["event"] == "stop":
                    await aai_ws.send(json.dumps({"type": "Terminate"}))
                    break

        async def aai_to_logic():
            """Catch finalized turns and hand them to the agent loop."""
            async for message in aai_ws:
                data = json.loads(message)
                if data.get("type") != "Turn":
                    continue
                if data.get("end_of_turn"):
                    caller_said = data["transcript"]
                    if not caller_said.strip():
                        continue

                    reply = await run_llm_turn(caller_said)
                    await speak(reply, twilio_ws)

                    # Tell the speech model what the agent just asked, so the
                    # next reply—a date, an account number, a bare "yes"—
                    # is transcribed with that question in context.
                    await aai_ws.send(json.dumps({
                        "type": "UpdateConfiguration",
                        "agent_context": reply,
                    }))

        await asyncio.gather(twilio_to_aai(), aai_to_logic())

run_llm_turn and speak are yours to fill in—the LLM call and the TTS synthesis that streams audio frames back down the Twilio socket. If you're calling several model providers, the LLM Gateway gives you one API across OpenAI, Anthropic, and Google so swapping models is a parameter change.

Three details in that config worth calling out.

speech_model=universal-3-5-pro. Set it explicitly. Omitting it works, right up until the account default moves underneath you.

agent_context. Seeded at connect time with the greeting, then updated after every agent reply. This is the single highest-leverage parameter for phone agents: passing the agent's own question into the session cut WER by 10.2% across a benchmark of 20,000 voice agent audio files, with the largest gains exactly where phone agents hurt—spelled-out emails, account IDs, and one-word confirmations. Cap is 1,750 characters per value, so send the substantive question, not your whole system prompt.

voice_focus=near-field. Callers are on handsets and headsets. Near-field isolation suppresses the background speech and road noise before it reaches the transcription model.

The shortcut: point Twilio at a managed agent over SIP

The bridge above is maybe 80 lines and it's the version you want if you need to control every component independently. If you don't, there's a shorter path.

Point a Twilio number you already own at AssemblyAI over a SIP trunk and attach a Voice Agent API agent. Twilio passes inbound calls to AssemblyAI directly—no media server, no audio bridge, no webhook of your own to keep running. STT, LLM, and TTS come through one connection at a flat $4.50/hr, running on Universal-3.5 Pro Realtime for speech, at around 1 second end-to-end.

There's a third option in between: managed voice agent platforms like Vapi, which provision the number, the pipeline, and a dashboard on top, and let you select AssemblyAI as the speech layer through a native integration. That's the fastest route to a working phone number if you want the orchestration handled and you're comfortable with a platform owning the conversation design.

Which one you want comes down to a single question: is the STT/LLM/TTS integration your differentiator, or is it the thing standing between you and the product you actually want to build? We laid out the reasoning in how to build with the Voice Agent API.

Design decisions that make or break a phone-based voice agent

The code gets you a working call. These decisions get you a call people don't hang up on.

Turn detection. Fixed silence timers are the most common source of a bad phone agent. Set the timer short and you interrupt anyone reading a card number; set it long and every turn has dead air. Let the transcript drive endpointing and tune it to the call type—min_turn_silence at 128ms with max_turn_silence at 640ms via min_latency for rapid order confirmations, or 512ms/2560ms via max_accuracy for healthcare or legal calls where callers think mid-sentence.

Barge-in. Callers interrupt. When they do, you need to stop TTS playback mid-word, flush the audio already queued at Twilio, and get back to listening without tearing down either WebSocket. An agent that talks over an interruption is worse than an IVR, because at least the IVR never pretended to be listening.

Alphanumeric accuracy. Account numbers, confirmation codes, dates of birth, email addresses. This is what phone agents exist to collect and it's the hardest thing to transcribe. Three levers: pass agent_context so the model knows a number is coming, add keyterm prompting for your domain vocabulary (product names, plan tiers, clinic names), and raise min_turn_silence on the turns where you've just asked for a long string so a mid-number pause doesn't end the turn.

Function calling. The agent needs to reach your systems—order lookup, appointment availability, account balance. Two rules: make every tool call idempotent, because retries happen on flaky calls, and speak a filler line ("let me pull that up") before a call that takes more than ~600ms. Silence during a lookup reads as a dropped call.

Audio quality and environment. Phone callers are in cars, hallways, and drive-thrus. voice_focus handles the isolation; the mode preset handles the accuracy/latency tradeoff. balanced is right for most agents, max_accuracy is worth it when the audio is genuinely bad and you can absorb a small delay.

PII and data handling. Phone agents collect sensitive data by design. Decide up front what you retain: whether you keep call audio at all, how long transcripts live, and where. For recorded audio, Speech Understanding provides PII redaction on both text and audio. If your agent touches protected health information, AssemblyAI is considered a business associate under HIPAA and offers a standard Business Associate Addendum (BAA) that can be signed without a sales call. If you have an EU data residency requirement, the same models run at streaming.eu.assemblyai.com at the same price.

Concurrency. Phone traffic is spiky in a way web traffic isn't—a billing run or a product recall triples your call volume in an afternoon. Universal-3.5 Pro Realtime has no concurrency limit, and new-session rate limits scale automatically, so the constraint moves to your own bridge servers and your LLM provider's quota. Load-test both before launch, not after.

Common use cases for phone-based voice agents

The pattern that works: high call volume, predictable conversation scope, and a clear success condition.

Healthcare. Appointment scheduling, reminders, prescription refill requests, and intake triage. Accuracy on medication names and clinic names is the whole ballgame, and Medical Mode is available on the streaming models for exactly that. BAA available for deployments processing PHI—see our medical solutions page.

Contact centers. Tier-one deflection, callback collection, and after-hours coverage. The measurable win is usually containment rate, and the second-order win is that human agents stop spending their day on password resets. More on the pattern in AI use cases in contact centers.

Financial services. Balance inquiries, card activation, transaction disputes, and payment scheduling. Every one of these turns on getting a long number right the first time.

E-commerce and logistics. Order status, returns initiation, delivery rescheduling. Bounded conversations with a clean API behind them.

Field service. Dispatch confirmation, arrival windows, job status updates from technicians calling in from a truck. This is where far-field audio and background noise handling earn their keep.

How to evaluate a phone-based voice agent before shipping

Five steps, in order, and none of them are "check the vendor's benchmark."

1. Record real calls. Fifty to a hundred from your actual traffic, including the bad ones—accents, noise, callers on speakerphone. This becomes your test set. Vendor benchmarks are a filter for which models to try, not evidence about your audio.
2. Measure end-to-end latency, per stage. Log a timestamp when the caller stops speaking, when the final transcript arrives, when the LLM returns its first token, and when the first audio byte plays. Find the biggest gap. It's rarely the one you assumed.
3. Audit the transcripts for entity errors, not just WER. Pull every account number, name, date, and email out of the transcript and diff it against ground truth. That number predicts task completion; pooled WER doesn't. Our guide on how to evaluate speech recognition models walks through building the harness.
4. Score task completion. Did the caller get the thing they called for, without a transfer? This is the only metric that means anything to the business.
5. Read transcripts by hand. All of them, at first. Nothing surfaces a broken endpointing config faster than reading twenty calls where the agent cut someone off mid-sentence.

Phone-based voice agent vs. IVR vs. chatbot

That last cell carries a real condition. A phone voice agent with an 1,800ms turn gap scores worse than a well-built IVR, because the IVR at least never implied it was going to keep up. Latency isn't a polish item you get to later. It's the feature.

Putting it together

A phone-based voice agent is four components and one WebSocket bridge, and the whole thing is about 80 lines of Python to stand up.

But here's the thing worth taking away, and it's not in the code. The components in this stack stopped being independent sometime in the last year, and the phone use case is where you feel it most.

agent_context is an STT parameter whose entire value comes from what the LLM just said. Turn detection is a conversational decision now made inside the speech model rather than by a timer in your orchestrator. Keyterm prompting is your product catalog, injected into the acoustics layer. The clean boxes in the architecture diagram are still a useful way to explain the system, and an increasingly bad way to build it—the accuracy is in the wiring between them.

Which means when you evaluate the speech layer for a phone agent, the question isn't just how accurate it is on a benchmark. It's how much you can tell it about the conversation it's in, and what it does with that. On a phone call—8kHz, no video, a caller reading out a 16-digit number—that's most of the difference between an agent that works and one that gets hung up on.

Frequently asked questions

What is a phone-based voice agent?

A phone-based voice agent is an AI system that holds a full spoken conversation over the phone network, understanding natural speech instead of touch-tone menus and responding with synthesized audio in real time. It combines telephony, streaming speech-to-text, an LLM, and text-to-speech into a single low-latency loop. Typical uses are inbound support, appointment scheduling, order status, and after-hours coverage.

How does a phone-based voice agent work?

The telephony provider answers the call and opens a bidirectional audio stream to your server, usually over a WebSocket. Your server forwards the caller's audio to a streaming speech-to-text model, which returns partial transcripts continuously and a final transcript when it detects end-of-turn—Universal-3.5 Pro Realtime does this at roughly 300ms using the punctuation it predicts rather than a silence timer. That transcript goes to an LLM, whose streamed response goes to text-to-speech and back to the caller, with the whole round trip targeting about 800ms.

What is the best speech-to-text for a phone-based voice agent?

Universal-3.5 Pro Realtime (universal-3-5-pro) is built for this: it takes 8kHz mulaw telephony audio natively, scores 6.99% word error rate and 15.31% entity error rate on Pipecat's open STT benchmark, and hits 3.55% entity error rate on phone numbers—the category phone agents live or die on. It accepts agent_context, so passing the agent's own question into the session sharpens short replies like spelled-out emails and account IDs. It runs at $0.45/hr base over wss://streaming.assemblyai.com/v3/ws with unlimited concurrency.

How do I build a phone-based voice agent with Twilio?

Point your Twilio number at a webhook that returns TwiML with <Connect><Stream url="wss://your-server/ws" />, which opens a bidirectional media stream to your server. In your WebSocket handler, decode Twilio's base64 mulaw frames and forward the raw bytes to wss://streaming.assemblyai.com/v3/ws with sample_rate=8000, encoding=pcm_mulaw, and speech_model=universal-3-5-pro, then run the transcript-to-LLM-to-TTS loop concurrently with the inbound audio loop. If you'd rather not run a media server at all, you can point the same Twilio number at the Voice Agent API over a SIP trunk instead.

What is the difference between a phone-based voice agent and an IVR?

An IVR follows a fixed decision tree driven by touch tones or a handful of recognized keywords, so callers have to translate what they want into the menu's vocabulary. A phone-based voice agent understands free-form speech, keeps context across the whole conversation, handles interruptions, and can call your systems mid-conversation to look something up. The practical difference callers notice is that they can just say what they want, once.

How much does it cost to run a phone-based voice agent?

Streaming speech-to-text on Universal-3.5 Pro Realtime is $0.45/hr base, which works out to about $0.0075 per minute of connected call time, billed per second with no minimums. Add telephony (typically around a cent a minute), LLM tokens, and TTS, and a self-assembled phone agent generally lands in the low single-digit cents per minute. The managed alternative is the Voice Agent API at a flat $4.50/hr—7.5 cents per minute covering STT, LLM, and TTS together, with one bill instead of four. Full breakdown on the pricing page.

‍