New AssemblyAI-TTS-1: Text-to-speech for voice agents. Learn more
Insights & Use Cases

How to build a voice agent with Twilio and AssemblyAI

Build an inbound phone voice agent that bridges Twilio Media Streams into AssemblyAI Universal-3.6 Pro Realtime, GPT-4o with tool calling, and AssemblyAI Text-to-Speech in Twilio's native 8kHz mulaw. Full Python code, barge-in, deployment guide, and forkable repo included.

Abstract green mobius illustration

Written by

Kelsey Foster

Published on

28 October 2026

Putting an AI voice agent on a Twilio phone number comes down to one bridge: Twilio Media Streams into Universal-3.6 Pro Realtime for speech-to-text, your LLM for the reasoning, and AssemblyAI Text-to-Speech for the reply. Both AssemblyAI ends speak Twilio's 8kHz mulaw, so audio passes through in both directions with no transcoding on your side, and the whole speech stack runs on one AssemblyAI API key.

This guide is the complete FastAPI tutorial, updated for AssemblyAI Text-to-Speech. Every hop, from the TwiML webhook to barge-in, is code you own, so you can pick the LLM, the voice and the turn logic.

Why AssemblyAI for phone audio specifically

Phone audio is 8kHz mulaw. That is a narrow, lossy, compressed format built in the 1970s, and most speech models were trained on 16kHz-and-up wideband audio. The usual workaround is to resample on the way in, which costs latency and does not recover the information the codec already threw away.

Streaming Speech-to-Text accepts 8kHz mulaw natively, and AssemblyAI Text-to-Speech returns it when you open the session with encoding pcm_mulaw at an 8000 Hz sample rate. You forward Twilio's frames through unchanged and stream mulaw back out. No resampling stage in your code, no format negotiation, no extra buffer.

The accuracy argument matters more on a phone line than anywhere else, because phone agents spend their whole lives collecting things that have to be exactly right: order numbers, phone numbers, names, addresses, confirmation codes. Here is Universal-3.6 Pro Realtime on AssemblyAI's English voice-agent benchmark, 12,460 scripted voice-agent scenarios built around the entities agents have to capture (lower is better):

Metric Universal-3.6 Pro Realtime Deepgram Flux EN ElevenLabs Scribe v2 Deepgram Nova-3
Word error rate 5.19% 13.50% 7.78% 8.64%
Entity error rate 14.4% 30.1% 18.5% 26.1%
Names 10.9% 29.0% 14.8% 24.3%
Codes / IDs 10.0% 46.1% 12.0% 27.6%
Phone numbers 2.4% 11.5% 3.4% 4.5%

Read the phone numbers row first. A 2.4% error rate against Deepgram Flux EN's 11.5% is the difference between an agent that reads a callback number back correctly and one that asks the caller to repeat it. The names row (10.9% against 29.0%) is the difference between an agent that can look up an account and one that transfers to a human. On Pipecat's open STT benchmark, which measures real agent conversations, it independently posts a 0.96% pooled semantic WER. More comparisons live on the benchmarks page.

This is the category CallRail works in — call tracking and phone-based marketing attribution, where every transcript feeds an attribution decision:

"The capabilities AssemblyAI enables us to build help businesses market confidently while saving time and money. It's powerful, almost magical to see it work."

— Ryan Johnson, Chief Product Officer, CallRail

Put a voice agent on a phone number

Get an API key and start streaming 8kHz mulaw from Twilio in a few minutes. No resampling, no format conversion, and free credit to test with.

Sign up free

Build the bridge with FastAPI

Owning the bridge gives you control over every hop: the LLM you're contractually or architecturally committed to, the voice (AssemblyAI Text-to-Speech has thousands of voices, plus instant voice cloning in the dashboard), turn-taking logic tuned to your domain, or an existing agent pipeline that already works and just needs a phone number bolted on.

The architecture is a WebSocket relay. Twilio opens a Media Streams connection to your server. Your server opens a second WebSocket to AssemblyAI. Audio flows caller → Twilio → you → AssemblyAI, transcripts come back, you call an LLM, a Text-to-Speech session on the same AssemblyAI key speaks the reply, and synthesized mulaw goes back down the same Twilio socket.

Before you start

You need:

  • An AssemblyAI API key. The same key covers Universal-3.6 Pro Realtime and Text-to-Speech
  • A Twilio phone number with Voice enabled
  • An LLM API key — this example uses OpenAI, but you can route through the LLM Gateway to reach Anthropic, Google and others through one endpoint
  • Python 3.11+
  • ngrok, to expose your local server during development
pip install fastapi uvicorn websockets python-dotenv openai assemblyai twilio

Step 1: Return TwiML that opens a media stream

When a call arrives, Twilio hits your webhook and expects XML telling it what to do. <Connect><Stream> opens a bidirectional WebSocket to your server and keeps the call alive for its duration.

from fastapi import FastAPI, Request, Response

app = FastAPI()

@app.post("/twilio/voice")
async def voice(request: Request):
    host = request.headers["host"]
    twiml = f"""<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://{host}/media-stream" />
  </Connect>
</Response>"""
    return Response(content=twiml, media_type="application/xml")

Register this URL as the number's incoming voice webhook in the Twilio console, with the method set to POST.

Step 2: Bridge Twilio Media Streams to Universal-3.6 Pro Realtime

This is the core of the build. Twilio sends JSON frames with an event field: start when the stream opens (carrying the streamSid you need to send audio back), media for each 20ms chunk of base64-encoded mulaw, and stop when the call ends. Twilio also sends a dtmf event when the caller presses a key — more on that below.

Note the query parameters on the AssemblyAI socket. Streaming uses the singular speech_model parameter, unlike the async API which takes a plural speech_models list.

import asyncio, base64, json, os
import websockets
import assemblyai as aai
from fastapi import WebSocket

AAI_URL = (
    "wss://streaming.assemblyai.com/v3/ws"
    "?speech_model=universal-3-6-pro"
    "&encoding=pcm_mulaw"
    "&sample_rate=8000"
    
    "&voice_focus=near-field"
)

# Text-to-Speech in Twilio's own format, so replies need no transcoding
TTS_CONFIG = aai.TTSConfig(
    voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3",  # Audrey
    language="english",
    encoding="pcm_mulaw",  # 8-bit G.711 mu-law
    sample_rate=8000,      # Twilio Media Streams rate
)

@app.websocket("/media-stream")
async def media_stream(twilio_ws: WebSocket):
    await twilio_ws.accept()
    stream_sid = None

    async with websockets.connect(
        AAI_URL,
        additional_headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    ) as aai_ws, aai.AsyncTTSSession(
        TTS_CONFIG, api_key=os.environ["ASSEMBLYAI_API_KEY"]
    ) as tts:  # one TTS session per call, reused for every reply

        async def pump_to_assemblyai():
            nonlocal stream_sid
            async for raw in twilio_ws.iter_text():
                msg = json.loads(raw)
                event = msg.get("event")

                if event == "start":
                    stream_sid = msg["start"]["streamSid"]

                elif event == "media":
                    audio = base64.b64decode(msg["media"]["payload"])
                    await aai_ws.send(audio)

                elif event == "dtmf":
                    await handle_dtmf(msg["dtmf"]["digit"], tts, twilio_ws, stream_sid)

                elif event == "stop":
                    await aai_ws.close()
                    break

        async def pump_from_assemblyai():
            reply_task = None
            caller_speaking = False
            async for raw in aai_ws:
                msg = json.loads(raw)
                if msg.get("type") != "Turn":
                    continue
                transcript = msg.get("transcript", "").strip()
                if transcript and not caller_speaking:
                    caller_speaking = True  # first words of a new caller turn
                    await barge_in(reply_task, tts, twilio_ws, stream_sid)
                if msg.get("end_of_turn"):
                    caller_speaking = False
                    if transcript:
                        # Run the reply as a task so this loop keeps reading turns
                        reply_task = asyncio.create_task(
                            handle_turn(transcript, tts, twilio_ws, stream_sid)
                        )

        async def keep_tts_alive():
            while True:
                await asyncio.sleep(60)
                await tts.keep_alive()  # the session closes after 2 idle minutes

        keepalive = asyncio.create_task(keep_tts_alive())
        try:
            await asyncio.gather(pump_to_assemblyai(), pump_from_assemblyai())
        finally:
            keepalive.cancel()

Four things worth calling out.

format_turns=true is a legacy parameter that has no effect on the U3 Pro family. Universal-3.6 Pro Realtime always returns formatted, punctuated, cased text, so there is nothing to set here — you get it for free.

voice_focus=near-field isolates the primary speaker and suppresses background noise. near-field is the correct setting for headsets and phone handsets; far-field is for rooms, kiosks and drive-thrus. On a call from a car or a busy office this is the single highest-leverage flag on the socket.

End-of-turn detection combines semantic context with voice activity rather than waiting on a silence timer. That is why you can act on end_of_turn directly instead of layering your own VAD heuristic on top.

The Text-to-Speech session opens once, when the call starts, and every reply in the call is a request on it. Opening a session per sentence would pay the connection handshake on every turn and spend your sessions-per-minute rate limit; a session opened before the first reply also gives the server time to load the voice. The session closes after two minutes without a frame, so keep_tts_alive() sends a KeepAlive while the caller does the talking, and a session lasts at most an hour. Replies run as a task, so pump_from_assemblyai() keeps reading turns while the agent speaks: that is what lets barge-in (Step 5) fire. See Optimizing accuracy and latency for the session pattern.

Step 3: Add agent_context — the highest-value change in this post

If you make one change to a phone agent's STT config, make it this one. agent_context lets you pass the agent's own last question into the stream, so the model knows what kind of answer to expect. When the agent asks "what's your zip code?", a mumbled "double-oh two one three" resolves correctly instead of becoming words.

Across a benchmark of 20,000 voice agent audio files, agent_context cut word error rate by 10.2%. The breakdown is where it gets interesting for phone agents specifically: fabrications down 18.3%, hallucinations down 17.2%, place-name entities down 15.5%, short-utterance errors down 13.7%, and name entities down 9.4%.

Short-utterance errors are the ones that kill phone agents. "Yes," "nine," "B as in boy" — these are the turns with the least acoustic information and the highest cost when they're wrong.

Send it as a config update on the socket right before you play the agent's question:

async def ask(question: str, aai_ws, tts, twilio_ws, stream_sid):
    await aai_ws.send(json.dumps({
        "type": "UpdateConfiguration",
        "agent_context": question,
    }))
    await speak(question, tts, twilio_ws, stream_sid)

Context Carryover — a short rolling memory of the conversation — is on by default and works alongside this, so you are not responsible for feeding back the whole history yourself.

You can also pick a latency posture with the min_latency, balanced and max_accuracy modes rather than tuning low-level flags. balanced is the default and the right starting point for a phone agent. Move to min_latency for terse, transactional flows where turns are short, and to max_accuracy when a turn is carrying something expensive to get wrong, like a payment reference.

Step 4: Run the LLM loop with tool calling

On each finalized turn, append the transcript to the conversation, call the LLM with your tool definitions, dispatch any tool calls, and feed the results back for a final response.

from openai import AsyncOpenAI

client = AsyncOpenAI()

TOOLS = [{
    "type": "function",
    "function": {
        "name": "get_order_status",
        "description": "Look up the status of a customer order by ID.",
        "parameters": {
            "type": "object",
            "properties": {"order_id": {"type": "string"}},
            "required": ["order_id"],
        },
    },
}, {
    "type": "function",
    "function": {
        "name": "transfer_to_human",
        "description": "Transfer the caller to a human agent.",
        "parameters": {
            "type": "object",
            "properties": {"reason": {"type": "string"}},
            "required": ["reason"],
        },
    },
}]

async def run_llm(messages):
    resp = await client.chat.completions.create(
        model="gpt-4o", messages=messages, tools=TOOLS, stream=False,
    )
    msg = resp.choices[0].message
    if not msg.tool_calls:
        return msg.content

    messages.append(msg)
    for call in msg.tool_calls:
        result = await dispatch(call.function.name, json.loads(call.function.arguments))
        messages.append({
            "role": "tool",
            "tool_call_id": call.id,
            "content": json.dumps(result),
        })

    follow_up = await client.chat.completions.create(
        model="gpt-4o", messages=messages,
    )
    return follow_up.choices[0].message.content

Keep tool handlers fast. A slow lookup does not just add latency, it adds silent latency, and callers hang up on silence long before they hang up on a slow answer. If a tool can take more than a second, say something first: "let me pull that up."

Step 5: Stream AssemblyAI Text-to-Speech back to Twilio as mulaw

The TTS session from Step 2 is opened with encoding="pcm_mulaw" and sample_rate=8000, so its audio is already in Twilio's format. The model's native output is 24kHz 16-bit PCM; the server resamples it to 8kHz and encodes it as mulaw before sending, so there is nothing to convert on your side. tts.speak() sends the text, flushes it and yields the reply's audio as raw chunks with no header. Base64-encode each chunk as it arrives and send it as a media event on the same socket, tagged with the streamSid you captured in the start event.

async def speak(text: str, tts, twilio_ws, stream_sid: str):
    # Each chunk is raw 8 kHz mu-law with no header: Twilio's format as-is
    async for chunk in tts.speak(text):
        await twilio_ws.send_text(json.dumps({
            "event": "media",
            "streamSid": stream_sid,
            "media": {"payload": base64.b64encode(chunk).decode()},
        }))


async def barge_in(reply_task, tts, twilio_ws, stream_sid: str):
    """The caller started talking: stop the reply and empty Twilio's buffer."""
    if reply_task and not reply_task.done():
        reply_task.cancel()  # stops the LLM call or the speak() loop
    await tts.cancel()       # stops synthesis of the request in flight
    await twilio_ws.send_text(json.dumps({"event": "clear", "streamSid": stream_sid}))

Twilio plays frames as they arrive, so the caller starts hearing the reply before the sentence has finished synthesizing. Send chunks as you get them — do not buffer the full response.

For barge-in, Step 2 calls barge_in() on the first words of each new caller turn. It cancels the reply task, calls tts.cancel() to stop synthesis of the request in flight, and sends Twilio a clear event on the same socket. Synthesis runs faster than playback, so most of a reply is usually sitting in Twilio's playback buffer; clear flushes it, so the agent stops talking the instant the caller does.

Step 6: Handle DTMF

Phone agents need keypad input. Callers will punch in account numbers, menu selections and PINs whether or not you designed for it, and speech recognition is the wrong tool for a 16-digit card number read aloud in a parking lot.

Twilio sends a dtmf event on the Media Streams socket with the digit and the track. Buffer digits, terminate on # or on a short idle timeout, and route the collected string into your conversation state:

dtmf_buffer = []

async def handle_dtmf(digit: str, tts, twilio_ws, stream_sid):
    if digit == "#":
        collected, dtmf_buffer[:] = "".join(dtmf_buffer), []
        reply = await run_llm(build_messages(f"[caller entered: {collected}]"))
        await speak(reply, tts, twilio_ws, stream_sid)
    elif digit == "*":
        dtmf_buffer.clear()
    else:
        dtmf_buffer.append(digit)

The rule of thumb: collect anything long, numeric and high-stakes over DTMF, and everything else by voice.

Step 7: Run it and connect Twilio

uvicorn server:app --port 8000
ngrok http 8000

Take the ngrok HTTPS URL, set the number's voice webhook to https://<your-ngrok-host>/twilio/voice with method POST, and call it.

Test streaming accuracy on your own call audio

Drop a recording of a real customer call into the playground and see how Universal-3.6 Pro Realtime handles your order numbers, names and addresses before you write any code.

Try playground

Latency budget: where your milliseconds go

Phone calls are unforgiving about silence. On a call there is no spinner, no typing indicator, and no visual cue that anything is happening — a pause reads as a dropped call, and callers start talking over the agent.

Here is the part the speech model decides for you:

  • End-of-turn detection. Universal-3.6 Pro Realtime decides a turn has ended with turn detection that combines semantic context with voice activity, rather than a fixed silence threshold. This matters more than the raw number: a silence timer set aggressively cuts people off mid-thought, and set conservatively it makes every turn feel slow. Judging completeness avoids that tradeoff.

On a bridge you build, your end-to-end number is the sum of your own choices, and we are not going to invent component figures for a pipeline we don't control. What we can tell you is where the budget actually goes wrong:

Resampling. If any hop in your chain converts 8kHz mulaw to something else and back, you pay for it twice and gain nothing. Keep mulaw end to end.

Non-streaming LLM calls. Waiting for a complete response before starting TTS creates a dead zone the length of the whole generation. Stream tokens into the TTS session with generate() and call flush() once the reply is complete: the server starts speaking as soon as it holds complete sentences. Never flush per token or mid-sentence.

Slow tools. A database lookup that takes 800ms is 800ms of dead air unless you cover it. Acknowledge first, look up second.

Serialized TTS. Synthesizing the entire reply before sending the first frame wastes everything you saved upstream. Send chunks as they come.

Get those four right and the pipeline stays inside the range callers tolerate. Get any one of them wrong and no amount of model speed will save you. There is more on pipeline design in our voice agent architecture guide.

Why AssemblyAI Text-to-Speech for the phone leg

AssemblyAI Text-to-Speech is built for conversational voice agents, and it is trained on millions of hours of audio, including real conversational data. It delivers frontier-level quality and runs faster than other production voice agent TTS models. On a Twilio bridge, a few properties matter most.

Twilio's format, straight out. Ask for pcm_mulaw at 8000 Hz and every chunk is ready to base64-encode into a media event. pcm_alaw is there too for carriers that use A-law.

One session per call. Open the WebSocket session when the call connects and send every reply on it, so no turn waits for a handshake. Rate limits count new sessions per minute, so calls already in progress don't count against the limit.

Barge-in built into the protocol. cancel() stops the request in flight and drops anything queued behind it, and Twilio's clear event empties the playback buffer.

Streams with your LLM. Send tokens with generate() as they arrive and flush() when the reply is complete; the server starts speaking once it has complete sentences.

One key, one account. Speech-to-text and text-to-speech run on the same AssemblyAI API key. Text-to-Speech is $15 per 1M characters, billed on input characters; current rates are on the pricing page.

Six languages and thousands of voices. English, Spanish, French, German, Italian and Portuguese, with the session's language set explicitly (the API doesn't detect it), plus instant voice cloning in the dashboard. See Voices.

Where to go next

Start building your Twilio voice agent

Sign up for an API key and get free credit to test the whole bridge, speech-to-text and text-to-speech on one key, against your own call audio.

Sign up free

Frequently asked questions

How do I build a voice agent with Twilio and AssemblyAI?

Build a Media Streams bridge: return TwiML with <Connect><Stream>, forward Twilio's 8kHz mulaw frames to wss://streaming.assemblyai.com/v3/ws with speech_model=universal-3-6-pro, encoding=pcm_mulaw and sample_rate=8000, pass finalized turns to an LLM with tool definitions, and speak each reply with AssemblyAI Text-to-Speech on a session opened when the call starts with encoding pcm_mulaw and sample_rate 8000, sending the audio back on the same socket as media events.

Can AssemblyAI Text-to-Speech send audio Twilio can play directly?

Yes. Open the Text-to-Speech session with encoding pcm_mulaw and sample_rate 8000, and every audio chunk is raw 8kHz mulaw with no header, which is what Twilio Media Streams expects. Base64-encode each chunk into a media event with the call's streamSid. The model's native output is 24kHz 16-bit PCM, and the server does the resampling and encoding before it sends the audio, so your code converts nothing.

Why use AssemblyAI for a Twilio voice agent?

Universal-3.6 Pro Realtime accepts Twilio's native 8kHz mulaw, and AssemblyAI Text-to-Speech returns it, so there is no resampling stage in your code adding latency and no format conversion to debug. On AssemblyAI's English voice-agent benchmark it posts a 2.4% error rate on phone numbers against Deepgram Flux EN's 11.5%, and 10.9% on names against 29.0% — the two categories phone agents live and die on. Turn detection combines semantic context with voice activity rather than waiting out a silence timer, and both speech directions run on one API key.

How do I handle barge-in on a Twilio voice agent?

When the caller starts talking over the agent, do three things: cancel the task generating the reply, call cancel() on the AssemblyAI Text-to-Speech session to stop the request in flight, and send Twilio a clear event with the streamSid to flush audio it has buffered but not played. Trigger it on the first words of a new caller turn from Universal-3.6 Pro Realtime, which means your transcript loop has to keep running while the agent speaks.

Can a Twilio voice agent handle DTMF keypad input?

Yes. On a Media Streams bridge, Twilio sends a dtmf event on the same WebSocket carrying the digit and track — buffer the digits, terminate on #, and inject the collected string into your conversation state. Collect long numeric values like account numbers and card references over DTMF rather than speech; it is faster for the caller and it removes an entire class of recognition risk.

What latency should I expect from a Twilio voice agent?

On a Media Streams bridge the number depends on the LLM you choose and how you wire the pipeline. Keeping one Text-to-Speech session open for the whole call means no reply waits for a connection handshake. The four things that reliably wreck the budget are resampling audio between formats, waiting for a complete LLM response before starting TTS, slow tool calls that produce dead air, and buffering the full TTS output before sending the first frame.

How much does it cost to run a phone-based voice agent?

You pay Streaming Speech-to-Text at $0.45/hr base, AssemblyAI Text-to-Speech at $15 per 1M characters (billed on input characters), your LLM provider, and Twilio's per-minute voice charges, which vary by country. Current AssemblyAI rates are on the pricing page.

Can a Twilio voice agent handle multiple simultaneous calls?

Yes. AssemblyAI streaming sessions are billed per session-hour and streaming rate limits scale with your usage, so check your account's concurrency headroom rather than assuming there is no quota. Twilio applies concurrency limits per account, which you can raise. Text-to-Speech counts new sessions per minute too, not sessions open at once, so calls in progress don't count against it. On a Media Streams bridge, each call is one FastAPI WebSocket handler, one Streaming Speech-to-Text socket and one Text-to-Speech session, so a single uvicorn process handles hundreds of concurrent calls comfortably — run several behind a load balancer and scale horizontally from there.