New AssemblyAI-TTS-1: Text-to-speech for voice agents. Learn more
Insights & Use Cases

Build a phone support voice agent that looks up orders with tool calling

Build a Python phone agent that looks up orders: Twilio Media Streams, Universal-3.6 Pro Realtime, an LLM tool loop through LLM Gateway, AssemblyAI Text-to-Speech in 8 kHz mu-law, keypad entry and warm transfer.

Abstract green cylinder illustration

Written by

Kelsey Foster

Published on

28 October 2026

A customer support voice agent that looks up orders needs one thing a general support agent doesn't: tool calling. The caller reads an order number, the LLM decides to call get_order_status with it, your code runs the lookup, and the LLM turns the result into a sentence that AssemblyAI Text-to-Speech speaks back. This guide builds that agent on a real phone line: a Twilio number streams the call to your server, Universal-3.6 Pro Realtime transcribes it, the tools run inside an OpenAI-style request to LLM Gateway, and the reply goes back as 8 kHz μ-law, all on one AssemblyAI API key. It also covers the accuracy settings that matter at 8 kHz, keypad entry, warm transfer to a human, and the failure modes that only show up once real callers are on the line.

What this guide builds on

This is the order lookup half of the story. The conversation basics (streaming the LLM's reply into speech, the system prompt, barge-in, and a handoff summary) are covered in how to build an AI voice agent for customer support in Python, which runs on a laptop microphone and answers policy questions without tools. Here the agent gets tools, a phone number, and a backend to query.

The gap between those two states is larger than most teams expect. A laptop gives you 16 kHz audio, a stable connection, and a user who is looking at a screen. A phone line gives you 8 kHz μ-law, carrier jitter, a caller who may be driving, a keypad, and an expectation that a human is reachable within about ninety seconds. Everything below is about closing that gap.

How the call reaches your agent

Twilio Media Streams opens a WebSocket from your Twilio number to a server you run, and that server talks to three AssemblyAI endpoints on one key:

Caller → your Twilio number → Media Streams WebSocket → your server
    your server → Universal-3.6 Pro Realtime   (8 kHz mu-law in, Turn events out)
    your server → LLM Gateway                  (messages + tools in, text or tool_calls out)
    your server → AssemblyAI Text-to-Speech    (text in, 8 kHz mu-law out) → Twilio → caller

Both AssemblyAI ends speak Twilio's format natively. Universal-3.6 Pro Realtime takes encoding pcm_mulaw at a sample rate of 8000, and the Text-to-Speech session returns pcm_mulaw at 8000 Hz, resampled on the server from its native 24 kHz. Your server forwards audio in both directions without transcoding it, so the only code you own is the conversation: turns, tools and replies. AssemblyAI Text-to-Speech delivers frontier-level quality and runs faster than other production voice agent TTS models.

What runs where

Hop Handled by Audio or data
Phone call to your server Twilio Media Streams 8 kHz μ-law, base64 in JSON frames
Caller speech to text Universal-3.6 Pro Realtime pcm_mulaw at 8000 in, Turn events out
Deciding and looking up LLM Gateway plus your tools messages and tools in, text or tool_calls out
Reply text to speech AssemblyAI Text-to-Speech text in, pcm_mulaw at 8000 out
Keypad and transfer Twilio (dtmf events, REST API) digits in, call redirect out

Owning the server is what makes tool calling possible: the lookup runs in your code, next to your order system, with your credentials, and nothing about your backend is exposed to the call. Teams building on top of an existing UCaaS or contact center stack, the category business communications providers like Nextiva operate in, often run this same bridge inside their own media plane.

Put an order lookup agent on a phone number

Universal-3.6 Pro Realtime, LLM Gateway and Text-to-Speech all speak to your server on one AssemblyAI key, with 8 kHz mu-law in and out. Start with free credit.

Sign up free

Connect a Twilio number

You need an AssemblyAI API key, a Twilio phone number with Voice enabled, Python 3.9+, and ngrok to expose your local server during development. Install the dependencies:

pip install -U "assemblyai>=1.8.2" openai fastapi uvicorn "websockets>=14" twilio python-dotenv

Save your credentials in .env. The AssemblyAI key covers speech-to-text, LLM Gateway and Text-to-Speech. HUMAN_QUEUE_NUMBER is where escalated calls go.

ASSEMBLYAI_API_KEY=your_assemblyai_key
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=your_twilio_auth_token
HUMAN_QUEUE_NUMBER=+15551234567

LLM Gateway may require a paid account. If you'd rather start with your own OpenAI key, point the AsyncOpenAI client in the tool loop at OpenAI instead; nothing else changes.

Everything below goes into one file, server.py, in the order shown. When a call arrives, Twilio hits your webhook and expects TwiML telling it what to do. <Connect><Stream> opens a bidirectional WebSocket to your server for the length of the call:

# server.py
import asyncio
import base64
import json
import os
import re
from urllib.parse import urlencode

import assemblyai as aai
import websockets
from dotenv import load_dotenv
from fastapi import FastAPI, Request, Response, WebSocket
from openai import AsyncOpenAI
from twilio.rest import Client as TwilioClient

load_dotenv()
API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
twilio = TwilioClient(os.environ["TWILIO_ACCOUNT_SID"], os.environ["TWILIO_AUTH_TOKEN"])

app = FastAPI()


@app.post("/twilio/voice")
async def voice(request: Request):
    host = request.headers["host"]
    twiml = f"""<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://{host}/media-stream" />
  </Connect>
</Response>"""
    return Response(content=twiml, media_type="application/xml")

Run the server and expose it:

uvicorn server:app --port 8000
ngrok http 8000

In the Twilio console, set the number's incoming voice webhook to https://<your-ngrok-host>/twilio/voice with the method set to POST. A number that appears to do nothing when called is usually still pointed at an old webhook or TwiML app.

Define the order lookup tools

Tools are OpenAI-style function definitions that you send with every LLM request. The model gets each tool's name, description and parameter schema, and decides from the conversation when to call one. Write each description for the model: it is the routing logic, including when a tool should stay unused.

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "get_order_status",
            "description": (
                "Look up an order by its ID. Order IDs are two letters, a hyphen and "
                "five digits, like AB-12345. Call this whenever the caller asks where "
                "an order is, when it ships, or for tracking. Do not call it for "
                "returns or refunds."
            ),
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {
                        "type": "string",
                        "description": "The order ID as the caller said it, e.g. AB-12345.",
                    }
                },
                "required": ["order_id"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "lookup_orders_by_email",
            "description": (
                "List recent orders for the account with this email address. Use it "
                "when the caller doesn't have an order ID. Confirm the address with "
                "the caller before calling."
            ),
            "parameters": {
                "type": "object",
                "properties": {"email": {"type": "string"}},
                "required": ["email"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "transfer_to_human",
            "description": (
                "Transfer the call to a person. Call it when the caller asks for one, "
                "when a request is outside order status, or after two failed lookups."
            ),
            "parameters": {
                "type": "object",
                "properties": {
                    "department": {"type": "string", "enum": ["billing", "returns", "support"]},
                    "summary": {
                        "type": "string",
                        "description": "One sentence on what the caller needs, with any order ID.",
                    },
                },
                "required": ["department", "summary"],
            },
        },
    },
]

Now the backend. Mock it for the tutorial and keep the error shape structured, so the agent can say something useful instead of guessing.

ORDERS = {
    "AB-12345": {"status": "shipped", "ships_by": "Friday, May 9th",
                 "tracking": "1Z999AA10123456784", "email": "jane@example.com"},
    "AB-67890": {"status": "processing", "ships_by": "Tuesday, May 13th",
                 "tracking": None, "email": "jane@example.com"},
}


def normalize_order_id(raw):
    """Callers read IDs in groups: 'a b dash 1 2 3 4 5' must still match AB-12345."""
    chars = re.sub(r"[^A-Z0-9]", "", raw.upper().replace("DASH", ""))
    return f"{chars[:2]}-{chars[2:]}" if re.fullmatch(r"[A-Z]{2}\d{5}", chars) else None


async def run_tool(name, args, call):
    if name == "get_order_status":
        order_id = normalize_order_id(args.get("order_id", ""))
        order = ORDERS.get(order_id) if order_id else None
        if not order:
            return {"error": "not_found", "message": "No order with that ID."}
        return {"order_id": order_id, **{k: v for k, v in order.items() if k != "email"}}

    if name == "lookup_orders_by_email":
        email = args.get("email", "").strip().lower()
        found = [{"order_id": oid, "status": o["status"], "ships_by": o["ships_by"]}
                 for oid, o in ORDERS.items() if o["email"] == email]
        return {"orders": found} if found else {"error": "not_found",
                                                "message": "No account with that email."}

    if name == "transfer_to_human":
        call.handoff = args  # acted on after the agent says goodbye (see below)
        return {"status": "transferring"}

    return {"error": "unknown_tool", "message": f"No tool named {name}."}

normalize_order_id() is there because callers read IDs in groups and the model passes along what it heard. "A B dash one two three four five" and "AB 12345" should both find the order. A lookup that only accepts the tidy form traps the caller in a re-ask loop, and that's the single most common cause of a support agent that "keeps saying it didn't catch that."

Run the tool loop in the LLM request

Tool calling happens inside the LLM request. Each caller turn appends a user message and calls LLM Gateway with the tools. If the response contains tool_calls, your code runs each one, appends the results as tool messages, and calls the LLM again. When a response comes back with text and no tool calls, that text is the reply, and Text-to-Speech speaks it.

llm = AsyncOpenAI(base_url="https://llm-gateway.assemblyai.com/v1", api_key=API_KEY)
LLM_MODEL = "claude-haiku-4-5-20251001"  # any model on Available models that supports tools
MAX_TOOL_ROUNDS = 4


async def run_turn(call, caller_text):
    """One caller turn: call the LLM with tools until it answers in plain text."""
    call.messages.append({"role": "user", "content": caller_text})

    for _ in range(MAX_TOOL_ROUNDS):
        response = await llm.chat.completions.create(
            model=LLM_MODEL, messages=call.messages, tools=TOOLS, max_tokens=300,
        )
        call.log("llm", request_id=getattr(response, "request_id", None))

        # Claude models can return the text and the tool calls as separate
        # choices, so read every choice instead of only choices[0].
        text = " ".join(c.message.content for c in response.choices if c.message.content)
        tool_calls = [tc for c in response.choices for tc in (c.message.tool_calls or [])]
        if not tool_calls:
            call.messages.append({"role": "assistant", "content": text})
            return text

        call.messages.append({
            "role": "assistant",
            "content": text or None,
            "tool_calls": [tc.model_dump() for tc in tool_calls],
        })
        # Talk while the lookup runs, so the line never goes quiet.
        filler = None
        if text or any(tc.function.name != "transfer_to_human" for tc in tool_calls):
            filler = asyncio.create_task(call.say(text or "One moment while I pull that up."))
        for tc in tool_calls:
            args = json.loads(tc.function.arguments or "{}")
            result = await run_tool(tc.function.name, args, call)
            call.log("tool", name=tc.function.name, args=args, result=result)
            call.messages.append({
                "role": "tool",
                "tool_call_id": tc.id,
                "content": json.dumps(result),
            })
        if filler:
            await filler

    return "Sorry, I'm having trouble with that. Let me get you to a teammate."

Three details make this loop hold up in production.

Read every choice. LLM Gateway returns tool calls in the OpenAI shape for every provider, but Claude models can put the spoken text and the tool call in separate choices. Reading only choices[0] silently drops one of them. The loop collects text and tool calls across all choices, and doesn't branch on finish_reason, which is passed through provider-native.

Bound the rounds. MAX_TOOL_ROUNDS stops a model that keeps calling tools from holding the caller in silence. Four rounds covers an email lookup followed by an order lookup with room to spare.

Log the request ID. Every LLM Gateway response carries a request_id. Keep it with the call SID, so a bad answer on a specific call can be traced to the exact request. See the tool calling docs for the full request and response shapes, and Available models for which models support tools.

Bridge the call to speech-to-text and Text-to-Speech

This is the core of the server. Twilio sends JSON frames with an event field: start when the stream opens (carrying the streamSid you need to send audio back and the callSid you need for a transfer), media for each chunk of base64 μ-law, dtmf when the caller presses a key, and stop when the call ends. Each call gets its own speech-to-text socket, its own Text-to-Speech session, and its own message history.

STT_URL = "wss://streaming.assemblyai.com/v3/ws?" + urlencode({
    "speech_model": "universal-3-6-pro",
    "encoding": "pcm_mulaw",  # Twilio Media Streams: 8 kHz G.711 mu-law
    "sample_rate": 8000,
    "voice_focus": "near-field",
})

# Text-to-Speech in Twilio's own format, so replies need no transcoding
TTS_CONFIG = aai.TTSConfig(
    voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3",  # Audrey
    language="english",
    encoding="pcm_mulaw",
    sample_rate=8000,  # resampled on the server from the native 24 kHz
)

GREETING = "Thanks for calling Acme Corp. I can check on an order for you. What's the order number?"


class Call:
    """State for one phone call: one STT socket, one TTS session, one history."""

    def __init__(self, twilio_ws, stt_ws, tts):
        self.twilio_ws, self.stt_ws, self.tts = twilio_ws, stt_ws, tts
        self.stream_sid = self.call_sid = None
        self.messages = [{"role": "system", "content": SYSTEM_PROMPT}]
        self.reply_task = None
        self.handoff = None
        self.digits = []

    def log(self, kind, **fields):
        print(json.dumps({"call": self.call_sid, "kind": kind, **fields}, default=str))

    async def say(self, text):
        # Prime the speech model with the question the caller is about to answer.
        await self.stt_ws.send(json.dumps({
            "type": "UpdateConfiguration",
            "agent_context": text[-1750:],
        }))
        # Synthesis runs faster than playback. Pace the audio at the wire rate
        # (mu-law is 8,000 bytes a second) with a small lead, so this task lives
        # exactly as long as the caller hears it and barge-in can cut it short.
        loop = asyncio.get_running_loop()
        start, queued = loop.time(), 0.0
        async for chunk in self.tts.speak(text):  # raw 8 kHz mu-law, no header
            await self.twilio_ws.send_text(json.dumps({
                "event": "media",
                "streamSid": self.stream_sid,
                "media": {"payload": base64.b64encode(chunk).decode()},
            }))
            queued += len(chunk) / 8000
            lead = queued - (loop.time() - start)
            if lead > 0.2:
                await asyncio.sleep(lead - 0.2)
        await asyncio.sleep(max(0.0, queued - (loop.time() - start)))

    async def respond(self, caller_text):
        reply = await run_turn(self, caller_text)
        await self.say(reply)
        if self.handoff:
            await transfer(self)

    async def barge_in(self):
        """The caller started talking: stop the reply and empty Twilio's buffer."""
        if self.reply_task and not self.reply_task.done():
            self.reply_task.cancel()  # stops the LLM call, the tools or the playback
            await self.tts.cancel()   # drops synthesis of the request in flight
            await self.twilio_ws.send_text(json.dumps(
                {"event": "clear", "streamSid": self.stream_sid}))  # empties Twilio's buffer

    def start_reply(self, caller_text):
        self.reply_task = asyncio.create_task(self.respond(caller_text))


@app.websocket("/media-stream")
async def media_stream(twilio_ws: WebSocket):
    await twilio_ws.accept()
    async with websockets.connect(
        STT_URL, additional_headers={"Authorization": API_KEY}
    ) as stt_ws, aai.AsyncTTSSession(TTS_CONFIG, api_key=API_KEY) as tts:
        call = Call(twilio_ws, stt_ws, tts)
        call.log("tts_session", id=tts.begin.id)

        async def from_twilio():
            async for raw in twilio_ws.iter_text():
                msg = json.loads(raw)
                event = msg.get("event")
                if event == "start":
                    call.stream_sid = msg["start"]["streamSid"]
                    call.call_sid = msg["start"]["callSid"]
                    call.messages.append({"role": "assistant", "content": GREETING})
                    call.reply_task = asyncio.create_task(call.say(GREETING))
                elif event == "media":
                    await stt_ws.send(base64.b64decode(msg["media"]["payload"]))
                elif event == "dtmf":
                    await handle_dtmf(call, msg["dtmf"]["digit"])
                elif event == "stop":
                    await stt_ws.send(json.dumps({"type": "Terminate"}))
                    break

        async def from_assemblyai():
            caller_speaking = False
            async for raw in stt_ws:
                msg = json.loads(raw)
                if msg.get("type") == "Begin":
                    call.log("stt_session", id=msg.get("id"))
                if msg.get("type") != "Turn":
                    continue
                transcript = msg.get("transcript", "").strip()
                if transcript and not caller_speaking:
                    caller_speaking = True  # first words of a new caller turn
                    await call.barge_in()
                if msg.get("end_of_turn"):
                    caller_speaking = False
                    if transcript:
                        call.log("caller", text=transcript)
                        call.start_reply(transcript)

        async def keep_tts_alive():
            while True:
                await asyncio.sleep(60)
                await tts.keep_alive()  # the session closes after 2 idle minutes

        keepalive = asyncio.create_task(keep_tts_alive())
        listener = asyncio.create_task(from_assemblyai())
        try:
            await from_twilio()
        finally:
            for task in (keepalive, listener, call.reply_task):
                if task:
                    task.cancel()

Four things worth calling out.

One Text-to-Speech session per call. The session opens when the call connects, and every reply in the call, including the greeting and the filler lines, is a request on it. No turn waits for a handshake, and calls already in progress don't count against the sessions-per-minute rate limit. A session closes after two minutes without a frame, so keep_tts_alive() sends a keep-alive while the caller does the talking. See Limits.

Paced playback. Synthesis runs faster than real time, and Twilio will buffer whatever you send. say() paces the μ-law out at the wire rate, 8,000 bytes per second, about 200 ms ahead of the caller. That keeps the reply task alive exactly as long as the caller hears it, so a barge-in that cancels it is never late.

Barge-in in three moves. On the first words of a new caller turn, barge_in() cancels the reply task (which stops the LLM call, the tools or the playback, wherever it is), calls tts.cancel() to stop synthesis of the request in flight, and sends Twilio a clear event to empty its playback buffer. The agent stops talking the moment the caller does.

Turns come from the speech model. Universal-3.6 Pro Realtime decides when the caller has finished and marks it with end_of_turn, so there's no silence timer of your own to tune. Leave the turn-detection parameters at their defaults unless you've measured a reason to change them.

Tune the agent for phone audio

A phone line is a worse microphone than a laptop. 8 kHz μ-law throws away everything above about 3.4 kHz, which is exactly where the acoustic differences between S and F, M and N, and five and nine live. That is why order IDs and email addresses are the first thing to break when a laptop agent moves to a phone.

Two settings and one benchmark matter here.

Isolate the caller

voice_focus, set on the speech-to-text connection in the bridge, strips background audio before it reaches transcription. That improves recognition and, more importantly, stops the agent from treating a car engine or a coworker as a turn. near-field is the right choice for handsets and headsets, the majority of support calls. Switch to far-field when callers are on speakerphone or in a room, and use voice_focus_threshold to set how aggressively it suppresses background audio. Match the setting to how the caller is actually captured.

Don't stack your own denoiser on top. The server already isolates the speaker, and a second layer adds artifacts that cost more than the noise did.

Know what accuracy you're actually getting

Here is how Universal-3.6 Pro Realtime performs on AssemblyAI's English voice-agent benchmark, which scores 12,460 scripted voice-agent scenarios rather than clean read speech. Lower is better.

Metric Universal-3.6 Pro Realtime Deepgram Flux EN ElevenLabs Scribe v2 Deepgram Nova-3
Word error rate 5.19% 13.50% 7.78% 8.64%
Entity error rate 14.4% 30.1% 18.5% 26.1%
Names 10.9% 29.0% 14.8% 24.3%
Codes / IDs 10.0% 46.1% 12.0% 27.6%
Phone numbers 2.4% 11.5% 3.4% 4.5%

Two rows carry the weight for order lookup. Codes and IDs at 10.0% is the order number itself, against 46.1% for Deepgram Flux EN. Phone numbers at 2.4% is the digit-sequence case: account numbers, ZIP codes, callback numbers. Names at 10.9% is the harder one, and the reason the prompt reads values back before acting on them. On Pipecat's open benchmark, the same model posts a 307 ms median time to final transcript. See the full benchmarks for methodology.

Give the model the conversation

Short answers are ambiguous in isolation. "It's B as in boy, four two" only parses correctly if you know the agent just asked for an order number. say() handles this: before every line the agent speaks, it sends the line to the speech-to-text socket as agent_context in an UpdateConfiguration message, so the caller's next turn is transcribed against the question it answers. It helps most on short replies and spelled-out entities, which is most of an order lookup.

For vocabulary you know in advance, such as product names, add keyterms_prompt to the connection parameters, or update it mid-call when the conversation moves to a new stage. See prompting and keyterms and the voice agent best practices.

Use the keypad when the microphone won't do

Some values should never go through speech recognition. Card numbers are the obvious case, but long account IDs and claim numbers have the same problem: a caller reads sixteen digits, one gets dropped, and they hear "I didn't catch that" three times in a row.

Twilio sends a dtmf event on the Media Streams socket when the caller presses a key. Collect digits until #, then hand them to the same tool loop as a caller turn:

async def handle_dtmf(call, digit):
    """Keypad entry: digits end with #, and * starts over."""
    if digit == "*":
        call.digits.clear()
    elif digit != "#":
        call.digits.append(digit)
    elif call.digits:
        entered, call.digits = "".join(call.digits), []
        await call.barge_in()
        call.start_reply(f"[caller entered on the keypad: {entered}]")

Use the keypad for anything long, numeric and high-stakes, and speech for everything else: callers resent keypad menus, and a voice agent should feel like a conversation. For PCI-sensitive values like card numbers, don't pass the digits to the LLM at all. Send them straight to your payment system from handle_dtmf() and tell the model only whether it worked.

Kill the dead air during a slow lookup

An order lookup that hits your order management system takes two or three seconds. In a browser that reads as a pause. On a phone line, two seconds of silence reads as a dropped call, and callers hang up or start talking over the agent.

The tool loop covers it. When the LLM asks for a lookup, run_turn() starts speaking before the tool runs: the model's own text if it wrote some ("Let me check AB-12345"), or a fixed "One moment while I pull that up." The filler plays in a task while the lookup runs, so the caller hears the agent working, and the result is ready by the time the sentence ends. The transfer tool skips the filler, because the agent's goodbye covers it.

If a dependency can take longer than a sentence, put a timeout on it in run_tool() and return a structured error. "I can't reach the order system right now, let me get you to someone who can help" is a fine answer. Thirty seconds of silence loses the caller.

Test it on your own call audio

Drop a recording of a real support call into the playground and see how Universal-3.6 Pro Realtime handles your order numbers, names and email addresses.

Try playground

Escalation and warm transfer

Every support agent needs an exit. The difference between a good handoff and a bad one is whether the caller has to repeat themselves.

transfer_to_human is a tool like the others, with a department and a one-sentence summary. run_tool() records the request, the LLM's next reply tells the caller they're being connected, and once that line has played, transfer() redirects the live call with Twilio's REST API:

HUMAN_QUEUE = os.environ["HUMAN_QUEUE_NUMBER"]  # E.164, e.g. +15551234567


async def transfer(call):
    """Warm transfer: post the summary where your agents work, then redirect the call."""
    call.log("handoff", **call.handoff)  # swap for your ticketing or screen-pop API
    twiml = f'<Response><Dial>{HUMAN_QUEUE}</Dial></Response>'
    await asyncio.to_thread(twilio.calls(call.call_sid).update, twiml=twiml)

The summary is what makes it warm. Write it, along with the order ID the agent already looked up and the call SID, into the ticket or screen-pop your human agents work from, so the person picking up starts with context instead of "how can I help you today?"

Gate the transfer behind the lookup in the prompt, as the system prompt below does: transfer after two failed lookups. Fewer tools in play at once also keeps the model's tool choice accurate, so expose only what the current stage of the call needs.

Harden it before you point real traffic at it

Never let the model invent a value

A support agent that can read an order status is a system where a hallucinated value is a broken promise to a customer. The last piece of server.py is the system prompt, and its most important paragraph is the anti-fabrication clause:

SYSTEM_PROMPT = """You are Avery, a phone support agent for Acme Corp. You check on orders.

Order IDs are two letters, a hyphen and five digits. Read the ID back and wait for a
yes before you call get_order_status. Callers who don't have an ID can give the email
on the account: spell it back, then call lookup_orders_by_email. A message like
"[caller entered on the keypad: ...]" is digits the caller typed; treat them as said.

NEVER state an order status, delivery date, tracking number or refund amount unless
that exact value came from a tool result in this conversation. If you have not seen a
tool result, you do not have these values. Do not estimate.

If a tool returns not_found, say so plainly and offer the other path (ID or email).
Call transfer_to_human when the caller asks for a person, when the request is not
about order status, or after two failed lookups. Before the transfer, tell the caller
you're connecting them and that the teammate will have the details.

This is a phone call: one or two short sentences per reply, no lists, no markdown."""

The prompt stops the bad claim. Code stops the bad action: this tutorial only reads data, but before you add issue_refund or update_shipping_address (the next two tickets in the tier-1 queue) check every argument against what the caller said or typed and what your tools returned, and refuse anything that came from neither.

Calls that drop

Mobile calls drop. When Twilio sends stop, the bridge sends Terminate to the speech-to-text socket and closes the Text-to-Speech session. Do that on every exit path: streaming speech-to-text is billed for as long as the socket stays open, and an unclosed session stays open until it times out. A caller who calls back starts a new conversation, so if context matters, key it on the caller's number and load the last summary into the system prompt.

Concurrency

Each call opens one speech-to-text session and one Text-to-Speech session. Both are rate limited on new sessions per minute rather than sessions open at once, and LLM Gateway is rate limited per model. Confirm your headroom before a product launch or a holiday spike, and handle a rate-limit error with backoff and a retry rather than dropping the call.

What you do need to size is everything downstream. Your order management API is usually the constraint. Set a per-tool timeout your backend can actually meet under load, and make sure a slow backend degrades into "let me get someone who can help" rather than a long silence.

What to log

Log every call, including the ones that go well. call.log() writes one JSON line per event, keyed on the Twilio call SID: the speech-to-text session id from Begin, the Text-to-Speech session's begin.id, every caller turn, every LLM request_id, every tool call with its arguments and result, and the handoff summary. When a caller says the agent gave them the wrong ship date, that log tells you whether it was the transcript, the tool or the prompt, and it's the timeline to run evaluations against.

Where to go from here

The phone line is the easy half now. A Twilio webhook, a Media Streams bridge, and μ-law on both AssemblyAI ends get you a working number in an afternoon. What takes longer is the operational work: reading back values before acting on them, keeping write tools behind code checks, covering slow lookups with speech, and running evaluations against call logs instead of guessing.

Universal-3.6 Pro Realtime transcribes 32 languages with native code-switching, and AssemblyAI Text-to-Speech speaks six (English, Spanish, French, German, Italian and Portuguese), which covers most North American and Western European support lines but is worth checking against your actual call mix. See supported languages for the speech-to-text list, the Text-to-Speech quickstart for the session reference, and voice agent architecture for how the pieces fit together.

Planning a production support line?

Talk through call volume, escalation paths and deployment options, including EU data residency, with someone who has done it.

Talk to AI expert

Frequently asked questions

How do I build a customer support voice agent that looks up orders?

Define the lookup as an OpenAI-style tool, such as get_order_status with an order_id parameter, and send it with every LLM request through LLM Gateway. When the response contains tool_calls, run the lookup in your code, append the result as a tool message, and call the LLM again until it answers in plain text. Then speak that text with AssemblyAI Text-to-Speech. Universal-3.6 Pro Realtime supplies the caller's turns, and all three run on one AssemblyAI API key.

How do I connect a voice agent to a Twilio phone number?

Point the number's voice webhook at an endpoint that returns TwiML with <Connect><Stream>, which opens a Media Streams WebSocket to your server for the length of the call. Your server forwards the caller's 8 kHz μ-law audio to Universal-3.6 Pro Realtime with encoding pcm_mulaw at a sample rate of 8000, and sends replies back from an AssemblyAI Text-to-Speech session opened with pcm_mulaw at 8000 Hz, so neither direction needs transcoding.

How does tool calling work through LLM Gateway?

LLM Gateway is OpenAI-compatible: pass a tools array of function definitions in the chat completions request, and tool calls come back under choices[].message.tool_calls in the OpenAI shape for every provider. Claude models can return the text and the tool call in separate choices, so read every choice, and don't branch on finish_reason, which is passed through provider-native. Append the assistant message with its tool_calls and a tool message per result, then call the model again.

Can callers enter an order number on the keypad?

Yes. Twilio sends a dtmf event on the Media Streams socket for each key press. Collect digits until the caller presses #, then pass them to the tool loop as a caller turn so the LLM can look them up. Use the keypad for long numeric values and speech for everything else, and keep PCI-sensitive digits such as card numbers out of the LLM entirely.

How do I stop a voice agent from going silent during a slow order lookup?

Start speaking before the tool runs. When the LLM returns a tool call, play its accompanying text, or a short line like "one moment while I pull that up," in a background task while the lookup runs, then speak the answer when the result is back. Put a timeout on every backend call and return a structured error the agent can explain, so a slow dependency turns into a handoff instead of dead air.

How do I transfer a phone call from a voice agent to a human without dead air?

Model the transfer as a tool, such as transfer_to_human with a department and a one-sentence summary. When the model calls it, let the agent tell the caller they're being connected, then redirect the live call with Twilio's REST API to TwiML that dials your human queue. Write the summary, the order ID and the call SID into the ticket your agents see, so the caller doesn't repeat themselves.

How many concurrent phone calls can an AssemblyAI voice agent handle?

Each call uses one speech-to-text session and one Text-to-Speech session, and both are rate limited on new sessions per minute rather than sessions open at once; LLM Gateway is rate limited per model. Confirm your headroom ahead of a launch or a seasonal spike. In practice the limit is usually your own backend, so set per-tool timeouts your order system can meet under load.

What should I log for a production voice agent deployment?

Log one line per event keyed on the Twilio call SID: the speech-to-text session id, the Text-to-Speech session id, each caller turn, each LLM Gateway request_id, each tool call with its arguments and result, and any handoff summary. Alert on failed transfers and on lookups that time out. That log is how you trace a wrong answer to the transcript, the tool or the prompt, and it's the timeline to run evaluations against.