Build an AI voice agent for customer support that can look up orders

You can build a customer support voice agent that looks up real orders in an afternoon, using one WebSocket connection to the Voice Agent API instead of wiring together separate speech-to-text, LLM, and text-to-speech vendors. The agent transcribes the caller, decides when to call your backend, runs the tool, and speaks the result. This tutorial builds three workflows in Python — order status lookup, account verification by email, and escalation to a human — plus the two things that decide whether a support agent actually works in production: entity accuracy and what the agent does while a slow lookup runs.

Tier-1 customer support is mostly the same five conversations on repeat. Where's my order. I need to change my shipping address. I want a refund. When does this ship. Let me talk to a person. They are high volume, low variance, and structured — which makes them a bad use of a human agent's day and a very good fit for a voice agent. Teams building in this category, like Gorgias, the AI helpdesk for Shopify and DTC merchants, are automating exactly this tier.

Why most support voice agents fail

Two failure modes, and neither one is the LLM.

The first is entity accuracy. A support conversation is mostly alphanumerics: order IDs, account numbers, email addresses, phone numbers, ZIP codes. Headline word error rate barely touches this. A model can score well on overall WER and still hear "ORD-12345" as "ORD-12–345" or turn "jane at example dot com" into something your database has never seen. When that happens the tool call doesn't fail loudly — it returns "no order found," the customer repeats themselves, and the call ends in a transfer. Entity error rate is the number that predicts whether your agent resolves the ticket.

Here is how Universal-3.5 Pro Realtime — the speech model underneath the Voice Agent API — performs on the Pipecat open STT benchmark, which scores models on real agent conversations rather than read speech. Lower is better.

Read the phone-numbers row first, because that is your callback field: 3.55% against Deepgram Flux's 10.41%. Then read the names row, which is your account-verification step: 16.92% against 39.21%. On a model that misses roughly four names in ten, a verification workflow doesn't degrade gracefully — it stops working. More benchmark detail lives on the benchmarks page.

The second failure is dead air during tool calls. The customer asks a question, the agent calls your order system, and there is a two-to-three second silence while the lookup runs. Callers interpret silence as a dropped call and start talking over the agent. That used to be a design problem you worked around. It isn't anymore — the Voice Agent API has a tool-call hold mode, covered in Step 3, that lets the agent acknowledge the request and hold the turn until the result lands.

What you'll build

Three workflows, one agent:

1. Order status lookup — the caller reads an order ID, the agent calls get_order_status and reads back status, ship date, and tracking.
2. Account verification by email — the caller gives an email address, the agent resolves it to an account and lists recent orders.
3. Escalation to a human — the agent recognizes frustration or an out-of-scope request, summarizes the conversation, and hands off.

The stack is the Voice Agent API, Python 3.9+, and a mock backend you'll swap for your real order system. There is no SDK requirement — it's a standard JSON protocol over one WebSocket.

Setup

pip install websockets pyaudio python-dotenv

Put your key in .env:

ASSEMBLYAI_API_KEY=your_key_here

Step 1: Define the support tools

Tools are JSON Schema function definitions. The agent gets the name, the description, and the parameter schema, and decides from the conversation when to call each one. Write the descriptions for the model, not for your teammates — the description is the routing logic.

TOOLS = [
    {
        "type": "function", "name": "get_order_status",
        "description": (
            "Look up the current status of an order by its order ID. "
            "Order IDs look like ORD-12345: three letters, a hyphen, five digits. "
            "Use this whenever the customer asks where their order is, when it ships, "
            "or for tracking information."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "order_id": {
                    "type": "string",
                    "description": "The order ID, formatted as ORD-12345.",
                }
            },
            "required": ["order_id"],
        },
    },
    {
        "type": "function", "name": "lookup_account_by_email",
        "description": (
            "Find a customer account from an email address. Use this when the customer "
            "does not have their order ID but can give an email address."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "email": {"type": "string", "description": "The customer's email address."}
            },
            "required": ["email"],
        },
    },
    {
        "type": "function", "name": "list_recent_orders",
        "description": "List a customer's most recent orders once their account is known.",
        "parameters": {
            "type": "object",
            "properties": {
                "account_id": {"type": "string"},
                "limit": {"type": "integer", "description": "How many orders to return. Default 5."},
            },
            "required": ["account_id"],
        },
    },
    {
        "type": "function", "name": "transfer_to_human",
        "description": (
            "Transfer the caller to a human agent. Use when the customer asks for a person, "
            "is frustrated, or raises something outside order status and account lookup."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "reason": {"type": "string", "description": "Why the transfer is happening."},
                "summary": {"type": "string", "description": "Short summary of the conversation so far."},
            },
            "required": ["reason", "summary"],
        },
    },
]

Now the backend. Mock it for the tutorial, and keep the dispatcher's error shape structured so the agent can say something useful instead of guessing.

ORDERS_DB = {
    "ORD-12345": {
        "status": "shipped",
        "ships_by": "Friday, May 9th",
        "tracking": "1Z999AA10123456784",
        "account_id": "ACC-001",
    },
    "ORD-67890": {
        "status": "processing",
        "ships_by": "Tuesday, May 13th",
        "tracking": None,
        "account_id": "ACC-001",
    },
}

ACCOUNTS_DB = {"jane@example.com": {"account_id": "ACC-001", "name": "Jane Doe"}}
ACCOUNT_ORDERS = {"ACC-001": ["ORD-12345", "ORD-67890"]}


def run_tool(name, args):
    if name == "get_order_status":
        order = ORDERS_DB.get(args.get("order_id", "").upper())
        if not order:
            return {"error": "not_found", "message": "No order with that ID."}
        return order

    if name == "lookup_account_by_email":
        account = ACCOUNTS_DB.get(args.get("email", "").lower())
        if not account:
            return {"error": "not_found", "message": "No account with that email."}
        return account

    if name == "list_recent_orders":
        ids = ACCOUNT_ORDERS.get(args.get("account_id"), [])[: args.get("limit", 5)]
        return {"orders": [{"order_id": i, **ORDERS_DB[i]} for i in ids]}

    if name == "transfer_to_human":
        return {"status": "queued", "wait_minutes": 3}

    return {"error": "unknown_tool", "message": f"No tool named {name}."}

Run tools server-side with HTTP tool calling

The code above executes tools in your client process, which is the right shape for a laptop prototype. In production you usually don't want the caller's audio client to be the thing that talks to your order database. HTTP tool calling lets you configure a tool with an endpoint and have AssemblyAI call it directly, so the lookup executes server-side and your client stays a thin audio pipe. Same tool definitions, one less hop, and no order-system credentials sitting next to a microphone. Field names and auth options are in the tool calling docs.

Parameter hints and guardrails

Two things worth turning on before you ship a support agent.

Tool parameter hints tell the agent what kind of entity a parameter holds — that order_id is three letters, a hyphen, then five digits. That signal improves both transcription accuracy and turn detection while the entity is being spoken, because the system knows an alphanumeric string is in progress and shouldn't treat the pause after "O-R-D" as the end of a turn.

Tool parameter guardrails constrain where argument values can come from: arguments can only be inferred from user turns and tool results, never from the model's own generations. On an internal set of 3,000 tool calls, hallucinated arguments went from 2.5% to 0%. That number matters more here than it does on a demo agent. This tutorial only reads data, but the moment you add issue_refund or update_shipping_address — and you will, they're the next two tickets in the tier-1 queue — a hallucinated argument stops being a transcript error and becomes money moving to an address nobody gave you.

Step 2: Write the system prompt

The system prompt is where you encode the workflow. Be specific about the read-back behavior and absolutely explicit about fabrication.

SYSTEM_PROMPT = """You are Avery, a customer support agent for Acme Corp.

Greeting: open with a short greeting and ask how you can help. One sentence.

Order status:
- If the customer gives an order ID, repeat it back digit by digit to confirm before
  calling get_order_status.
- If they don't have it, ask for the email address on the account, confirm it back,
  then call lookup_account_by_email followed by list_recent_orders.

Identity: confirm the name on the account before reading order details aloud.

Tool transitions: say a short natural phrase before a lookup, such as
"let me pull that up" — never go silent.

Errors: if a tool returns not_found, say so plainly and offer the other path
(order ID vs. email). Never make up an order ID, tracking number, ship date,
or account. If you do not have it from a tool result, you do not have it.

Escalation: call transfer_to_human if the customer asks for a person, sounds
frustrated, or raises anything outside order status and account lookup.

Style: short sentences. No filler. This is a phone call, not an essay."""

The no-fabrication rule is the single most important line in the prompt. An agent that invents a tracking number is worse than an agent that says it can't find one.

Step 3: Connect to the Voice Agent API

One WebSocket to wss://agents.assemblyai.com/v1/ws. You send a session.update with your prompt and tools, stream microphone audio in, and handle events coming back.

import asyncio, base64, json, os

import pyaudio
import websockets
from dotenv import load_dotenv

load_dotenv()
API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
URL = "wss://agents.assemblyai.com/v1/ws"
SAMPLE_RATE = 24000
CHUNK = 1024


async def run_agent():
    async with websockets.connect(URL, additional_headers={"Authorization": API_KEY}) as ws:
        await ws.send(json.dumps({
            "type": "session.update", "session": {
            "system_prompt": SYSTEM_PROMPT,
            "greeting": "Thanks for calling Acme Corp. How can I help?",
            "output": {"voice": "jane"},
            "tools": TOOLS},
        }))

        audio = pyaudio.PyAudio()
        mic = audio.open(format=pyaudio.paInt16, channels=1, rate=SAMPLE_RATE,
                         input=True, frames_per_buffer=CHUNK)
        speaker = audio.open(format=pyaudio.paInt16, channels=1, rate=SAMPLE_RATE,
                             output=True)

        async def send_audio():
            while True:
                data = await asyncio.to_thread(mic.read, CHUNK, exception_on_overflow=False)
                await ws.send(json.dumps({
                    "type": "input.audio",
                    "audio": base64.b64encode(data).decode(),
                }))

        async def handle_messages():
            pending = []
            async for raw in ws:
                msg = json.loads(raw)
                kind = msg.get("type")

                if kind == "session.ready":
                    print("session ready")

                elif kind == "transcript.user":
                    print("caller:", msg.get("text"))

                elif kind == "transcript.agent":
                    print("agent:", msg.get("text"))

                elif kind == "reply.audio":
                    speaker.write(base64.b64decode(msg["data"]))

                elif kind == "tool.call":
                    result = run_tool(msg["name"], msg.get("arguments", {}))
                    pending.append({
                        "type": "tool.result",
                        "call_id": msg["call_id"],
                        "result": json.dumps(result),
                    })

                elif kind == "reply.done":
                    for item in pending:
                        await ws.send(json.dumps(item))
                    pending.clear()

        await asyncio.gather(send_audio(), handle_messages())


asyncio.run(run_agent())

Three things that bite people here. Accumulate tool results and flush them in the reply.done handler rather than firing them the instant the tool returns. Discard pending if the caller interrupts, so a stale result doesn't get spoken into a new topic. And voice names are case-sensitive — see the voices reference for the current catalog.

Hold the turn while a slow lookup runs

The two-to-three second gap from the first section is a solved problem. Tool-call hold mode lets the agent take the turn, acknowledge the request out loud, and keep holding while your backend works, instead of falling silent and inviting the caller to talk over it. For lookups that run longer than a beat, you can also push a manual status update with reply.create — "still pulling up that order, one moment" — so a four-second warehouse query sounds like an agent working rather than a dead line.

This is the difference between a demo and a support line. Real order systems are slow, and they are slowest on exactly the queries customers care most about. Configure hold behavior on the tools that hit your slowest dependencies and leave your fast ones alone.

Tune the speech layer for alphanumerics

The Voice Agent API runs on Universal-3.5 Pro Realtime, and a few Streaming STT controls are worth knowing because they target exactly the failure mode in the benchmark table.

agent_context is the big one. You pass in the agent's own question, and the model uses it to resolve the reply. When your agent has just asked "what's your order number?", a mumbled "oh-arr-dee one two three four five" is no longer an open-ended audio problem — the model knows an order ID is the expected shape of the answer. Across a benchmark of 20,000 voice agent audio files, agent_context cut WER by 10.2%, with short-utterance errors down 13.7% and name entities down 9.4%. Short utterances and names are most of a support call: yes, no, the third one, Jane Doe, ORD-12345.

streaming_params = {
    "speech_model": "universal-3-5-pro",
    "agent_context": "What's your order number?",
    "voice_focus": "near-field",
}

voice_focus isolates the primary speaker and suppresses background noise — near-field for headsets and phone handsets, far-field for rooms, kiosks and drive-thrus. Support callers are rarely in a quiet room.

Modes replace low-level tuning flags with three settings: min_latency, balanced (the default), and max_accuracy. Start on balanced. Move to max_accuracy if your callers read long alphanumerics; move to min_latency only if the conversation feels laggy and you've confirmed the speech layer is the cause.

End-of-turn detection checks whether what the caller has said sounds complete rather than waiting out a silence — which is why a caller pausing mid-order-ID doesn't get cut off. End-to-end, the Voice Agent API runs at roughly one second.

One scoping note before you plan a rollout: the Voice Agent API supports six languages — English, Spanish, French, German, Italian and Portuguese — with native code-switching across all six, so a caller who slips between two of them mid-sentence stays transcribed correctly. If your support queue needs coverage beyond those six, the supported languages post has the current list, and Universal-3.5 Pro covers 18 languages for pre-recorded transcription.

Step 4: Test the three workflows

Run the script and talk to it. What you're checking is not whether the LLM sounds nice — it's whether the entities survive.

Order lookup. Say "I want to check on order O-R-D one two three four five." The agent should confirm "ORD-12345" back to you before calling anything, then report that it shipped and is arriving by Friday, May 9th, with the tracking number. Deliberately mumble the digits. Deliberately pause in the middle. That's what the model has to handle in production — hesitation, mumbles, accents, and ordinary disfluency.

‍Email lookup. Say "I don't have the order number, my email is jane at example dot com." The agent should confirm the address, resolve it to Jane Doe's account, and list both recent orders with dates and totals. Email addresses are where weak models fall apart, because there is no dictionary to fall back on.

Escalation. Say "just put me through to a person." The agent should call transfer_to_human with a reason and a written summary, tell you it's transferring, and give the queue time. Check the summary in your logs — that's the artifact the human agent inherits, and it's the difference between a warm handoff and making the customer start over.

Step 5: Take it to the phone

The browser microphone is fine for development. Production support runs on phone numbers, and the Voice Agent API ships SIP telephony — it plugs into any SIP endpoint, through Twilio, an AssemblyAI-owned number, or a number you already own, at under 500ms. DTMF keypad entry is supported for the cases where a caller would rather type an account number than say it.

That's the whole picture you need for this build. The full telephony walkthrough — SIP configuration, inbound and outbound call flows, webhooks for call lifecycle events — lives in the companion post on taking a support agent to the phone, and in the Twilio connection docs.

What to harden before production

Short list, because the companion post covers hardening in depth:

• Swap the mock databases for your real order system, with timeouts. Pair every slow dependency with tool-call hold mode so a sluggish backend costs you a few seconds of "one moment," not the call.
• Log everything against a session ID — transcripts, tool calls, arguments, results, agent replies. When a caller says the agent gave them the wrong ship date, the log is the only way to tell whether it was the transcript, the tool, or the prompt.
• Turn on parameter guardrails before you ship any tool that writes, refunds, or changes an address.
• Use session resumption. Reconnect within 30 seconds and context is preserved, so a brief network blip doesn't restart the conversation.

Where to go from there

The tool-calling pattern generalizes to the rest of the tier-1 queue with no infrastructure change: cancel an order, update a shipping address, request a refund, schedule a callback, answer from a knowledge base. Each one is a function definition and a few lines in the system prompt. That's the argument for keeping the whole agent on one connection — the transcription, the reasoning, the voice, the turn detection and the billing stay in one place while the workflow list grows.

Keep the scope structured and transactional. Voice agents earn their keep on lookups, verification, scheduling, ordering and triage — bounded workflows with a right answer — not on open-ended conversation. That's also where the accuracy numbers in the benchmark table translate directly into resolved tickets. If you're sizing this across a whole support org, the contact center solutions page and pricing are the next stops; the Voice Agent API docs have the full protocol reference.

Frequently asked questions

How do I build an AI voice agent for customer support that can look up orders?

Define your order lookup as a JSON Schema tool such as get_order_status, connect to the Voice Agent API at wss://agents.assemblyai.com/v1/ws, and send a session.update containing your system prompt and tool definitions. The API handles transcription, decides when to call the tool, and speaks the result back over the same connection, so you write the tool logic and the prompt rather than integrating three separate vendors. Most developers get a working prototype running in an afternoon because there is no SDK to learn — it's standard JSON over one WebSocket.

Why does speech-to-text accuracy matter so much for support voice agents?

Support workflows run on alphanumerics — order IDs, account numbers, emails, phone numbers — and a single mis-heard character breaks the tool call silently, returning "not found" instead of an error you can see. On the Pipecat open STT benchmark, Universal-3.5 Pro Realtime posts a 15.31% entity error rate against Deepgram Flux at 50.50% and ElevenLabs Scribe v2 at 39.70%, with phone numbers at 3.55% versus Flux's 10.41%. Overall word error rate is a poor proxy here; entity error rate is what determines whether the lookup succeeds.

What is agent_context and why does it help support agents?

agent_context passes the agent's own question into the speech model so short or mumbled replies resolve against what was actually asked. If the agent just asked "what's your order number?", the model interprets the caller's next utterance as an order ID rather than as open-ended speech. Across a benchmark of 20,000 voice agent audio files it reduced word error rate by 10.2%, with short-utterance errors down 13.7% and name entities down 9.4% — the two categories that make up most of a tier-1 support call.

How does tool calling work with the AssemblyAI Voice Agent API?

You pass an array of JSON Schema function definitions in the session.update message. When the agent decides to use one, it emits a tool.call event with the function name and arguments; you accumulate results and send tool.result events in the reply.done handler rather than immediately. Tools can also be executed server-side over HTTP so lookups run on AssemblyAI's side instead of in your audio client, and parameter guardrails restrict arguments to values inferred from user turns and tool results — which took hallucinated arguments from 2.5% of 3,000 calls to 0%. For slow backends, tool-call hold mode plus manual reply.create status updates keep the agent talking instead of going silent while the lookup finishes.

Can I connect AssemblyAI's Voice Agent API to phone calls with Twilio?

Yes. SIP telephony is shipped and runs at under 500ms, connecting to any SIP endpoint via Twilio, an AssemblyAI-owned number, or a number you already own, with DTMF keypad support for callers who prefer to type an account number. The full setup for inbound and outbound calls is in the Twilio connection docs and in the companion post on taking a support agent to the phone.

What's the best way to handle escalation to a human in a customer support voice agent?

Register a transfer_to_human tool that takes a reason and a summary, and instruct the agent in the system prompt to call it when a customer asks for a person, sounds frustrated, or raises something outside the agent's scope. The agent writes a summary of the conversation that travels with the transfer, so the human picking up the call doesn't make the customer start over. Treat that summary as a product surface — it is what determines whether escalation feels like a handoff or a failure.

How much does it cost to run a customer support voice agent on AssemblyAI?

The Voice Agent API is a flat $4.50/hr, or $0.075/min, covering speech-to-text, LLM reasoning, text-to-speech, turn detection and tool calling on one connection. There are no per-token surcharges, no separate bills per component, and keyterm prompting is included at no extra charge. Billing is by the minute on actual conversation duration, and there's a free tier for testing.

Do voice agents built with AssemblyAI work with healthcare workflows?

AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure that AAI appropriately safeguards PHI; the platform also holds SOC 2 Type 2, ISO 27001:2022 and PCI DSS v4.0. Note the scoping: BAA-backed deployments run through the Speech-to-Text and Streaming STT products, so for Voice Agent API healthcare use cases you should contact sales before building. Details on the addendum are on the BAA legal page.