How to build a voice agent that transfers to a human

Every good voice agent knows its limits. At some point a caller asks for something the agent shouldn't handle alone — a refund past policy, a billing dispute, a distressed customer — and the right move is to bring in a human. The hard part isn't dialing another number. It's handing off without dropping context, so the customer doesn't have to repeat everything they just said to a live agent who's starting from zero.

In this guide we'll build that warm handoff with AssemblyAI's Voice Agent API. The agent will detect when it needs help, call a transfer tool, pass along a summary of the conversation, and keep the caller informed while the connection is made. Because the agent runs on Universal-3.5 Pro Realtime, the transcript you hand off is actually accurate — which matters a lot when that summary includes an order number or an email address.

What we're building

The flow has four moving parts. The agent listens and talks like normal. When the conversation hits an escalation trigger, the model calls a transfer_to_human tool. Your backend receives that call, pulls the running transcript into a short summary, and initiates a transfer through your telephony provider (we'll use Twilio as the example). Meanwhile the agent tells the caller what's happening so the silence during connection doesn't feel like a dropped call.

Prerequisites

You'll need an AssemblyAI API key, a telephony provider for the actual phone transfer (Twilio here — AssemblyAI has a dedicated Twilio connection guide), and an HTTP endpoint of your own to receive the transfer request and kick off the handoff. If you're new to the Voice Agent API, our build with the Voice Agent API guide covers the basic connection first; this post assumes you have a talking agent and focuses on the transfer.

The architecture

Conceptually: your client (or server) opens a WebSocket to the Voice Agent API and configures the agent with a system prompt, a greeting, and a set of tools. The one that matters here is transfer_to_human. When the model decides to escalate, it emits a tool.call event. You run the transfer logic and reply with a tool.result. The transcript that feeds your handoff summary comes from the same accurate stream powering the conversation.

Step 1 — define the transfer_to_human tool

Tools are declared as JSON Schema functions in your session.update message. The key detail for a transfer is execution_mode. Most tools use "interactive" — the agent keeps chatting while a quick lookup runs. A human transfer is different: it's a longer, sensitive operation where you want the agent to go quiet rather than fill the air with small talk. That's exactly what "hold" mode is for.

{
  "type": "session.update",
  "session": {
    "system_prompt": "You are a support agent for Acme. Help with orders and
account questions. If the caller asks for a refund outside policy, disputes a 
charge, is upset, or explicitly asks for a person, call transfer_to_human with a
short reason and a summary of what the caller needs.",
    "greeting": "Hi, thanks for calling Acme support. How can I help?",
    "output": { "type": "audio", "voice": "ivy" },
    "tools": [
      {
        "type": "function",
        "name": "transfer_to_human",
        "description": "Escalate the call to a human agent. Call this when the
request is outside policy, the caller is upset, or the caller explicitly asks for
a person.",
        "parameters": {
          "type": "object",
          "properties": {
            "reason": {
              "type": "string",
              "description": "Short reason for the transfer, e.g. 'refund outside
30-day policy'."
            },
            "summary": {
              "type": "string",
              "description": "One- or two-sentence summary of what the caller
needs, including any order or account IDs already collected."
            }
          },
          "required": ["reason", "summary"]
        },
        "execution_mode": "hold",
        "timeout_seconds": 60
      }
    ]
  }
}

Notice that the tool asks the model for a summary and a reason. You're using the LLM that's already in the loop to compress the conversation into a clean handoff note — no separate summarization call needed.

Step 2 — detect the escalation trigger

You don't hard-code triggers with keyword matching. You describe them in the system prompt and let the model decide, which handles the messy reality of how people actually ask for help ("can I talk to someone," "this isn't working," an audibly frustrated tone). The prompt above lists the conditions; the model calls the tool when it recognizes one. If you want tighter control, make the conditions more specific in the prompt — but resist the urge to also keyword-match in your client, which tends to fire on false positives like "no, I don't need a person."

Step 3 — execute the transfer and pass the transcript

When the model escalates, you receive a tool.call event with the arguments. Because the tool is in hold mode, the agent stays quiet while you work. Your handler kicks off the telephony transfer and passes the summary along so the human sees context before they even say hello.

async for raw in ws:
    event = json.loads(raw)

    if event.get("type") == "tool.call" and event["name"] == "transfer_to_human":
        args = event["arguments"]
        # Combine the model's summary with your own running transcript/CRM data
        handoff_note = build_handoff(args["reason"], args["summary"], transcript)

        # Kick off the warm transfer via your telephony provider (e.g. Twilio),
        # attaching the handoff note so the human agent sees it on connect.
        transfer_id = start_warm_transfer(handoff_note)

        # Reply to the agent so it knows the transfer is underway.
        await ws.send(json.dumps({
            "type": "tool.result",
            "call_id": event["call_id"],
            "result": json.dumps({"status": "transferring", "transfer_id": 
transfer_id})
        }))

The handoff_note is where the "warm" in warm transfer lives. It carries the model's summary plus anything you've collected — order numbers, verified identity, the accurate transcript of what was said. Since the transcript comes from Universal-3.5 Pro Realtime, spelled-out emails and read-aloud numbers in that note are right, not approximations.

Step 4 — keep the caller informed and handle fallback

Silence during a transfer feels like a dropped call. Have the agent say something before it goes quiet — the system prompt can instruct it to say "let me connect you with a specialist, one moment" as it triggers the tool. And plan for the unhappy path: if no human is available, your transfer function should return a status the agent can act on, so it can offer a callback or take a message instead of leaving the caller in limbo. Return that outcome in the tool.result and let the model handle it gracefully.

A note on telephony

The Voice Agent API handles the conversation — speech-to-text, the LLM, text-to-speech, turn detection, and tool calling — but the actual phone transfer happens through your telephony provider. AssemblyAI provides a Twilio connection guide for wiring the audio together, and the same pattern applies to other providers. The division of labor is clean: the API owns what the agent hears and says; your telephony layer owns the call routing.

Why accuracy matters in a handoff

It's tempting to treat transcription as a background detail in a transfer flow. It isn't. The whole point of a warm handoff is that the human doesn't start from scratch — and that promise breaks if the summary says the wrong order number. This is where building on a model with strong entity accuracy pays off: the names, numbers, and identifiers in your handoff note are the ones the caller actually said. For agents that collect identifiers before escalating, pair this with conversation context so those values are captured cleanly in the first place.

The takeaway

A human transfer is a small feature that carries a lot of customer trust. Done poorly, it's a cold dropped-context restart; done well, it's a seamless step-up that makes your automated agent feel like part of a real support team. The build comes down to one tool in hold mode, a model that decides when to use it, and an accurate transcript that makes the handoff genuinely warm. Wire those together and your agent knows not just how to help — but when to get help. If you're weighing how this fits a larger build, our overview of AI voice agents and the Voice Agent API introduction are good next reads.

Frequently asked questions

How do I build a voice agent that transfers to a human?

Define a transfer tool as a JSON Schema function in your Voice Agent API session, set its execution_mode to "hold" so the agent stays quiet during the transfer, and describe the escalation conditions in your system prompt. When the model calls the tool, your backend receives a tool.call event, initiates the transfer through your telephony provider, and passes a summary so the human has context on connect.

What is a warm handoff in a voice agent?

A warm handoff is a transfer where the human agent receives the conversation's context — a summary, collected identifiers, and the transcript — before taking the call, so the customer doesn't have to repeat themselves. It's the difference between "let me connect you" that works and one that forces a cold restart.

Why use "hold" execution mode for a transfer instead of "interactive"?

Interactive mode keeps the agent talking while a tool runs, which is right for quick lookups but wrong for a transfer — the agent would fill a 30-second connection with awkward small talk. Hold mode lets the agent go quiet during longer, sensitive operations like phone transfers and escalations, which is the natural behavior for a handoff.

Does the Voice Agent API handle the phone call itself?

The Voice Agent API handles the conversation — speech-to-text, the LLM, text-to-speech, turn detection, and tool calling — while the actual phone routing happens through a telephony provider such as Twilio. AssemblyAI provides a Twilio connection guide, and the same tool-calling pattern works with other providers.

How do I make sure the handoff summary is accurate?

Build the agent on a model with strong entity accuracy so the order numbers, emails, and names in the summary match what the caller actually said. Universal-3.5 Pro Streaming powers the transcription, and pairing it with conversation context helps capture spelled-out identifiers correctly before the escalation happens.

‍