How to create an AI cold-calling agent with the Voice Agent API

To build an outbound cold-calling voice agent on the AssemblyAI Voice Agent API, you need four pieces: a compliance gate that blocks the call before it is placed, a telephony path that carries the audio, a configured agent that runs the script, and a tool layer that writes the outcome back to your CRM. The conversation layer is a single WebSocket at wss://agents.assemblyai.com/v1/ws that bundles speech-to-text, LLM, and text-to-speech at a flat $4.50/hour.

This is an implementation guide. It assumes you have already decided to build an outbound dialer and want to know exactly which calls to make, in what order, and where the sharp edges are. Every code sample below reflects the current API surface, including the managed SIP telephony path that removes the audio bridge entirely.

Scope this to structured qualification, not open-ended persuasion

Before any code: the agents that work in production are the ones with a defined script and a closed set of outcomes. A qualification agent asks three known questions, branches on the answers, and ends by writing one of six dispositions. That is a state machine with a voice on it.

An agent asked to "handle objections naturally" and talk a prospect into a meeting is an open-ended persuasion problem, and it degrades fast on real calls. Build the structured version. The rest of this guide assumes a script with defined branches and explicit disposition logic.

This is the same shape as the transactional voice agent patterns that work elsewhere — support lookups, scheduling, order taking. Outbound qualification is one of them; it just dials first. Sales engagement platforms in the B2B prospecting category, Apollo.io among them, are built on exactly this structured-outcome model.

What you're building

CRM / lead list
      │
      ▼
compliance gate ──► blocked (DNC, time window, consent)
      │ passes
      ▼
telephony  ──►  Voice Agent API  ──►  tools  ──►  CRM
(SIP or bridge)   wss://agents.assemblyai.com/v1/ws
      │
      ▼
   webhooks ──► disposition writeback

The compliance gate runs first and it runs before the dial, not during the call. Everything downstream assumes the number is legal to call right now.

Step 1: Build the compliance gate first

Compliance is where outbound programs lose money, and the failure mode is not a bad conversation — it is a legal one. Under the TCPA, statutory damages are $500 per violating call, rising to as much as $1,500 per call for willful or knowing violations. Those are per-call, and an autodialer makes a lot of calls quickly.

Four checks, all before the number is dialed:

1. Internal suppression list. Anyone who has ever asked you to stop.
2. Federal DNC registry. Scrub against it on a current subscription, not a stale export.
3. Calling window. No calls before 8am or after 9pm in the recipient's local time, derived from area code or, better, from a stored address.
4. State consent rules. Several states require all-party consent to record — California, Florida, Pennsylvania, Washington, Illinois, Maryland, Montana, and New Hampshire among them. Several also run their own DNC registries.

from datetime import datetime
from zoneinfo import ZoneInfo

TWO_PARTY_CONSENT = {"CA", "FL", "PA", "WA", "IL", "MD", "MT", "NH"}

def gate(lead, dnc_client) -> tuple[bool, str]:
    """Return (allowed, reason). Fail closed on every unknown."""
    if lead.phone in internal_suppression:
        return False, "internal_dnc"

    if dnc_client.is_listed(lead.phone):
        return False, "federal_dnc"

    tz = lead.timezone
    if not tz:
        return False, "unknown_timezone"   # fail closed
    local = datetime.now(ZoneInfo(tz))
    if not (8 <= local.hour < 21):
        return False, "outside_tcpa_window"

    if lead.state in TWO_PARTY_CONSENT and not lead.recording_disclosure_enabled:
        return False, "consent_disclosure_required"

    return True, "ok"

Two rules that matter more than the code. Fail closed: an unknown timezone or an unreachable DNC service means do not dial. And make the gate the only path to the dialer, so no retry queue, no manual re-run, and no "just this campaign" flag can route around it.

The agent also needs a runtime escape hatch. If a prospect asks to be removed mid-call, that has to land in your suppression list within seconds, not on the next batch sync. That is a tool call, covered in Step 4.

Step 2: Choose a telephony path

There are two, and most teams should start with the first.

Path A — managed SIP

SIP telephony is shipped. You point a Twilio SIP trunk at AssemblyAI, register the number, bind an agent to it, and there is no media server, no audio bridge, and no webhook of your own to run. Latency on this path is under 500ms, and DTMF keypad entry is supported, which matters when a prospect has to punch an extension to reach the person you actually want.

Two Twilio steps and two AssemblyAI steps:

# 1. Create a SIP trunk and route its origination to AssemblyAI
TRUNK_SID=$(twilio api:trunking:v1:trunks:create \
  --friendly-name "Outbound qualification" \
  --domain-name "$TRUNK_DOMAIN" -o json | jq -r '.[0].sid')

twilio api:trunking:v1:trunks:origination-urls:create \
  --trunk-sid "$TRUNK_SID" --friendly-name "AssemblyAI SIP" \
  --sip-url "sip:sip.assemblyai.com" --priority 1 --weight 1 --enabled

# 2. Attach your number to the trunk
NUMBER_SID=$(twilio api:core:incoming-phone-numbers:list \
  --phone-number "$NUMBER" -o json | jq -r '.[0].sid')
twilio api:trunking:v1:trunks:phone-numbers:create \
  --trunk-sid "$TRUNK_SID" --phone-number-sid "$NUMBER_SID"

AAI="https://agents.us.assemblyai.com/v1"
AUTH=(-H "Authorization: Bearer $AAI_API_KEY" -H "Content-Type: application/json")

# 3. Register the number against your trunk
curl -fsS -X POST "$AAI/phone-numbers/import" "${AUTH[@]}" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d "{\"phone_number\":\"$NUMBER\",\"termination_uri\":\"$TRUNK_DOMAIN\"}"

# 4. Bind the agent to the number
curl -fsS -X PUT "$AAI/phone-numbers/$NUMBER/agent" "${AUTH[@]}" \
  -d "{\"agent_id\":\"$AGENT_ID\"}"

Repointing the number at a different agent later is just the PUT again. Twilio needs no changes, and editing the agent itself takes effect on the next call.

One scoping note for an outbound program: the documented SIP flow registers a number and attaches an agent to it, which is what carries the phone leg — including the callback and return-call traffic a cold-calling campaign generates, which is a large share of booked meetings. For originating the outbound dial today you still place the call from Twilio. The full setup is in Connect to Twilio, and both starter repos script all of it as one idempotent command — the Python one is at github.com/AssemblyAI/voice-agent-starter-python (Python 3.9+, standard library only).

Path B — the DIY Twilio Media Streams bridge

Take this path when you need per-call control the managed path does not expose: your own dial-time logic, custom answering-machine detection, call recording you own end to end, or routing across multiple carriers.

You place the outbound call with Twilio, point its Media Stream at your WebSocket server, and relay frames in both directions. The whole bridge is about a hundred lines.

import asyncio, base64, json, os, websockets

AAI_WS = "wss://agents.assemblyai.com/v1/ws"

SESSION_CONFIG = {
    "type": "session.update",
    "session": {
        "system_prompt": SYSTEM_PROMPT,
        "greeting": "Hi, is this {{FIRST_NAME}}?",
        "input":  {"format": {"encoding": "audio/pcmu"}},
        "output": {"format": {"encoding": "audio/pcmu"}, "voice": "alba"},
        "tools": TOOLS,
    },
}

async def bridge(twilio_ws):
    headers = {"Authorization": f"Bearer {os.environ['ASSEMBLYAI_API_KEY']}"}
    async with websockets.connect(AAI_WS, additional_headers=headers) as aai:
        await aai.send(json.dumps(SESSION_CONFIG))

        # Wait for session.ready before sending a single audio frame.
        while True:
            msg = json.loads(await aai.recv())
            if msg["type"] == "session.ready":
                break
            if msg["type"] == "session.error":
                raise RuntimeError(msg)

        await asyncio.gather(
            twilio_to_aai(twilio_ws, aai),
            aai_to_twilio(aai, twilio_ws),
        )

Five things the bridge has to get right:

• Encoding. Twilio Media Streams are G.711 μ-law at 8 kHz, so set both input.format.encoding and output.format.encoding to audio/pcmu. If you send μ-law while the session is configured for audio/pcm (PCM16 at 24 kHz), you get noise, not an error.
• Ordering. The first message on the socket is session.update. There is no session.start. Configuration is nested under a session object.
• Readiness. Wait for session.ready before streaming input.audio. Frames sent before it are dropped.
• Tool result timing. When a tool.call arrives in interactive mode, accumulate the result and send tool.result on reply.done, not the instant the tool returns. That is what lets the agent speak a transition phrase ("let me pull that up") while your CRM lookup runs. Sending it early makes tools fire repeatedly.
• Ending cleanly. Always send session.end. Skipping it leaves the session in a 30-second grace window that you are billed for — trivial on one call, real money across fifty thousand.

The events you handle are session.ready, transcript.user, transcript.agent, reply.started, reply.audio, reply.done, tool.call, and session.ended. You send session.update, input.audio, tool.result, optionally reply.create, and session.end. If the socket drops mid-call, session.resume reconnects within 30 seconds with context preserved — worth wiring up, because carrier hiccups are routine.

Step 3: Write the script as a system prompt

Keep it short, keep the branches explicit, and put the disclosure first. Nothing in this prompt should require the model to improvise a claim.

You are a scheduling assistant calling on behalf of {{COMPANY}}.

DISCLOSURE (say this in the first sentence, always):
"Quick heads up — I'm an AI assistant calling from {{COMPANY}}, and this
call is recorded."

OPENER (under 15 seconds):
Confirm you're speaking to {{FIRST_NAME}}. State why you're calling in one
sentence. Ask for permission to continue.

DISCOVERY (maximum 2 questions, then stop):
1. Does {{TEAM_FUNCTION}} sit with you or someone else?
2. How are you handling {{PROBLEM_AREA}} today?

PITCH (two sentences, no more):
{{COMPANY}} provides {{ONE_LINE_DESCRIPTION}}. Teams use it to
{{PRIMARY_OUTCOME}}.

CALL TO ACTION:
Offer two specific times. Never offer an open-ended "sometime next week."

OBJECTIONS — map, don't argue:
"Not interested"    → thank them, call log_disposition, end the call.
"Send me an email"  → confirm the address, call log_disposition, end.
"Not the right person" → ask who is, call log_disposition, end.
"Bad time"          → offer a callback, call mark_callback, end.

STOP IMMEDIATELY:
If the person asks not to be called again, in any wording, call honor_dnc
before saying anything else. Then confirm and end the call.

NEVER state a price, a customer name, a statistic, or a commitment that did
not come from a tool result in this conversation. If you do not have the
value, say you will follow up with it.

That last clause is not optional. It is the anti-fabrication rule, and on a call that books meetings and quotes terms it is the difference between a lead and a liability.

For the voice, pick an ID from the voice catalog — alba, michael, anna, and the rest. The voice is immutable once the session is established, so choose before connecting. Output is available in six languages (English, Spanish, French, German, Italian, Portuguese) with native code-switching; the agent recognizes considerably more on the input side.

Step 4: Tools are what make the call worth placing

A cold call that ends without a written outcome is a wasted dial. Four tools cover a qualification agent: book_meeting, log_disposition, honor_dnc, and mark_callback.

There are two kinds of tool, and the choice matters operationally:

• HTTP tools are configured on the stored agent with a URL and a parameter list, and AssemblyAI makes the request server-side. Your client does nothing. This is what you want for calendar booking and CRM writes, because it works identically on the managed SIP path where you are not running a bridge at all.
• Function tools are declared inline in session.tools; the agent emits tool.call and your code runs the logic. Use these when the tool needs local per-call state.

Parameter hints are the accuracy lever

parameters is a JSON Schema object, and the keywords you put in it do two jobs at once: they reject a malformed value before the tool runs, and they tell the agent what a complete value sounds like so it stops cutting in halfway through a phone number.

{
  "name": "book_meeting",
  "description": "Book a meeting. Call this only after the prospect has agreed to a specific time. Do not call it to check availability.",
  "parameters": {
    "type": "object",
    "properties": {
      "email": {
        "type": "string",
        "description": "Prospect's work email, as they spelled it out.",
        "format": "email",
        "examples": ["j.smith@acme.co", "alex@example.com"]
      },
      "starts_at": {
        "type": "string",
        "description": "Meeting start, ISO 8601 with offset.",
        "pattern": "\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}",
        "examples": ["2026-09-24T15:00"]
      },
      "disposition": {
        "type": "string",
        "enum": ["booked", "not_now", "not_interested",
                 "wrong_person", "left_voicemail", "dnc"]
      }
    },
    "required": ["email", "starts_at", "disposition"]
  },
  "http": {"url": "https://your-app.com/book", "http_method": "POST"}
}

Use enum for every fixed set — disposition codes above all. A model that invents a seventh disposition value corrupts your funnel reporting silently.

Argument hallucination is the risk that matters here

On an agent that books meetings and writes CRM records, the dangerous failure is not a missed word. It is a confidently fabricated argument: an email address the prospect never said, a time nobody agreed to. Tool parameter guardrails constrain arguments so they can only be inferred from user turns and tool results, never from the model's own generations. Hallucinated tool arguments went from 2.5% of 3,000 calls to 0% after those guardrails shipped, and overall tool-call accuracy improved from 69% to 80% on an internal dataset.

Pair that with the anti-fabrication clause in the prompt. The guardrail stops the bad write; the prompt stops the agent from saying the thing out loud.

Hold mode for slow lookups

Most tools should be interactive — the agent says "let me check" and keeps the line warm. But a CRM enrichment call that takes fifteen seconds, or a live transfer to a human rep, should be hold:

{
  "type": "function",
  "name": "transfer_to_rep",
  "description": "Transfer to a human rep. Takes 15-30 seconds.",
  "parameters": {"type": "object", "properties": {}, "required": []},
  "execution_mode": "hold",
  "timeout_seconds": 60
}

In hold mode the agent goes silent until you send tool.result, which auto-fires the next reply. If the wait runs long, send reply.create with instructions to speak a status update without ending the hold. The mistake to avoid is the reverse: wrapping a two-second database query in hold "to be safe" makes the agent go mute and the prospect thinks the call dropped. Full detail is in the tool calling docs.

Step 5: Webhooks close the loop

Tools handle in-call writes. Webhooks handle everything after the call ends — which is where disposition reconciliation, recording archival, and campaign attribution live.

Subscribe once, scoped to the agent or account-wide:

curl -X POST https://agents.assemblyai.com/v1/webhook-subscriptions \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-app.com/webhooks/assemblyai",
    "events": ["session.completed", "call.ended"],
    "secret": "a-long-random-signing-secret-at-least-32-chars",
    "agent_id": "'"$AGENT_ID"'"
  }'

call.ended carries the call ID, direction, from/to numbers, status, and recording and transcript URLs where available. session.completed carries duration and a close reason — which is how you reconcile billed session time against dials placed.

Verify every delivery before you trust it. Compute HMAC-SHA256 over the raw body and compare in constant time:

import hashlib, hmac

def verify(raw_body: bytes, signature_header: str, secret: str) -> bool:
    expected = "sha256=" + hmac.new(
        secret.encode(), raw_body, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, signature_header or "")

Deliveries retry with backoff, so dedupe on event_id or X-AAI-Delivery-Id and make the handler idempotent. A double-written "booked" disposition is a bad afternoon for whoever owns the pipeline report.

Why transcription accuracy decides the outcome

A cold call collects exactly the values that are hardest to transcribe: a name, a company, a work email, a callback number. Get the email wrong and the meeting invite bounces. Get the callback number wrong and the whole call was for nothing.

The Voice Agent API runs on Universal-3.6 Pro Realtime. Here is how it measures on AssemblyAI's English voice-agent benchmark, which scores 12,460 scripted voice-agent scenarios rather than clean read speech — lower is better:

Read the phone-number row against a dialer's actual job. A 2.4% error rate versus 11.5% for Deepgram Flux EN is the difference between a callback list you can dial and one somebody has to clean by hand. The names row is the same story for the CRM record. On Pipecat's independent open STT benchmark of agent conversations, the same model posts a 0.96% pooled semantic word error rate. More comparisons are on the benchmarks page.

End-of-turn detection combines semantic context with voice activity rather than relying on silence alone, and on Pipecat's open benchmark the final transcript lands a median 307 ms after the speaker stops. On outbound that matters more than it sounds: prospects pause mid-sentence when they are half-listening, and an agent that treats every pause as a turn boundary interrupts constantly and gets hung up on.

agent_context if you assemble your own stack

If you build the pipeline yourself on Streaming STT rather than the bundled Voice Agent API, pass the agent's own last utterance into the transcription session via agent_context. This is what makes "yeah, it's j-dot-smith at acme dot co" resolve to j.smith@acme.co instead of a literal spelling — the model knows an email is coming because it knows what the agent just asked.

ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "agent_context": "Great — what's the best email for the invite?",
}))

Across a benchmark of 20,000 voice agent audio files, agent_context cut word error rate by 10.2%, with the largest gains on exactly the categories a cold call depends on: fabrications down 18.3%, hallucinations down 17.2%, short-utterance errors down 13.7%, and name entities down 9.4%. The user half of the conversation is carried forward automatically; agent_context fills in the agent half. On the Voice Agent API this is handled for you, which is a large part of why the bundled path is the shorter one. The architecture comparison goes deeper on the tradeoff.

Unit economics

The Voice Agent API is a flat $4.50 per hour of session time, or $0.075 per minute, covering speech-to-text, LLM, text-to-speech, turn detection, and tool calls in one line item. See pricing for the current rate card.

The arithmetic for a 90-second qualification call:

• 1.5 minutes × $0.075 = $0.1125 of session time
• Plus carrier minutes and connect/teardown overhead
• ≈ $0.12–$0.18 all-in per completed 90-second call

Two things move that number in production. Sessions you never close stay in a 30-second billed grace window, so always send session.end — at scale that alone is a few percent of spend. And dials that never connect still burn carrier minutes without burning session time, so your cost per conversation is meaningfully higher than your cost per dial.

For the comparison you actually care about, plug in your own fully loaded SDR hourly cost. Teams commonly model somewhere in the $70–100/hour range once salary, benefits, tooling, and management overhead are counted — treat that as an illustrative planning input to replace with your real figure, not as a benchmark.

What to measure

Instrument three rates from day one, and hold them separately. Every call ends with a disposition, so this comes straight out of log_disposition plus the call.ended webhook.

• Connection rate — dials that reach a human.
• Conversation rate — connections that last past 30 seconds. This is the honest read on whether your opener works.
• Book rate — conversations that produce a calendar hold.

The following ranges are illustrative planning assumptions for modeling, not AssemblyAI data — replace them with your own numbers as soon as you have a few thousand dials:

‍

Also track the two operational metrics nobody thinks about until they hurt: gate block rate by reason (a spike in unknown_timezone means your lead data degraded) and tool-call failure rate (a spike means your CRM endpoint is timing out and calls are ending without a written outcome).

Where to go next

The shortest working path is: clone voice-agent-starter-python, publish an agent with your script and your four tools, run the SIP connect script to put it on a number, and call it yourself. Then build the compliance gate in front of the dialer and do not let anything route around it.

The Voice Agent API overview covers the rest of the surface, and the docs have the full event reference.

Frequently asked questions

How do I build a cold-calling voice agent with the Voice Agent API?

Build four pieces: a compliance gate that scrubs DNC registries and TCPA calling windows before any dial, a telephony path, a configured agent, and a tool layer that writes outcomes back to your CRM. For telephony you can either use managed SIP — point a Twilio SIP trunk at sip:sip.assemblyai.com, register the number, and bind an agent to it with no media server of your own — or run a Twilio Media Streams bridge into wss://agents.assemblyai.com/v1/ws when you need custom per-call control. The tool layer typically comes down to book_meeting, log_disposition, honor_dnc, and mark_callback.

Is AI cold calling legal?

AI cold calling is legal in most U.S. jurisdictions provided you comply with the TCPA, state consent laws, and AI disclosure requirements. In practice that means: scrub against the federal Do Not Call registry before every dial, respect calling windows of 8am to 9pm in the recipient's local time, obtain all-party consent for recording in states that require it (California, Florida, Pennsylvania, Washington, Illinois, Maryland, Montana, and New Hampshire among them), and disclose that the caller is an AI at the top of the call. TCPA statutory damages run $500 per violating call and up to $1,500 for willful or knowing violations, so build these checks as hard gates that fail closed rather than as advisory warnings. This is general implementation guidance, not legal advice — have counsel review your program.

How much does it cost to run a cold-calling voice agent?

The AssemblyAI Voice Agent API costs a flat $4.50 per hour of session time, which is $0.075 per minute, and that single rate covers speech-to-text, the LLM, text-to-speech, turn detection, and tool calls. A 90-second qualification call is therefore about $0.11 of session time, or roughly $0.12–$0.18 all-in once carrier minutes and connect overhead are included. Because dials that never connect still cost carrier minutes, your cost per actual conversation is higher than your cost per dial — model both.

Can the Voice Agent API place outbound calls directly?

SIP telephony is shipped and runs under 500ms with DTMF support: you point a Twilio SIP trunk at AssemblyAI, register a number, and bind an agent to it with no audio bridge to run. The documented flow attaches an agent to a number, which carries the phone leg including the callback and return-call traffic an outbound campaign generates. To originate the outbound dial itself you still place the call through Twilio, then either let SIP carry it or bridge the Media Stream into the Voice Agent API WebSocket yourself.

What speech-to-text accuracy do I need for cold calling?

The number that matters is entity accuracy on phone audio, not headline word error rate, because a cold call exists to capture names, emails, and callback numbers correctly the first time. On AssemblyAI's English voice-agent benchmark of 12,460 scripted voice-agent scenarios, Universal-3.6 Pro Realtime — the model under the Voice Agent API — records a 2.4% error rate on phone numbers and 10.9% on names, against 11.5% and 29.0% for Deepgram Flux EN. A wrong digit in a callback number does not degrade the call, it voids it.

How do I stop the agent from inventing meeting times or CRM values?

Use two controls together. Tool parameter guardrails constrain arguments so they can only be inferred from user turns and tool results rather than the model's own generations — that change took hallucinated tool arguments from 2.5% of 3,000 calls to 0%, and lifted overall tool-call accuracy from 69% to 80% on an internal dataset. Then add an explicit anti-fabrication clause to the system prompt instructing the agent never to state a price, name, statistic, or confirmation that did not come from a tool result in that conversation. The guardrail blocks the bad write; the prompt stops the agent from saying it out loud.

How do I write call outcomes back to my CRM?

Two mechanisms, used together. In-call, define an HTTP tool such as log_disposition with an enum on the disposition field — AssemblyAI makes the request server-side, so it works even on the managed SIP path where you run no client code. After the call, subscribe to the call.ended and session.completed webhooks, which POST a signed JSON payload carrying call status, from and to numbers, session duration, and recording and transcript URLs where available. Verify the X-AAI-Signature HMAC over the raw body and dedupe on event_id, since failed deliveries retry with backoff.

Which languages can a cold-calling voice agent speak?

The Voice Agent API speaks six languages — English, Spanish, French, German, Italian, and Portuguese — with native code-switching across all of them, and each is backed by at least one voice with a matching accent. The agent recognizes considerably more languages on the input side than it speaks, at 18 with mid-sentence code-switching, which is what makes it workable on lists where prospects switch languages mid-sentence. Pick the voice before the session is established, since it is immutable once the agent connects.