Build a voice agent that can make outbound calls with AssemblyAI

This quickstart builds a voice agent that places a real phone call. Twilio's Calls API dials the number, and when someone answers, a Media Stream carries the call audio to a small Node.js server. The server sends the callee's speech to Universal-3.6 Pro Realtime, gets a reply from an LLM through LLM Gateway, and speaks it back with AssemblyAI Text-to-Speech. Both AssemblyAI ends speak Twilio's 8 kHz μ-law, so the audio passes through without transcoding, and all three services run on one AssemblyAI API key.

It's one file of under 250 lines. For the full outbound program, with a compliance gate, a sales script, tool calling and CRM writeback, see how to build an AI cold-calling agent.

Why outbound voice agents matter

An agent that dials out reaches people on the channel they actually pick up, which opens workflows that text channels drop the ball on:

In every case the win is the same: the agent reaches the customer by phone, holds a real conversation, and writes the outcome to your system of record.

Architecture

Three services sit behind your server, connected to Twilio by one WebSocket:

POST /call ──► Twilio Calls API ──► callee's phone rings and they answer
                     │
                     ▼  Twilio fetches /twiml → <Connect><Stream>
Twilio Media Stream (8 kHz mu-law, both directions)
                     │
                     ▼
your server.js ──► Universal-3.6 Pro Realtime   (mu-law in, Turn events out)
               ──► LLM Gateway                  (conversation in, reply text out)
               ──► AssemblyAI Text-to-Speech    (reply text in, mu-law out) ──► Twilio ──► callee

The key detail is the audio format. Twilio Media Streams carry base64 G.711 μ-law at 8 kHz in both directions. Universal-3.6 Pro Realtime accepts that as encoding=pcm_mulaw at sample_rate=8000, and the Text-to-Speech session returns pcm_mulaw at 8000 Hz, resampled on the server from its native 24 kHz. Your server only moves bytes and runs the conversation.

AssemblyAI Text-to-Speech is built for conversational voice agents, with frontier-level quality, faster and cheaper than other production voice agent TTS models, and unlimited concurrency.

Prerequisites

• Node.js 18+ and npm
• An AssemblyAI API key. The same key covers speech-to-text, LLM Gateway and Text-to-Speech. LLM Gateway may require a paid account; if you'd rather start with your own OpenAI key, point the OpenAI client at OpenAI instead.
• A Twilio account with a voice-capable phone number
• ngrok (or any public HTTPS tunnel) so Twilio can reach your machine

Consent matters. Automated outbound calls are regulated in most places, including telemarketing, do-not-call and call-recording rules that vary by country and state. Say that the call is automated in the opening line, honor "don't call me again" right away, and have counsel review your program before you dial anyone who hasn't asked to hear from you. This guide is general implementation guidance, not legal advice.

Set up the project

1. Create the project and install

mkdir outbound-voice-agent && cd outbound-voice-agent
npm init -y
npm pkg set type=module
npm install assemblyai@^4.42.0 openai express ws twilio dotenv

assemblyai 4.42.0 or later includes the Text-to-Speech session used below.

2. Configure your environment

Create .env next to server.js. TWILIO_FROM_NUMBER is your Twilio number, and PUBLIC_URL is the ngrok address you'll get in a moment.

ASSEMBLYAI_API_KEY=your_assemblyai_key
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=your_twilio_auth_token
TWILIO_FROM_NUMBER=+15551234567
PUBLIC_URL=https://your-subdomain.ngrok-free.app
PORT=8080

Build the server

Everything below goes into one file, server.js, in the order shown.

1. Set up the clients

One AssemblyAI key drives all three services. The speech-to-text URL and the Text-to-Speech options both ask for Twilio's format, and the voice is set explicitly. Outbound calls need the agent to speak first, so the greeting is defined here too, with the automated-call disclosure in its first sentence.

// server.js
import "dotenv/config";
import http from "node:http";
import { setTimeout as sleep } from "node:timers/promises";
import express from "express";
import { WebSocketServer, WebSocket } from "ws";
import twilio from "twilio";
import OpenAI from "openai";
import { AssemblyAI } from "assemblyai";

const {
  ASSEMBLYAI_API_KEY,
  TWILIO_ACCOUNT_SID,
  TWILIO_AUTH_TOKEN,
  TWILIO_FROM_NUMBER,
  PUBLIC_URL,
  PORT = "8080",
} = process.env;

const twilioClient = twilio(TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN);
const aai = new AssemblyAI({ apiKey: ASSEMBLYAI_API_KEY });

// LLM Gateway is OpenAI-compatible and uses the same AssemblyAI key
const llm = new OpenAI({
  baseURL: "https://llm-gateway.assemblyai.com/v1",
  apiKey: ASSEMBLYAI_API_KEY,
});
const LLM_MODEL = "claude-haiku-4-5-20251001"; // any model from Available models

// Universal-3.6 Pro Realtime, reading Twilio's 8 kHz mu-law as-is
const STT_URL =
  "wss://streaming.assemblyai.com/v3/ws?" +
  new URLSearchParams({
    speech_model: "universal-3-6-pro",
    encoding: "pcm_mulaw",
    sample_rate: "8000",
  });

// Text-to-Speech in Twilio's format, so replies need no transcoding
const TTS_OPTIONS = {
  voiceId: "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", // Audrey
  language: "english",
  encoding: "pcm_mulaw",
  sampleRate: 8000, // resampled on the server from the native 24 kHz
};

const SYSTEM_PROMPT = `You are an automated assistant calling on behalf of Bright Smile Dental
to confirm tomorrow's appointment. You already said you are an automated assistant.
Ask whether the patient can still make it. If yes, confirm and say goodbye.
If no, offer to have the front desk call back to reschedule.
If they ask not to be called again, apologize, confirm it, and say goodbye.
This is a phone call: one or two short sentences per reply, no lists, no markdown.`;

const GREETING =
  "Hi, this is an automated assistant calling from Bright Smile Dental " +
  "about your appointment tomorrow at 10 a.m. Do you still plan to come in?";

const app = express();
app.use(express.json());
app.use(express.urlencoded({ extended: false }));

2. Place the call

POST /call takes the number to dial and asks Twilio to place the call. When the callee picks up, Twilio fetches the URL passed as url to learn what to do next.

// Start an outbound call. Twilio dials, and when someone answers it
// fetches /twiml to learn what to do with the call.
app.post("/call", async (req, res) => {
  try {
    const call = await twilioClient.calls.create({
      to: req.body.to,
      from: TWILIO_FROM_NUMBER,
      url: `${PUBLIC_URL}/twiml`,
    });
    res.json({ callSid: call.sid });
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

Put authentication in front of this endpoint before it leaves your laptop: anyone who can reach it can make your number dial out.

3. Return TwiML that opens a media stream

When Twilio fetches /twiml, the server answers with a <Connect><Stream> verb. Twilio then opens a WebSocket back to the server and streams the call audio over it, in both directions, for as long as the call lasts.

// Connect the answered call to a bidirectional Media Stream on this server
app.all("/twiml", (req, res) => {
  const host = new URL(PUBLIC_URL).host;
  res.type("text/xml").send(`<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://${host}/twilio-stream" />
  </Connect>
</Response>`);
});

4. Open one speech-to-text socket and one Text-to-Speech session per call

Each call gets its own Universal-3.6 Pro Realtime socket, its own Text-to-Speech session and its own message history. The session opens when the call connects and serves every line the agent speaks on that call, so no reply waits for a handshake. It closes after two minutes without traffic, so a keep-alive runs while the callee does the talking. See Limits for session lifetime and rate limits.

// One phone call: one speech-to-text socket, one TTS session, one history
function openCall(twilioWs) {
  const call = {
    twilioWs,
    streamSid: null,
    messages: [{ role: "system", content: SYSTEM_PROMPT }],
    reply: null, // AbortController of the reply in flight
    callerSpeaking: false,
    ended: false,
  };

  call.stt = new WebSocket(STT_URL, {
    headers: { Authorization: ASSEMBLYAI_API_KEY },
  });
  call.stt.on("error", (err) => console.error("[stt]", err.message));

  call.tts = aai.tts.session(TTS_OPTIONS);
  call.ttsReady = call.tts.connect(); // text sent before Begin is held, not lost
  call.ttsReady
    .then((begin) => console.log("[tts] session", begin.id))
    .catch((err) => console.error("[tts]", err.message));
  // The session closes after 2 idle minutes; keep it open while the callee talks
  call.keepAlive = setInterval(() => call.tts.keepAlive(), 60_000);
  return call;
}

function sendToTwilio(call, msg) {
  if (call.twilioWs.readyState === WebSocket.OPEN) {
    call.twilioWs.send(JSON.stringify(msg));
  }
}

5. Speak at the speed of the phone line

session.speak() sends the text, flushes it and yields the reply as raw 8 kHz μ-law chunks with no header. Each chunk is base64-encoded into a Twilio media event. Synthesis runs faster than playback and Twilio buffers whatever you send, so say() paces the audio at the wire rate, 8,000 bytes a second, about 200 ms ahead of the callee. The reply then lasts exactly as long as the callee hears it.

// Speak one line. Synthesis runs faster than playback, so pace the audio at
// the wire rate (mu-law at 8 kHz is 8 bytes per millisecond) with a small
// lead. The reply then lasts exactly as long as the callee hears it, which
// is what lets barge-in cut it short.
async function say(call, text, signal) {
  const start = Date.now();
  let queuedMs = 0;
  for await (const chunk of call.tts.speak(text)) {
    if (signal.aborted) return;
    sendToTwilio(call, {
      event: "media",
      streamSid: call.streamSid,
      media: { payload: Buffer.from(chunk).toString("base64") },
    });
    queuedMs += chunk.length / 8;
    const lead = queuedMs - (Date.now() - start);
    if (lead > 200) await sleep(lead - 200, undefined, { signal });
  }
  const remaining = queuedMs - (Date.now() - start);
  if (remaining > 0) await sleep(remaining, undefined, { signal });
}

6. Get a reply from the LLM

Each finished callee turn goes to LLM Gateway, which is OpenAI-compatible, so the standard openai package works with a different base URL. Every reply runs as cancellable work with its own AbortController, and replies run one after another, so a new answer never talks over the one it replaced.

// One callee turn: ask the LLM for a reply, then speak it
async function reply(call, calleeText, signal) {
  call.messages.push({ role: "user", content: calleeText });
  const completion = await llm.chat.completions.create(
    { model: LLM_MODEL, messages: call.messages, max_tokens: 200 },
    { signal },
  );
  // Some models split a reply across choices, so join every text part
  const text = completion.choices
    .map((choice) => choice.message.content)
    .filter(Boolean)
    .join(" ")
    .trim();
  if (!text || signal.aborted) return;
  call.messages.push({ role: "assistant", content: text });
  await say(call, text, signal);
}

// Run a reply as cancellable work, so barge-in can stop it at any point.
// Turns run one after another: a new reply starts speaking only once the
// one it replaced has stopped.
function startTurn(call, work) {
  call.reply?.abort();
  const controller = new AbortController();
  call.reply = controller;
  call.turn = (call.turn ?? Promise.resolve())
    .then(() => work(controller.signal))
    .catch((err) => {
      if (!controller.signal.aborted) console.error("[reply]", err.message);
    })
    .finally(() => {
      if (call.reply === controller) call.reply = null;
    });
}

To shave more time off long replies, stream the LLM's tokens into the session with generate() and call flush() when the reply is complete. The server starts speaking once it holds complete sentences. See Optimizing accuracy and latency.

7. Handle barge-in cleanly

When the callee talks over the agent, three things have to stop: the LLM request or the playback loop, the synthesis still running on the server, and the audio already sitting in Twilio's buffer. bargeIn() aborts the reply, calls session.cancel(), and sends Twilio a clear event, which empties its playback buffer.

// The callee started talking over the agent: stop everything that's speaking
function bargeIn(call) {
  if (!call.reply) return;
  call.reply.abort(); // stops the LLM request or the playback loop
  call.reply = null;
  call.tts.cancel(); // stops synthesis of the request in flight
  sendToTwilio(call, { event: "clear", streamSid: call.streamSid }); // empties Twilio's buffer
}

8. Forward audio in both directions

Twilio sends connected, start, media and stop events. The server captures streamSid from start and greets the callee, forwards every media payload to Universal-3.6 Pro Realtime as raw bytes, and closes both AssemblyAI sessions on stop. From the speech-to-text side, the first words of a new turn trigger barge-in, and a Turn event with end_of_turn set starts the reply.

function handleCall(twilioWs) {
  const call = openCall(twilioWs);

  // Twilio -> AssemblyAI: capture the stream, greet first, forward audio
  twilioWs.on("message", (data) => {
    const msg = JSON.parse(data);
    if (msg.event === "start") {
      call.streamSid = msg.start.streamSid;
      console.log("[twilio] call", msg.start.callSid);
      // Outbound: the callee doesn't know why the phone rang, so speak first
      call.messages.push({ role: "assistant", content: GREETING });
      startTurn(call, (signal) => say(call, GREETING, signal));
    } else if (msg.event === "media") {
      if (call.stt.readyState === WebSocket.OPEN) {
        call.stt.send(Buffer.from(msg.media.payload, "base64"));
      }
    } else if (msg.event === "stop") {
      hangUp(call);
    }
  });
  twilioWs.on("close", () => hangUp(call));

  // AssemblyAI -> agent: turns drive barge-in and replies
  call.stt.on("message", (data) => {
    const msg = JSON.parse(data);
    if (msg.type === "Begin") console.log("[stt] session", msg.id);
    if (msg.type !== "Turn") return;
    const transcript = (msg.transcript || "").trim();
    if (transcript && !call.callerSpeaking) {
      call.callerSpeaking = true; // first words of a new turn
      bargeIn(call);
    }
    if (msg.end_of_turn) {
      call.callerSpeaking = false;
      if (transcript) {
        console.log("[callee]", transcript);
        startTurn(call, (signal) => reply(call, transcript, signal));
      }
    }
  });
}

// Close both AssemblyAI sessions on every exit path
function hangUp(call) {
  if (call.ended) return;
  call.ended = true;
  clearInterval(call.keepAlive);
  call.reply?.abort();
  if (call.stt.readyState === WebSocket.OPEN) {
    call.stt.send(JSON.stringify({ type: "Terminate" }));
  }
  call.tts.close().catch(() => {});
}

Turn detection comes from the speech model, which combines semantic context with voice activity, so there's no silence timer of your own to tune. Close both sessions on every exit path: streaming speech-to-text is billed while the socket stays open.

9. Start the server

const server = http.createServer(app);
const wss = new WebSocketServer({ server, path: "/twilio-stream" });
wss.on("connection", handleCall);
server.listen(Number(PORT), () => console.log(`Listening on port ${PORT}`));

Echo cancellation is the carrier's job

On a phone call you don't have to handle acoustic echo cancellation; the carrier and the handset take care of it. That's a meaningful difference from a laptop or terminal client, which needs headphones to keep the agent from interrupting itself.

Run it and place a call

Start the server:

node server.js

In a second terminal, expose it:

ngrok http 8080

Copy the HTTPS forwarding URL (for example https://ab12cd34.ngrok-free.app) into .env as PUBLIC_URL, then restart the server. Now place a call:

curl -X POST http://localhost:8080/call \
  -H "Content-Type: application/json" \
  -d '{"to": "+15559876543"}'

Use your own phone number for the first call. The phone rings, the agent opens with the disclosure, and you can talk to it, interrupt it, and hear it stop.

Tuning the agent

Voice and language

Swap voiceId in TTS_OPTIONS for any voice from the voice library. Text-to-Speech speaks six languages (English, Spanish, French, German, Italian and Portuguese), and the API doesn't detect the language, so set language to match the call.

System prompt and greeting

Both live near the top of server.js. Keep them short: phone replies should be one or two sentences. Keep the automated-call disclosure in the greeting's first sentence; some US states require it.

Turn detection

Leave the turn-detection parameters at their defaults. Universal-3.6 Pro Realtime decides when the callee has finished from what they said as well as from silence, which handles the pauses of someone who has just picked up. Change them only after you've measured a reason to; the voice agent best practices cover when it helps.

Recording, machine detection and time limits

Twilio's Calls API takes optional flags that you almost certainly want in production. Add them to the calls.create() call in /call:

const call = await twilioClient.calls.create({
  to: req.body.to,
  from: TWILIO_FROM_NUMBER,
  url: `${PUBLIC_URL}/twiml`,
  record: true, // keep a recording in Twilio; say so in the greeting
  machineDetection: "Enable", // Twilio adds AnsweredBy to the /twiml request
  timeLimit: 300, // end the call after five minutes, whatever happens
});

record: true saves the call in Twilio. machineDetection: "Enable" tells you whether a person or voicemail answered, so /twiml can branch on it. timeLimit puts a ceiling on any one call, so a stuck conversation can't run up minutes.

Tools (function calling)

Once the conversation works, give the agent tools: book a meeting, look up an account, mark the number as do-not-call. Tools live in the LLM request as OpenAI-style function definitions, so they're a change to reply(), and nothing in the audio bridge changes. The tool calling docs show the request and response shapes, and the cold-calling build adds four tools with argument validation.

Troubleshooting

The phone rings but the call drops right after you answer. Check the call log in the Twilio console. It's usually a TwiML fetch failure: Twilio couldn't reach PUBLIC_URL/twiml because ngrok stopped, the URL still says localhost, or it starts with http:// instead of https://.

Twilio connects but the agent never speaks. Look for [tts] session and [stt] session in the server logs. An authorization error means the AssemblyAI key is wrong. No AssemblyAI lines at all means the environment isn't loaded: check that .env sits next to server.js.

The agent's voice sounds sped up, slowed down or like static. Both ends must be μ-law at 8 kHz: encoding=pcm_mulaw and sample_rate=8000 on the speech-to-text URL, and encoding: "pcm_mulaw" with sampleRate: 8000 in TTS_OPTIONS. The session's default is 24 kHz 16-bit PCM, which Twilio plays as noise.

The agent keeps talking after you interrupt. Make sure bargeIn() sends the clear event. Without it, Twilio plays out everything it has buffered.

Twilio trial accounts only call verified numbers. That's a Twilio limitation. Verify the number in the Twilio console or upgrade the account.

Frequently asked questions

How do I make a voice agent that places outbound phone calls?

Use Twilio's Calls API to dial the number with a url that returns TwiML containing <Connect><Stream>. When the callee answers, Twilio opens a Media Streams WebSocket to your server. Forward the caller's audio to Universal-3.6 Pro Realtime, send each finished turn to an LLM through LLM Gateway, and speak the reply with an AssemblyAI Text-to-Speech session that returns 8 kHz μ-law, base64-encoded into Twilio media events.

What audio format should I use for a Twilio voice agent?

G.711 μ-law at 8 kHz in both directions. Open Universal-3.6 Pro Realtime with encoding=pcm_mulaw and sample_rate=8000, and the Text-to-Speech session with encoding set to pcm_mulaw and a sample rate of 8000. Twilio's audio then passes through untouched, and you pace the agent's audio at 8,000 bytes a second.

How do I handle barge-in on a phone call?

When the first words of a new caller turn arrive from speech-to-text, cancel the reply in flight, call cancel() on the Text-to-Speech session to stop synthesis, and send Twilio {"event": "clear", "streamSid": ...} over the Media Streams WebSocket so it drops the audio it has buffered.

Why does the agent need to speak first on an outbound call?

The person answering has no idea why their phone rang. Speak the greeting as soon as Twilio's start event arrives, identify who's calling and that it's an automated call, and state the reason in one sentence. Silence at the start of an outbound call gets hung up on.

How do I authenticate to AssemblyAI from a Node.js server?

Send your API key in the Authorization header of the speech-to-text WebSocket upgrade request, and pass it to new AssemblyAI({ apiKey }) for Text-to-Speech and as the apiKey of the OpenAI client pointed at LLM Gateway. Keep the key on the server; Twilio never sees it.

Is it legal to call people with an AI voice agent?

It depends on the jurisdiction and the use case. In the US, the TCPA, the national and state do-not-call registries, and state rules on recording and AI disclosure all apply to automated calls, and other regions have their own rules. Disclose that the call is automated, honor opt-out requests immediately, and have counsel review the program before you call anyone. This is general guidance, not legal advice.

How much does an outbound voice agent cost to run?

You pay AssemblyAI for streaming speech-to-text while the socket is open, for LLM Gateway by model usage, and for Text-to-Speech by input characters, at $15 per 1M characters. Current rates are on the pricing page. Twilio bills separately for outbound minutes and the phone number.