Build an AI voice agent for customer support in Python

You can build a customer support voice agent in Python in an afternoon from three AssemblyAI pieces on one API key: Universal-3.6 Pro Realtime to hear the caller, an LLM through LLM Gateway to decide what to say, and AssemblyAI Text-to-Speech to say it. The agent transcribes the caller, streams the reply into speech as the LLM writes it, stops talking when the caller cuts in, and hands off to a person with a written summary. This tutorial builds three workflows (policy answers, collecting caller details, and escalation to a human) plus the two things that decide whether a support agent actually works in production: entity accuracy and what the agent does with silence.

Tier-1 customer support is mostly the same five conversations on repeat. When will this ship. How do I return it. It arrived broken. When are you open. Let me talk to a person. They are high volume, low variance, and structured, which makes them a bad use of a human agent's day and a very good fit for a voice agent. Teams building in this category, like Gorgias, the AI helpdesk for Shopify and DTC merchants, are automating exactly this tier.

Why most support voice agents fail

Two failure modes, and neither one is the LLM.

The first is entity accuracy. A support conversation is full of details that have to be exactly right: names, email addresses, phone numbers, order IDs, ZIP codes. Headline word error rate barely touches this. A model can score well on overall WER and still turn "jane at example dot com" into an address your database has never seen, or drop a digit from a callback number. When that happens nothing fails loudly. The handoff summary carries the wrong email, the human agent can't find the account, and the customer starts over. Entity error rate is the number that predicts whether your agent resolves the ticket.

Here is how Universal-3.6 Pro Realtime performs on AssemblyAI's English voice-agent benchmark, which scores models on 12,460 scripted voice-agent scenarios rather than read speech. Lower is better.

Read the phone-numbers row first, because that is your callback field: 2.4% against Deepgram Flux EN's 11.5%. Then read the names row, which is your identity check: 10.9% against 29.0%. On a model that misses roughly three names in ten, a handoff summary doesn't degrade gracefully. It sends the human agent looking for the wrong person. More benchmark detail lives on the benchmarks page.

The second failure is dead air. In a cascaded pipeline every stage can add silence: waiting for the whole LLM reply before synthesizing it, opening a new speech connection for every turn, or talking straight through a caller who is trying to interrupt. Callers read silence as a dropped call and start talking over the agent. The fix is in how you wire the stages together, covered in Step 3: stream the LLM's tokens into one Text-to-Speech session that stays open for the whole call, and cancel it the moment the caller speaks.

What you'll build

Three workflows, one agent:

1. Policy answers: the caller asks about shipping times, returns, damaged items or hours, and the agent answers from a short knowledge block, never from guesswork.
2. Caller details: when a person needs to take over, the agent collects the caller's name, account email and callback number, and reads each one back before it moves on.
3. Escalation to a human: the agent recognizes frustration or an out-of-scope request, writes a one-sentence summary for the human agent, and hands off.

The stack is Universal-3.6 Pro Realtime over the Streaming Speech-to-Text WebSocket, an LLM through LLM Gateway (OpenAI-compatible, so the OpenAI Python SDK works unchanged), AssemblyAI Text-to-Speech through the AssemblyAI Python SDK, and Python 3.9+. Everything runs on one AssemblyAI API key. AssemblyAI Text-to-Speech delivers frontier-level quality, runs faster and costs less than other production voice agent TTS models, and has unlimited concurrency. This agent deliberately calls no tools: one streamed LLM call per turn keeps the first word of every reply as early as possible. When you need the agent to act on your systems, such as looking up an order, the companion post on building a support agent that looks up orders with tool calling adds that on a phone line.

Setup

pip install -U "assemblyai>=1.8.2" openai "websockets>=14" pyaudio python-dotenv

Put your key in .env. The same key covers speech-to-text, LLM Gateway and Text-to-Speech:

ASSEMBLYAI_API_KEY=your_key_here

LLM Gateway may require a paid account. If you'd rather start with your own OpenAI key, point the OpenAI client at OpenAI instead of the gateway in Step 3; nothing else changes.

Step 1: Give the agent its support knowledge

Everything below goes into one file, agent.py, in the order shown. Start with what the agent is allowed to say. A support agent that answers from a short, explicit knowledge block is far easier to trust than one that answers from whatever the model remembers about return policies in general. Keep it to the facts a tier-1 agent actually quotes, and keep the wording the way you'd want it read aloud.

# agent.py: support knowledge the agent may quote. Keep it short and factual.
SUPPORT_KNOWLEDGE = """
Shipping: standard orders ship within 2 business days; expedited orders ship the same day
if placed before 1 p.m. Eastern. Tracking emails go out when the label is created.
Returns: unused items can be returned within 30 days for a full refund. Start a return at
acme.example/returns. Refunds post 5 to 7 business days after the warehouse receives the item.
Damaged items: no need to return them; a teammate sends a replacement.
Hours: phone support is staffed 8 a.m. to 8 p.m. Eastern, Monday to Saturday.
"""

GREETING = "Thanks for calling Acme Corp, this is Avery. How can I help?"

# The model starts its reply with this marker when a person should take over.
HANDOFF = "HANDOFF:"
HANDOFF_LINE = (
    "I'm connecting you with a teammate now. They'll have the details, "
    "so you won't need to repeat yourself."
)

The HANDOFF marker is how the agent escalates without a tool call. The model starts its reply with the marker and a summary, the code catches it before any of it reaches the speaker, and the caller hears a fixed handoff line instead. The summary goes to your logs or your ticketing system.

Why there are no tools in this agent

Tool calling adds a round trip: the model asks for a function, you run it, and the model writes the reply in a second request. That's the right trade when the agent needs data it doesn't have, like an order status. For policy questions it only adds latency, because the answer is already in the prompt. Keeping this agent to one streamed LLM call per turn means the caller hears the start of the answer while the model is still writing the end of it.

The other thing that keeps a support agent honest is where its facts come from. Everything this agent says about policy comes from SUPPORT_KNOWLEDGE, and everything it says about the caller comes from what the caller said and confirmed. Those are the only two sources, and the prompt in Step 2 says so.

Step 2: Write the system prompt

The system prompt is where you encode the workflow. Be specific about the read-back behavior and absolutely explicit about fabrication.

SYSTEM_PROMPT = f"""You are Avery, a phone support agent for Acme Corp.

Answer from this knowledge only:
{SUPPORT_KNOWLEDGE}

Questions you can answer: shipping times, returns and refunds, damaged items, hours.
If the answer is not in the knowledge above, say you don't have that information and
offer to connect the caller with a teammate. Never guess at a policy, date or amount.

Caller details: if the caller needs a teammate, collect their full name, the email on
their account and a callback number. Read each one back and wait for a yes before you
move on. Spell back email addresses.

Handoff: when the caller asks for a person, sounds frustrated, or needs something you
can't answer (an order lookup, a refund decision, an account change), reply with one
line only: {HANDOFF} followed by a one-sentence summary for the human agent that
includes the caller's confirmed details. Say nothing else on that turn.

Style: this is a phone call. One or two short sentences per reply. No lists, no
markdown, no filler."""

The no-guessing rule is the single most important line in the prompt. An agent that invents a refund window is worse than an agent that says it doesn't know and offers a person.

Step 3: Connect speech-to-text, the LLM and Text-to-Speech

The pipeline has three connections, all opened once per conversation: a WebSocket to wss://streaming.assemblyai.com/v3/ws for Universal-3.6 Pro Realtime, an OpenAI client pointed at LLM Gateway, and one AssemblyAI Text-to-Speech session. Microphone audio streams in as 16 kHz PCM, and each finished caller turn (a Turn event with end_of_turn set) becomes an LLM request. The LLM's tokens go into the Text-to-Speech session as they arrive, and the session's audio plays at its native 24 kHz.

import asyncio
import json
import os
from urllib.parse import urlencode

import assemblyai as aai
import pyaudio
import websockets
from dotenv import load_dotenv
from openai import AsyncOpenAI

load_dotenv()
API_KEY = os.environ["ASSEMBLYAI_API_KEY"]

MIC_RATE = 16000      # what the microphone sends to Universal-3.6 Pro Realtime
SPEAKER_RATE = 24000  # AssemblyAI TTS output: raw 24 kHz 16-bit mono PCM
CHUNK = 800           # 50 ms of microphone audio per frame

STT_URL = "wss://streaming.assemblyai.com/v3/ws?" + urlencode({
    "speech_model": "universal-3-6-pro",
    "encoding": "pcm_s16le",
    "sample_rate": MIC_RATE,
    "voice_focus": "near-field",   # headset or handset; "far-field" for room mics
    "agent_context": GREETING,     # the first thing the caller hears
})

# LLM Gateway is OpenAI-compatible and takes the same AssemblyAI key.
llm = AsyncOpenAI(base_url="https://llm-gateway.assemblyai.com/v1", api_key=API_KEY)
LLM_MODEL = "gpt-5-mini"  # any model on the Available models page that supports stream

TTS_CONFIG = aai.TTSConfig(
    voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3",  # Audrey
    language="english",
)


class SupportAgent:
    def __init__(self, stt_ws, tts, speaker):
        self.stt_ws = stt_ws
        self.tts = tts          # one TTS session for the whole conversation
        self.speaker = speaker
        self.messages = [
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "assistant", "content": GREETING},
        ]
        self.reply_task = None
        self.call_over = asyncio.Event()

    def speaking(self):
        return self.reply_task is not None and not self.reply_task.done()

    async def play(self, chunks):
        async for chunk in chunks:
            await asyncio.to_thread(self.speaker.write, chunk)

    async def say(self, text):
        await self.set_agent_context(text)
        await self.play(self.tts.speak(text))

    async def set_agent_context(self, text):
        # Tell the speech model what the agent just asked, so a short or spelled
        # answer ("j-a-n-e at example dot com") resolves against the question.
        await self.stt_ws.send(json.dumps({
            "type": "UpdateConfiguration",
            "agent_context": text[-1750:],
        }))

    async def reply(self, caller_text):
        self.messages.append({"role": "user", "content": caller_text})
        stream = await llm.chat.completions.create(
            model=LLM_MODEL, messages=self.messages, stream=True,
        )

        # Hold back the first few characters until we know whether this is a
        # handoff; after that, every token goes straight to the TTS session.
        text, decided, handoff = "", False, False
        async for part in stream:
            token = part.choices[0].delta.content if part.choices else None
            if not token:
                continue
            text += token
            if decided:
                if not handoff:
                    await self.tts.generate(token)  # joined as sent, spaces included
            elif len(text.lstrip()) >= len(HANDOFF):
                decided = True
                handoff = text.lstrip().startswith(HANDOFF)
                if not handoff:
                    await self.tts.generate(text)

        if handoff:
            summary = text.strip()[len(HANDOFF):].strip()
            print("[handoff] for the human agent:", summary)
            self.messages.append({"role": "assistant", "content": HANDOFF_LINE})
            await self.say(HANDOFF_LINE)
            self.call_over.set()
            return

        if not text.strip():
            return
        if not decided:
            await self.tts.generate(text)  # a reply shorter than the marker
        self.messages.append({"role": "assistant", "content": text})
        flush_id = await self.tts.flush()
        print("agent:", text.strip())
        await self.set_agent_context(text)
        await self.play(self.tts.audio(flush_id))

    async def interrupt(self):
        """Barge-in: stop the LLM call or the playback, and drop queued audio."""
        if self.speaking():
            self.reply_task.cancel()
            await self.tts.cancel()


async def send_microphone(stt_ws, mic):
    while True:
        data = await asyncio.to_thread(mic.read, CHUNK, exception_on_overflow=False)
        await stt_ws.send(data)


async def listen(agent):
    async for raw in agent.stt_ws:
        msg = json.loads(raw)
        if msg.get("type") != "Turn":
            continue
        transcript = msg.get("transcript", "").strip()
        if not transcript:
            continue
        # Two or more words while the agent talks is a real interruption;
        # a lone "mhm" or "okay" is not.
        if agent.speaking() and len(transcript.split()) >= 2:
            await agent.interrupt()
        if msg.get("end_of_turn"):
            print("caller:", transcript)
            await agent.interrupt()
            agent.reply_task = asyncio.create_task(agent.reply(transcript))


async def keep_tts_alive(tts):
    while True:
        await asyncio.sleep(60)
        await tts.keep_alive()  # the session closes after 2 idle minutes


async def main():
    audio = pyaudio.PyAudio()
    mic = audio.open(format=pyaudio.paInt16, channels=1, rate=MIC_RATE,
                     input=True, frames_per_buffer=CHUNK)
    speaker = audio.open(format=pyaudio.paInt16, channels=1, rate=SPEAKER_RATE,
                         output=True)

    async with websockets.connect(
        STT_URL, additional_headers={"Authorization": API_KEY}
    ) as stt_ws, aai.AsyncTTSSession(TTS_CONFIG, api_key=API_KEY) as tts:
        agent = SupportAgent(stt_ws, tts, speaker)
        print("agent:", GREETING)
        agent.reply_task = asyncio.create_task(agent.say(GREETING))
        tasks = [
            asyncio.create_task(send_microphone(stt_ws, mic)),
            asyncio.create_task(listen(agent)),
            asyncio.create_task(keep_tts_alive(tts)),
        ]
        try:
            await agent.call_over.wait()
        finally:
            for task in tasks:
                task.cancel()
            await stt_ws.send(json.dumps({"type": "Terminate"}))

    mic.close()
    speaker.close()
    audio.terminate()


if __name__ == "__main__":
    asyncio.run(main())

Run it with python agent.py and wear headphones: a laptop microphone hears the laptop speakers, and the agent will try to answer itself.

Four things worth calling out.

One Text-to-Speech session per conversation. The session opens before the greeting and every reply is a request on it, so no turn pays for a new connection. A session closes after two minutes without a frame, which is what keep_tts_alive() is for, and lasts at most an hour. Rate limits count new sessions per minute, so a long conversation on one open session costs nothing extra against them. See Limits.

Streamed tokens, one flush. generate() sends each token as the LLM produces it, and flush() closes the request once the reply is complete. The server starts speaking before the flush, as soon as it holds complete sentences, so long replies don't wait for the last word. Tokens are joined exactly as sent, which is why the code passes the LLM's deltas through untouched, spaces included.

Turns come from the speech model. Universal-3.6 Pro Realtime decides when the caller has finished, and the code acts on end_of_turn directly. There's no voice-activity detector or silence timer of your own to tune, and the defaults are the right place to start.

The voice is set explicitly. voice_id selects the voice, and language tells the session which of its six languages to speak; the API doesn't detect it. Browse the voice library to pick your own.

Let the caller interrupt

Callers interrupt. They answer before the question is over, they correct themselves, and they say "no, wait." An agent that keeps talking through that sounds like an IVR.

listen() treats any turn with two or more words, arriving while the agent is speaking, as a barge-in. interrupt() then cancels the reply task, which stops the LLM call or the playback wherever it is, and calls tts.cancel(), which stops synthesis of the request in flight and drops anything queued behind it. A one-word "mhm" or "okay" doesn't count, so the agent doesn't stop mid-sentence every time the caller acknowledges it. If your callers' yes and no carry weight mid-reply, the Streaming API voice agent guide has a fuller backchannel filter you can drop in.

Tune the speech layer for names, emails and numbers

A few Streaming Speech-to-Text controls target exactly the failure mode in the benchmark table. These are the connection parameters from Step 3:

STT_URL = "wss://streaming.assemblyai.com/v3/ws?" + urlencode({
    "speech_model": "universal-3-6-pro",
    "encoding": "pcm_s16le",
    "sample_rate": MIC_RATE,
    "voice_focus": "near-field",   # headset or handset; "far-field" for room mics
    "agent_context": GREETING,     # the first thing the caller hears
    # "mode": "max_accuracy",      # optional preset; the server default is "balanced"
})

agent_context is the big one. It tells the model what your agent just said, so it can resolve the reply against the question. When your agent has just asked "what's the email on the account?", a fast "j-a-n-e at example dot com" is no longer an open-ended audio problem: the model knows an email address is the expected shape of the answer. It helps most on short replies and spelled-out entities, and short replies and names are most of a support call. The pipeline seeds it with the greeting at connect time and refreshes it after every reply with an UpdateConfiguration message, up to 1,750 characters.

voice_focus isolates the primary speaker and suppresses background noise: near-field for headsets and phone handsets, far-field for rooms, kiosks and drive-thrus. Support callers are rarely in a quiet room.

mode replaces low-level tuning with three presets: min_latency, balanced (the default) and max_accuracy. Start on balanced. Move to max_accuracy if your callers read out long account numbers; move to min_latency only if the conversation feels slow and you've confirmed the speech layer is the cause.

Universal-3.6 Pro Realtime transcribes 32 languages with native code-switching, so a caller who slips between two of them mid-sentence stays transcribed correctly; the supported languages post has the list. AssemblyAI Text-to-Speech speaks six (English, Spanish, French, German, Italian and Portuguese), so set the session's language to match the caller when you route by language.

Step 4: Test the three workflows

Run the script and talk to it. What you're checking is whether the facts and the entities survive.

Policy answer. Ask "how long do refunds take?" The agent should say five to seven business days after the warehouse receives the item, in one or two sentences. Then ask something the knowledge block doesn't cover, like "do you price match?", and check that it says it doesn't have that information rather than inventing a policy.

Caller details. Say "I need to change the address on my account." That's out of scope, so the agent should start collecting your details. Give your email fast, the way people do: "jane at example dot com." The agent should spell it back and wait for a yes. Deliberately mumble the callback number. Deliberately pause in the middle. That's what the model has to handle in production: hesitation, mumbles, accents and ordinary disfluency.

Escalation. Say "just put me through to a person." The agent should say the handoff line, and your terminal should print a one-sentence summary with the details you confirmed. Read that summary carefully. It's the artifact the human agent inherits, and it's the difference between a warm handoff and making the customer start over.

Step 5: Take it to the phone

The laptop microphone is fine for development. Production support runs on phone numbers, and the same three pieces work there with two changes: Twilio Media Streams carries the call to your server as 8 kHz μ-law, so the speech-to-text socket takes encoding pcm_mulaw at a sample rate of 8000, and the Text-to-Speech session returns pcm_mulaw at 8000 Hz so its audio goes back to Twilio without conversion.

The full phone walkthrough, with the Twilio bridge, keypad entry, order lookups through LLM tool calling and warm transfer to a human queue, lives in the companion post on building a support agent that looks up orders. For the bridge on its own, see how to build a voice agent with Twilio and AssemblyAI.

What to harden before production

Short list, because the companion post covers hardening on a live phone line in depth:

• Move the knowledge out of the code. Load SUPPORT_KNOWLEDGE from the same source your help center uses, so a policy change reaches the agent and the website at the same time.
• Send the handoff somewhere real. Replace the print with your ticketing or contact-center API, and attach the transcript of the call so far.
• Log everything against the session IDs. Keep the speech-to-text session id from the Begin message, the Text-to-Speech session's begin.id, and the request_id of every LLM Gateway response, along with transcripts and replies. When a caller says the agent quoted the wrong policy, the log is the only way to tell whether it was the transcript, the prompt or the knowledge block.
• Close sessions when the call ends. Streaming speech-to-text is billed for as long as the socket is open, so always send Terminate, as main() does.

Where to go from there

The pattern generalizes to the rest of the tier-1 queue: hours and holiday schedules, warranty terms, store locations, delivery areas. Each one is a few lines in the knowledge block. When a workflow needs live data, such as an order status, a refund, or a callback slot, that's the point to add LLM tool calling, and the order lookup post shows the tool loop end to end.

Keep the scope structured. Voice agents earn their keep on bounded workflows with a right answer, such as policy questions, lookups, verification, scheduling and triage, and that's also where the accuracy numbers in the benchmark table translate directly into resolved tickets. For the architecture behind this pipeline, read the voice agent architecture guide; for rates, see pricing; and the Text-to-Speech quickstart has the full session reference.

Frequently asked questions

How do I build an AI voice agent for customer support in Python?

Stream microphone audio to Universal-3.6 Pro Realtime over the Streaming Speech-to-Text WebSocket, send each finished caller turn to an LLM through LLM Gateway with a system prompt that holds your support knowledge, and stream the LLM's tokens into an AssemblyAI Text-to-Speech session that stays open for the whole call. All three run on one AssemblyAI API key, and the LLM Gateway is OpenAI-compatible, so the OpenAI Python SDK works with a changed base_url. Most developers get a working prototype in an afternoon.

Why does speech-to-text accuracy matter so much for support voice agents?

Support conversations run on entities: names, emails, phone numbers, order IDs. A single misheard character doesn't fail loudly; it puts the wrong email in the handoff summary or the wrong callback number in the ticket. On AssemblyAI's English voice-agent benchmark, Universal-3.6 Pro Realtime posts a 14.4% entity error rate against Deepgram Flux EN at 30.1% and ElevenLabs Scribe v2 at 18.5%, with phone numbers at 2.4% versus Flux EN's 11.5%. Entity error rate is the number that decides whether the ticket gets resolved.

What is agent_context and why does it help support agents?

agent_context passes your agent's most recent reply, up to 1,750 characters, to the speech model, so the caller's next utterance is transcribed against what was actually asked. If the agent just asked for an email address, the model expects an email address. Set it when you connect and update it after each reply with an UpdateConfiguration message on the open socket. It has the biggest effect on short replies and spelled-out entities, which make up most of a tier-1 support call.

How do I stop a voice agent from talking over the caller?

Watch the Turn events from Universal-3.6 Pro Realtime while the agent is speaking. When the caller says two or more words, cancel the reply task and call cancel() on the Text-to-Speech session, which stops synthesis of the request in flight and drops anything queued. Ignoring one-word acknowledgements like "mhm" keeps the agent from stopping every time the caller agrees with it.

Can I put this customer support voice agent on a phone number?

Yes. Twilio Media Streams sends the call to your server as 8 kHz μ-law, which Universal-3.6 Pro Realtime accepts directly with encoding pcm_mulaw and a sample rate of 8000, and AssemblyAI Text-to-Speech returns pcm_mulaw at 8000 Hz, so audio passes through in both directions without conversion. The companion post on order lookups walks through the full phone deployment, including keypad entry and warm transfer.

What's the best way to handle escalation to a human in a customer support voice agent?

Decide the escalation rules in the system prompt (the caller asks for a person, sounds frustrated, or needs something out of scope) and have the agent produce a one-sentence summary with the caller's confirmed details when it escalates. Send that summary to the tool your human agents work in, so the person who picks up doesn't make the customer start over. Treat the summary as a product surface: it decides whether escalation feels like a handoff or a failure.

How much does it cost to run a customer support voice agent on AssemblyAI?

You pay for each piece separately. Universal-3.6 Pro Realtime streaming speech-to-text is $0.45 per hour, billed for the time the socket is open; Text-to-Speech is $15 per 1M characters, billed on input characters; and LLM Gateway is billed per model. Current rates for all three are on the pricing page.

Do voice agents built with AssemblyAI work with healthcare workflows?

AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure that AAI appropriately safeguards PHI; the platform also holds SOC 2 Type 2, ISO 27001:2022 and PCI DSS v4.0. A voice agent uses several products (speech-to-text, LLM Gateway and Text-to-Speech), so contact sales to confirm which ones your BAA covers before you build. Details on the addendum are on the BAA legal page.