Build a Voice Agent in 5 Minutes with AssemblyAI

This is the fastest path from zero to a voice agent you can talk to: one Python file, one AssemblyAI API key, and about five minutes. You speak, Universal-3.6 Pro Realtime transcribes you, an LLM writes the reply through LLM Gateway, and the AssemblyAI Text-to-Speech API says it out loud. Turn detection and barge-in are included.

Want the longer version, with a full walkthrough of each component? Read how to easily build a voice agent with AssemblyAI. This post is the shortcut.

What you're building

A voice agent needs three things: ears, a brain, and a voice. Here, all three come from one SDK and run on one API key:

The script streams your microphone to speech-to-text. When the model decides you've finished your turn, the transcript goes to the LLM, and the reply streams back through text-to-speech to your speakers. If you talk over the agent, it stops.

Prerequisites

• Python 3.9 or later.
• A microphone. Use headphones: a terminal app gets no echo cancellation, so the agent will hear itself through your speakers and interrupt itself.
• An AssemblyAI API key. LLM Gateway uses the same key; check that your plan includes it.

On macOS, install PortAudio so PyAudio can reach your mic (on Debian or Ubuntu, apt install portaudio19-dev):

brew install portaudio

Quick start

1. Install the SDK

Text-to-speech is in the AssemblyAI Python SDK from version 1.8.2, so install with -U:

pip install -U assemblyai pyaudio

2. Set your API key

export ASSEMBLYAI_API_KEY=your_key_here

3. Save the agent

Save this as agent.py. It's the whole agent:

# agent.py
import os
import queue
import threading

import assemblyai as aai
import pyaudio
from assemblyai.streaming.v3 import (
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
)

API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
MIC_RATE, SPEAKER_RATE, CHUNK = 16000, 24000, 800  # 50 ms mic chunks

# Listen, think, speak: three AssemblyAI clients on one API key
stt = RealTimeTranscriber(RealTimeTranscriberOptions(), api_key=API_KEY)
llm = aai.LLMGateway(api_key=API_KEY)
tts = aai.TTSSession(
    aai.TTSConfig(voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", language="english"),  # Audrey
    api_key=API_KEY,
)

pa = pyaudio.PyAudio()
mic = pa.open(format=pyaudio.paInt16, channels=1, rate=MIC_RATE, input=True, frames_per_buffer=CHUNK)
speaker = pa.open(format=pyaudio.paInt16, channels=1, rate=SPEAKER_RATE, output=True)

messages = [{"role": "system", "content": "You are a friendly voice assistant. "
             "Reply in one or two short spoken sentences, with no lists or markdown."}]
turns = queue.Queue()
speaking, interrupted = threading.Event(), threading.Event()


def on_turn(client, event):
    text = event.transcript.strip()
    if speaking.is_set():
        if len(text.split()) < 2:
            return             # "mhm", "yeah": keep talking
        interrupted.set()      # the user is talking over the agent
    if event.end_of_turn and text:
        print(f"You:   {text}")
        turns.put(text)


def reply(user_text):
    messages.append({"role": "user", "content": user_text})
    completion = llm.chat.completions.create(
        model="claude-haiku-4-5-20251001", messages=messages, max_tokens=200
    )
    text = completion.choices[0].message.content or ""
    messages.append({"role": "assistant", "content": text})
    print(f"Agent: {text}")

    interrupted.clear()
    speaking.set()
    for chunk in tts.speak(text):  # raw 24 kHz 16-bit PCM, streamed as it is made
        if interrupted.is_set():
            tts.cancel()           # barge-in: drop the rest of the reply
            break
        speaker.write(chunk)
    speaking.clear()


def mic_audio():
    while True:
        yield mic.read(CHUNK, exception_on_overflow=False)


stt.on(RealTimeEvents.Turn, on_turn)
stt.connect(RealTimeParameters(speech_model="universal-3-6-pro", sample_rate=MIC_RATE))
tts.connect()
threading.Thread(target=stt.stream, args=(mic_audio(),), daemon=True).start()
print("Listening. Say something! (Ctrl+C to quit)")

try:
    while True:
        try:
            reply(turns.get(timeout=30))
        except queue.Empty:
            tts.keep_alive()  # the TTS session closes after 2 idle minutes
except KeyboardInterrupt:
    pass
finally:
    stt.disconnect(terminate=True)
    tts.terminate()
    pa.terminate()

4. Run it

python agent.py

Put on your headphones, wait for Listening, and start talking. Your words and the agent's replies print to the terminal as the conversation goes.

That's the whole thing. Five minutes from install to conversation.

How it works

The file is short because each service does its own heavy lifting. Three pieces do the work.

1. Listen. A background thread reads 50 ms chunks from the microphone and hands them to stt.stream(). Universal-3.6 Pro Realtime sends back Turn events as you speak, partial transcripts first, then one with end_of_turn set when it decides you've finished. Only finished turns go on the turns queue.

2. Think. The main loop takes each finished turn and sends the conversation so far to the LLM through LLM Gateway. The system prompt asks for one or two short spoken sentences, because everything the model writes gets read aloud.

3. Speak. One TTSSession stays open for the whole conversation. tts.speak(text) sends the reply and yields raw 24 kHz, 16-bit audio chunks as they're generated, so playback starts before the full reply is synthesized. If you talk over the agent, on_turn sets interrupted, and the playback loop calls tts.cancel() to drop the rest of the reply.

What the stack handles for you

You don't write code for any of these:

• Turn detection. Universal-3.6 Pro Realtime decides when you've finished speaking, so the agent doesn't jump in during a short pause. The defaults suit conversation, so the script sets no silence timers.
• Partial transcripts. Turn events stream while you talk. The script uses them for barge-in; you could also show them live.
• Streaming speech. Text-to-speech starts sending audio before the whole reply is synthesized, and cancel() stops it mid-sentence.
• Conversational voices. AssemblyAI Text-to-Speech is built for voice agents, trained on millions of hours of audio, including real conversational data. It delivers frontier-level quality, runs faster and costs less than other production voice agent TTS models, and has unlimited concurrency.

Tuning the agent

Pick a different voice

Swap voice_id for any voice in the voice library. There are thousands of voices across six languages: English, Spanish, French, German, Italian, and Portuguese. Set language to the language the agent should speak:

tts = aai.TTSSession(
    aai.TTSConfig(
        voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3",  # Audrey; replace with your pick
        language="english",
    ),
    api_key=API_KEY,
)

Change the brain

Edit the system prompt in messages to give the agent a job, and change model to any model on LLM Gateway's available models list. To make the agent call your own functions, follow our Python voice agent with tool calling tutorial.

Boost domain-specific terms

If your conversations include rare words, such as product names, medical terms, or customer names, pass them as keyterms_prompt to bias speech recognition toward them:

stt.connect(
    RealTimeParameters(
        speech_model="universal-3-6-pro",
        sample_rate=MIC_RATE,
        keyterms_prompt=["AssemblyAI", "Universal-3.6 Pro", "LLM Gateway"],
    )
)

Common issues

The agent keeps interrupting itself. Your microphone is picking up the agent's voice through your speakers. Use headphones, or move the client to a browser, where getUserMedia gives you echo cancellation.

Audio sounds sped up, slowed down, or garbled. The two directions use different rates: the microphone stream must be 16 kHz to match sample_rate=MIC_RATE, and the speaker stream must be 24 kHz to match the text-to-speech output. Check both pa.open() calls.

An authentication error on connect. Your API key is missing or wrong. Copy a fresh one from the AssemblyAI dashboard and run the export again in the same terminal.

The LLM call fails but transcription works. Check that your plan includes LLM Gateway. Or replace the llm call with your own OpenAI client; the rest of the file doesn't change.

Where to go next

Once the basic agent is talking, layer in capabilities:

• Add tools. Give the agent the ability to look things up and take actions: see build a real-time voice agent in Python with tool calling.
• Go deeper on the stack. Read how to easily build a voice agent with AssemblyAI for the full component-by-component build.
• Put it on the phone. Both services accept 8 kHz μ-law audio, the format Twilio streams; see our Twilio voice agent guide.
• Tune for production. The voice agent best practices cover turn detection, prompting, and mid-session updates.

Frequently asked questions

What's the fastest way to build a voice agent with AssemblyAI?

Use the AssemblyAI Python SDK for all three parts: RealTimeTranscriber with speech_model="universal-3-6-pro" to listen, aai.LLMGateway to generate replies, and aai.TTSSession to speak them. Stream your microphone in, send each finished turn to the LLM, and play the audio the session returns. The whole agent fits in one Python file and runs on one API key.

How is this different from streaming speech-to-text alone?

Streaming speech-to-text only gives you transcripts. To hold a conversation you also need an LLM to decide what to say and text-to-speech to say it. This quickstart connects all three with the same SDK and key, and adds turn handling and barge-in on top.

How do I authenticate?

Set ASSEMBLYAI_API_KEY in your environment; the script passes it to each client. For browser apps where you can't expose your key, mint a short-lived temporary token on your server instead.

What audio formats does it use?

The microphone sends 16 kHz, 16-bit mono PCM to speech-to-text. Text-to-speech returns raw 24 kHz, 16-bit mono PCM, its native rate, with no file header. For phone calls, both services also accept 8 kHz μ-law, the format Twilio uses.

Can the agent call tools or functions?

Yes. Tools move into the LLM request: pass OpenAI-style tool definitions to LLM Gateway, run the function the model asks for, and send the result back. The tool calling tutorial builds this step by step.

Why does my voice agent keep interrupting itself?

The microphone is hearing the agent through your speakers. Use headphones, or run the client in a browser with getUserMedia({ audio: { echoCancellation: true } }) for built-in echo cancellation.

How much does it cost?

You can sign up and get an API key for free. Speech-to-text, LLM Gateway, and text-to-speech are each billed on usage, and text-to-speech is $15 per 1M characters; see the AssemblyAI pricing page for current rates.