How to build a voice agent with Python in 5 minutes

This tutorial shows you how to build a complete voice agent that listens, thinks, and responds naturally using Python. You'll create a realtime application that processes speech in real-time, generates intelligent responses, and speaks back to users—all in under 100 lines of code.

The voice agent combines three APIs: AssemblyAI's Universal-3 Pror ealtime model for speech-to-text, OpenAI's GPT-4 for conversational AI, and ElevenLabs for natural voice synthesis. Each component streams data to minimize response delays and create smooth, human-like conversations.

What you'll need to get started

You need Python 3.9 or higher, three API keys, and a computer with a microphone and speakers. The setup takes about 2 minutes once you have everything ready.

Install Python dependencies

Open your terminal and run this command to install everything you need:

pip install "assemblyai>=1.0.0" openai "elevenlabs>=1.0.0" pyaudio python-dotenv

PyAudio requires the PortAudio system library: on macOS run brew install portaudio, and on Debian/Ubuntu run apt install portaudio19-dev before installing the Python packages.

Here's what each package does for your voice agent:

• assemblyai: Handles real-time speech recognition with Universal-3 Pro realtime
• openai: Connects to GPT models for smart responses
• elevenlabs: Creates natural-sounding voices
• pyaudio: Provides access to your microphone
• python-dotenv: Loads your API keys from a .env file

Configure your API keys

Create a .env file in your project directory with your API keys:

Code — Text

ASSEMBLYAI_API_KEY=your_assemblyai_key_here
OPENAI_API_KEY=your_openai_key_here
ELEVENLABS_API_KEY=your_elevenlabs_key_here

Never share this file or commit it to version control. Add .env to your .gitignore file to protect your API keys.

What are the components of a voice agent?

A voice agent is a program that talks to you like a human using three connected parts. These parts work together to create conversations: speech-to-text converts your voice into text, a language model thinks about what you said and creates a response, and text-to-speech turns that response back into spoken words.

This pipeline needs to work in real-time to feel natural. When you speak to Siri or Alexa, you expect quick responses—not awkward pauses that break the conversational flow. Here's what each component does in your voice agent:

The difference between good and bad voice agents comes down to speed. Batch processing—where each step waits for the previous one to finish completely—creates those robotic pauses that make conversations feel unnatural.

AssemblyAI's Universal-3 Pro realtime model solves this by processing speech as it happens. You get accurate transcription with minimal delay, making conversations feel smooth and responsive.

Set up speech-to-text with AssemblyAI

Speech recognition forms the foundation of your voice agent. AssemblyAI's Universal-3 Pro realtime API listens to your microphone and converts speech to text in real-time. The SDK handles all WebSocket complexity automatically—no manual connection management required.

Create a new file called voice_agent.py and add this code:

from assemblyai.streaming.v3 import (
    BeginEvent,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    TurnEvent,
    TerminationEvent,
)

from dotenv import load_dotenv
import os

load_dotenv()

class VoiceAgent:

    def __init__(self):
        self.client = RealTimeTranscriber(
            RealTimeTranscriberOptions(),
            api_key=os.getenv('ASSEMBLYAI_API_KEY'),
        )

        self.client.on(RealTimeEvents.Begin, self.on_begin)
        self.client.on(RealTimeEvents.Turn, self.on_turn)
        self.client.on(RealTimeEvents.Termination, self.on_terminated)
        self.client.on(RealTimeEvents.Error, self.on_error)

        self.is_processing = False

    def on_begin(self, client: RealTimeTranscriber, event: BeginEvent):
        print("Listening... Start speaking!")

    def on_turn(self, client: RealTimeTranscriber, turn: TurnEvent):
        if not turn.transcript:
            return

        if turn.end_of_turn:
            print(f"You said: {turn.transcript}")
            # AI processing added in next section
        else:
            print(f"Hearing: {turn.transcript}", end="\r")

    def on_error(self, client: RealTimeTranscriber, error: RealTimeError):
        print(f"Error: {error}")

    def on_terminated(self, client: RealTimeTranscriber, event: TerminationEvent):
        print("Connection closed")

This code creates a real-time transcription system that gives you two types of output. Partial transcripts (where end_of_turn is False) show you what the system is hearing as you speak, and final transcripts (where end_of_turn is True) provide the complete sentence when you pause.

Universal-3 Pro uses punctuation-based turn detection—it ends a turn when it detects terminal punctuation (. ? !) after a natural pause. This means you don't need to press buttons or give special commands—just speak naturally and pause.

Connect the language model

The language model serves as the thinking component of your voice agent—it comprehends user input and determines appropriate responses. OpenAI's GPT-4 produces output incrementally, enabling ElevenLabs to initiate audio playback before generating the complete message.

Incorporate this OpenAI connection into your VoiceAgent class:

from openai import OpenAI

class VoiceAgent:

    def __init__(self):
        # Previous code...

        self.openai_client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))

        self.conversation = [
            {"role": "system", "content": """You are a helpful voice assistant.

Keep responses short and conversational.

Talk like you're having a normal conversation with someone."""}
        ]

    def process_with_llm(self, user_text):
        self.conversation.append({"role": "user", "content": user_text})

        response_text = ""

        stream = self.openai_client.chat.completions.create(
            model="gpt-4",
            messages=self.conversation,
            stream=True,
            temperature=0.7,
            max_tokens=150
        )

        print("Assistant: ", end="")

        for chunk in stream:
            if chunk.choices[0].delta.content:
                content = chunk.choices[0].delta.content
                response_text += content
                print(content, end="", flush=True)

        print()

        self.conversation.append({"role": "assistant", "content": response_text})

        self.speak(response_text)

The conversation history maintains the complete dialogue, allowing your agent to understand context across multiple exchanges. The system prompt directs GPT-4 to produce concise, natural-sounding answers—verbose responses feel unnatural when delivered through voice.

Add text-to-speech output

Text-to-speech concludes your voice agent by transforming AI-generated text into human-quality audio. ElevenLabs delivers expressive voice synthesis that begins playback while additional content continues generating.

Integrate voice synthesis into your VoiceAgent class:

from elevenlabs.client import ElevenLabs
from elevenlabs import stream as play_stream
import threading

class VoiceAgent:

    def __init__(self):
        # Previous code...

        self.elevenlabs_client = ElevenLabs(api_key=os.getenv('ELEVENLABS_API_KEY'))

        self.voice_id = "EXAVITQu4vr4xnSDxMaL"  # Sarah voice

    def speak(self, text):

        def generate_and_play():
            try:
                audio_stream = self.elevenlabs_client.text_to_speech.stream(
                    voice_id=self.voice_id,
                    text=text,
                    model_id="eleven_turbo_v2_5",
                )

                play_stream(audio_stream)

            except Exception as e:
                print(f"Voice error: {e}")

        thread = threading.Thread(target=generate_and_play, daemon=True)
        thread.start()

ElevenLabs provides multiple voice options with unique characteristics:

• Sarah (EXAVITQu4vr4xnSDxMaL): Polished, professional female tone (demonstrated here)
• Josh (TxGEqnHWrfWFTfGW9XjX): Approachable, genuine male tone
• Elli (MF3mGyEYCl7XYWbV9V6O): Vibrant, spirited female tone

The background thread prevents voice generation from blocking program execution. As audio produces and plays, your agent simultaneously monitors for incoming speech.

Build the complete voice agent

Here represents your full voice_agent.py implementation:

import os
import sys
import threading

import pyaudio

from assemblyai.streaming.v3 import (
    BeginEvent,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    TurnEvent,
    TerminationEvent,
)

from elevenlabs.client import ElevenLabs
from elevenlabs import stream as play_stream
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

FRAMES_PER_BUFFER = 800  # 50 ms at 16 kHz

class VoiceAgent:

    def __init__(self):
        self.client = RealTimeTranscriber(
            RealTimeTranscriberOptions(),
            api_key=os.getenv('ASSEMBLYAI_API_KEY'),
        )

        self.client.on(RealTimeEvents.Begin, self.on_begin)
        self.client.on(RealTimeEvents.Turn, self.on_turn)
        self.client.on(RealTimeEvents.Termination, self.on_terminated)
        self.client.on(RealTimeEvents.Error, self.on_error)

        self.elevenlabs_client = ElevenLabs(api_key=os.getenv('ELEVENLABS_API_KEY'))
        self.openai_client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))

        self.is_processing = False
        self.voice_id = "EXAVITQu4vr4xnSDxMaL"

        self.conversation = [
            {"role": "system", "content": """You are a helpful voice assistant.

Keep responses short and conversational.

Talk like you're having a normal conversation with someone."""}
        ]

    def on_begin(self, client: RealTimeTranscriber, event: BeginEvent):
        print("\n Voice Agent Ready! Start speaking...\n")

    def on_turn(self, client: RealTimeTranscriber, turn: TurnEvent):
        if not turn.transcript:
            return

        if turn.end_of_turn:
            print("\r" + " " * 50 + "\r", end="")
            print(f"You: {turn.transcript}")

            if not self.is_processing:
                self.is_processing = True
                self.process_with_llm(turn.transcript)
                self.is_processing = False
        else:
            print(f"Listening: {turn.transcript}...", end="\r")

    def on_error(self, client: RealTimeTranscriber, error: RealTimeError):
        print(f"\n Error: {error}\n")

    def on_terminated(self, client: RealTimeTranscriber, event: TerminationEvent):
        print("\n Voice Agent stopped\n")

    def process_with_llm(self, user_text):
        self.conversation.append({"role": "user", "content": user_text})

        response_text = ""

        stream = self.openai_client.chat.completions.create(
            model="gpt-4",
            messages=self.conversation,
            stream=True,
            temperature=0.7,
            max_tokens=150
        )

        print("Agent: ", end="")

        for chunk in stream:
            if chunk.choices[0].delta.content:
                content = chunk.choices[0].delta.content
                response_text += content
                print(content, end="", flush=True)

        print()

        self.conversation.append({"role": "assistant", "content": response_text})

        self.speak(response_text)

    def speak(self, text):

        def generate_and_play():
            try:
                audio_stream = self.elevenlabs_client.text_to_speech.stream(
                    voice_id=self.voice_id,
                    text=text,
                    model_id="eleven_turbo_v2_5",
                )

                play_stream(audio_stream)

            except Exception as e:
                print(f"Voice error: {e}")

        voice_thread = threading.Thread(target=generate_and_play)
        voice_thread.daemon = True
        voice_thread.start()

    def start(self):
        self.client.connect(
            RealTimeParameters(
                sample_rate=16000,
                speech_model="u3-rt-pro",
            )
        )

        # Capture microphone audio with PyAudio and feed raw PCM16 chunks
        # to the transcriber via client.stream().
        pa = pyaudio.PyAudio()
        mic = pa.open(
            format=pyaudio.paInt16,
            channels=1,
            rate=16000,
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
        )
        try:
            while True:
                self.client.stream(mic.read(FRAMES_PER_BUFFER, exception_on_overflow=False))
        except KeyboardInterrupt:
            pass
        finally:
            mic.stop_stream()
            mic.close()
            pa.terminate()
            self.stop()

    def stop(self):
        print("\nStopping voice agent...")
        self.client.disconnect(terminate=True)
        sys.exit(0)

if __name__ == "__main__":
    agent = VoiceAgent()
    agent.start()

This complete implementation includes error handling, conversation memory, and clean shutdown. "The agent remembers what you've talked about during each session, enabling natural back-and-forth conversations."

Run your voice agent

Execute your voice agent using this command:

python voice_agent.py

When "Voice Agent Ready! Start speaking..." displays, the agent is listening. Speak into your microphone and the agent responds with both text and voice output.

Test your agent with these prompts:

• "What's the weather like today?"
• "Tell me a quick joke"
• "Help me plan dinner"
• "Explain how WiFi works simply"

Common issues and fixes:

• No microphone input: Verify system permissions and microphone settings
• Slow responses: Check internet connectivity and consider swapping gpt-3.5-turbo for faster processing
• Voice cuts off: Add a small pause after TTS playback or verify your ElevenLabs API quota

Final words

You've constructed a complete realtime voice agent that processes audio in real-time and participates in natural dialogue. This implementation integrates speech recognition, artificial intelligence processing, and voice synthesis into a unified application showcasing contemporary Voice AI capabilities.

"AssemblyAI's Universal-3 Pro realtimemodel makes this possible by providing the accuracy and speed that voice agents require." The SDK manages intricate WebSocket connectivity and audio handling, freeing you to concentrate on application development rather than infrastructure details.

To expand functionality, review the Universal-3 Pro realtime documentation for capabilities including keyterm prompting, speaker diarization, and live configuration adjustments—all functioning without agent restart.

Frequently asked questions

Do I need WebSocket knowledge to build this voice agent?

No. The AssemblyAI Python SDK manages WebSocket connection handling, reconnection procedures, and audio realtime protocol automatically. You implement event handlers while the SDK manages remaining details.

How much does running this voice agent cost per hour?

This voice agent costs approximately $0.50–$1.00 per conversation hour across all services. "AssemblyAI charges about $0.45/hr for Universal-3 Pro realtime transcription, OpenAI costs roughly $0.30/hour for GPT-4 responses, and ElevenLabs runs about $0.20/hour for voice synthesis."

Can I replace AssemblyAI with a different speech-to-text service?

While technically possible, switching providers demands manually implementing WebSocket coordination, audio realtime protocols, turn detection procedures, and connection restoration. You'd sacrifice AssemblyAI's automatic punctuation-based turn detection and the SDK ease enabling rapid implementation.

Can I use this pattern inside a framework like Pipecat or LiveKit?

Yes — AssemblyAI maintains first-party integrations for Pipecat, LiveKit, Vapi, and Twilio. These frameworks handle telephony, workflow coordination, and turn-taking, letting you emphasize agent capabilities.

Does this work with languages other than English?

Yes. Select the language with the language_code parameter, for instance: RealTimeParameters(speech_model="u3-rt-pro", sample_rate=16_000, language_code="es"). Consult the Supported Languages documentation for comprehensive language coverage.

‍