This tutorial shows you how to build a complete voice agent that listens, thinks, and responds naturally using Python. You'll create a realtime application that processes speech in real-time, generates intelligent responses, and speaks back to users—all in under 100 lines of code.
The voice agent combines three components: AssemblyAI-STT-3.7-Realtime-Agent for speech-to-text, OpenAI's GPT-4 for conversational AI, and AssemblyAI's Text-to-Speech API for the agent's voice. Speech-to-text and text-to-speech both run through the assemblyai Python SDK on one API key, and each component streams data to minimize response delays and create smooth, human-like conversations.
What you'll need to get started
You need Python 3.9 or higher, two API keys (AssemblyAI and OpenAI), and a computer with a microphone and speakers. The setup takes about 2 minutes once you have everything ready.
Install Python dependencies
Open your terminal and run this command to install everything you need:
pip install "assemblyai>=1.10.1" openai pyaudio python-dotenvPyAudio requires the PortAudio system library: on macOS run brew install portaudio, and on Debian/Ubuntu run apt install portaudio19-dev before installing the Python packages.
Here's what each package does for your voice agent:
- assemblyai: Handles real-time speech recognition with AssemblyAI-STT-3.7-Realtime-Agent, and gives the agent its voice with the Text-to-Speech API (version 1.10.1 or later)
- openai: Connects to GPT models for smart responses
- pyaudio: Provides access to your microphone and speakers
- python-dotenv: Loads your API keys from a .env file
Configure your API keys
Create a .env file in your project directory with your API keys:
Code — Text
ASSEMBLYAI_API_KEY=your_assemblyai_key_here
OPENAI_API_KEY=your_openai_key_here
The same AssemblyAI key covers both speech-to-text and text-to-speech, so these two keys are all the agent needs.
Never share this file or commit it to version control. Add .env to your .gitignore file to protect your API keys.
One AssemblyAI key covers both ends of the conversation: speech-to-text, the layer everything downstream depends on, and text-to-speech. Sign up free and paste your key into the .env file above.
What are the components of a voice agent?
A voice agent is a program that talks to you like a human using three connected parts. These parts work together to create conversations: speech-to-text converts your voice into text, a language model thinks about what you said and creates a response, and text-to-speech turns that response back into spoken words.
This pipeline needs to work in real-time to feel natural. When you speak to Siri or Alexa, you expect quick responses—not awkward pauses that break the conversational flow. Here's what each component does in your voice agent:
| Component | Role | Why streaming matters | Example |
|---|---|---|---|
| AssemblyAI | Speech-to-text | Transcribes audio as it arrives, so the LLM can start responding sooner | Converts "what's the weather" to text before you finish speaking |
| OpenAI | Language model | Generates a response token by token, so text-to-speech can begin immediately | Starts answering while still composing the full response |
| AssemblyAI | Text-to-speech | Plays audio while more is being generated | Speaks the first sentence while generating the second |
The difference between good and bad voice agents comes down to speed. Batch processing—where each step waits for the previous one to finish completely—creates those robotic pauses that make conversations feel unnatural.
AssemblyAI-STT-3.7-Realtime-Agent solves this by processing speech as it happens. You get accurate transcription with minimal delay, making conversations feel smooth and responsive.
Set up speech-to-text with AssemblyAI
Speech recognition forms the foundation of your voice agent. AssemblyAI-STT-3.7-Realtime-Agent listens to your microphone and converts speech to text in real-time. The SDK handles all WebSocket complexity automatically—no manual connection management required.
Create a new file called voice_agent.py and add this code:
from assemblyai.streaming.v3 import (
BeginEvent,
RealTimeTranscriber,
RealTimeTranscriberOptions,
RealTimeError,
RealTimeEvents,
RealTimeParameters,
TurnEvent,
TerminationEvent,
)
from dotenv import load_dotenv
import os
load_dotenv()
class VoiceAgent:
def __init__(self):
self.client = RealTimeTranscriber(
RealTimeTranscriberOptions(),
api_key=os.getenv('ASSEMBLYAI_API_KEY'),
)
self.client.on(RealTimeEvents.Begin, self.on_begin)
self.client.on(RealTimeEvents.Turn, self.on_turn)
self.client.on(RealTimeEvents.Termination, self.on_terminated)
self.client.on(RealTimeEvents.Error, self.on_error)
self.is_processing = False
def on_begin(self, client: RealTimeTranscriber, event: BeginEvent):
print("Listening... Start speaking!")
def on_turn(self, client: RealTimeTranscriber, turn: TurnEvent):
if not turn.transcript:
return
if turn.end_of_turn:
print(f"You said: {turn.transcript}")
# AI processing added in next section
else:
print(f"Hearing: {turn.transcript}", end="\r")
def on_error(self, client: RealTimeTranscriber, error: RealTimeError):
print(f"Error: {error}")
def on_terminated(self, client: RealTimeTranscriber, event: TerminationEvent):
print("Connection closed")This code creates a real-time transcription system that gives you two types of output. Partial transcripts (where end_of_turn is False) show you what the system is hearing as you speak, and final transcripts (where end_of_turn is True) provide the complete sentence when you pause.
AssemblyAI-STT-3.7-Realtime-Agent uses content-aware turn detection: after a natural pause, it ends the turn when the transcript reads as a finished sentence, and it holds the turn open when you stop partway through a number or other entity. This means you don't need to press buttons or give special commands—just speak naturally and pause.
Turn detection is what makes the agent respond at the right moment. Stream your own voice in the playground and watch partials finalize with your natural pacing.
Connect the language model
The language model serves as the thinking component of your voice agent—it comprehends user input and determines appropriate responses. OpenAI's GPT-4 streams its output token by token, so you see the reply appear as it's written and can hand it to text-to-speech the moment it's complete.
Incorporate this OpenAI connection into your VoiceAgent class:
from openai import OpenAI
class VoiceAgent:
def __init__(self):
# Previous code...
self.openai_client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))
self.conversation = [
{"role": "system", "content": """You are a helpful voice assistant.
Keep responses short and conversational.
Talk like you're having a normal conversation with someone."""}
]
def process_with_llm(self, user_text):
self.conversation.append({"role": "user", "content": user_text})
response_text = ""
stream = self.openai_client.chat.completions.create(
model="gpt-4",
messages=self.conversation,
stream=True,
temperature=0.7,
max_tokens=150
)
print("Assistant: ", end="")
for chunk in stream:
if chunk.choices[0].delta.content:
content = chunk.choices[0].delta.content
response_text += content
print(content, end="", flush=True)
print()
self.conversation.append({"role": "assistant", "content": response_text})
self.speak(response_text)The conversation history maintains the complete dialogue, allowing your agent to understand context across multiple exchanges. The system prompt directs GPT-4 to produce concise, natural-sounding answers—verbose responses feel unnatural when delivered through voice.
Add text-to-speech output
Text-to-speech concludes your voice agent by turning the AI-generated reply into speech. This tutorial uses AssemblyAI's Text-to-Speech API, which runs AssemblyAI-TTS-1, a model built for conversational voice agents and trained on millions of hours of audio, including real conversational data. It offers frontier-level quality and runs faster than other production voice agent TTS models.
The Python SDK's TTSSession keeps one WebSocket open for the whole conversation. Each reply goes in with generate(), flush() concludes it and returns its ID, and audio() yields that reply's raw 24 kHz, 16-bit mono PCM as it's generated, which PyAudio can play directly.
Integrate voice synthesis into your VoiceAgent class:
import queue
import threading
import assemblyai as aai
import pyaudio
TTS_SAMPLE_RATE = 24000 # Text-to-Speech output: 24 kHz, 16-bit mono PCM
class VoiceAgent:
def __init__(self):
# Previous code...
self.tts_config = aai.TTSConfig(
voice_id="aai-kyla", # Kyla
language="english",
sample_rate=TTS_SAMPLE_RATE,
)
# One Text-to-Speech session for the whole conversation
self.tts_session = aai.TTSSession(
self.tts_config,
api_key=os.getenv('ASSEMBLYAI_API_KEY'),
)
# Replies waiting to be played, by flush ID
self.replies = queue.Queue()
def speak(self, text):
try:
self.tts_session.generate(text)
self.replies.put(self.tts_session.flush())
except Exception as e:
print(f"Voice error: {e}")
def play_replies(self):
pa = pyaudio.PyAudio()
speaker = pa.open(
format=pyaudio.paInt16,
channels=1,
rate=TTS_SAMPLE_RATE,
output=True,
)
while True:
try:
flush_id = self.replies.get(timeout=30)
except queue.Empty:
flush_id = None
try:
if flush_id is None:
# An idle session closes after 2 minutes; this keeps it open
self.tts_session.keep_alive()
continue
# Raw PCM, played as it arrives
for chunk in self.tts_session.audio(flush_id):
speaker.write(chunk)
except Exception as e:
print(f"Voice error: {e}")A few details in this code are worth knowing:
- An explicit voice: The config sets voice_id to Kyla by her library slug, aai-kyla. Browse the voice library for thousands of voices, or set language to Spanish, French, German, Italian or Portuguese. AssemblyAI-TTS-1 speaks every voice in all six languages.
- One session per conversation: The session opens once and every reply is a request on it, so a reply doesn't wait for a new connection. A session closes after 2 minutes with no traffic, so the player sends keep_alive() whenever it has waited 30 seconds for a reply.
- Playback off the main path: speak() sends the text and returns at once. The play_replies thread pulls each reply's audio in order and writes it to the speakers, so your agent keeps listening while it talks.
The session and the playback thread start in start(), before the agent connects to speech-to-text, as the complete program below shows.
Build the complete voice agent
Here represents your full voice_agent.py implementation:
import os
import queue
import sys
import threading
import assemblyai as aai
import pyaudio
from assemblyai.streaming.v3 import (
BeginEvent,
RealTimeTranscriber,
RealTimeTranscriberOptions,
RealTimeError,
RealTimeEvents,
RealTimeParameters,
TurnEvent,
TerminationEvent,
)
from openai import OpenAI
from dotenv import load_dotenv
load_dotenv()
FRAMES_PER_BUFFER = 800 # 50 ms at 16 kHz
TTS_SAMPLE_RATE = 24000 # Text-to-Speech output: 24 kHz, 16-bit mono PCM
class VoiceAgent:
def __init__(self):
self.client = RealTimeTranscriber(
RealTimeTranscriberOptions(),
api_key=os.getenv('ASSEMBLYAI_API_KEY'),
)
self.client.on(RealTimeEvents.Begin, self.on_begin)
self.client.on(RealTimeEvents.Turn, self.on_turn)
self.client.on(RealTimeEvents.Termination, self.on_terminated)
self.client.on(RealTimeEvents.Error, self.on_error)
self.openai_client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))
self.tts_config = aai.TTSConfig(
voice_id="aai-kyla", # Kyla
language="english",
sample_rate=TTS_SAMPLE_RATE,
)
self.tts_session = aai.TTSSession(
self.tts_config,
api_key=os.getenv('ASSEMBLYAI_API_KEY'),
)
self.replies = queue.Queue()
self.is_processing = False
self.conversation = [
{"role": "system", "content": """You are a helpful voice assistant.
Keep responses short and conversational.
Talk like you're having a normal conversation with someone."""}
]
def on_begin(self, client: RealTimeTranscriber, event: BeginEvent):
print("\n Voice Agent Ready! Start speaking...\n")
def on_turn(self, client: RealTimeTranscriber, turn: TurnEvent):
if not turn.transcript:
return
if turn.end_of_turn:
print("\r" + " " * 50 + "\r", end="")
print(f"You: {turn.transcript}")
if not self.is_processing:
self.is_processing = True
self.process_with_llm(turn.transcript)
self.is_processing = False
else:
print(f"Listening: {turn.transcript}...", end="\r")
def on_error(self, client: RealTimeTranscriber, error: RealTimeError):
print(f"\n Error: {error}\n")
def on_terminated(self, client: RealTimeTranscriber, event: TerminationEvent):
print("\n Voice Agent stopped\n")
def process_with_llm(self, user_text):
self.conversation.append({"role": "user", "content": user_text})
response_text = ""
stream = self.openai_client.chat.completions.create(
model="gpt-4",
messages=self.conversation,
stream=True,
temperature=0.7,
max_tokens=150
)
print("Agent: ", end="")
for chunk in stream:
if chunk.choices[0].delta.content:
content = chunk.choices[0].delta.content
response_text += content
print(content, end="", flush=True)
print()
self.conversation.append({"role": "assistant", "content": response_text})
self.speak(response_text)
def speak(self, text):
try:
self.tts_session.generate(text)
self.replies.put(self.tts_session.flush())
except Exception as e:
print(f"Voice error: {e}")
def play_replies(self):
pa = pyaudio.PyAudio()
speaker = pa.open(
format=pyaudio.paInt16,
channels=1,
rate=TTS_SAMPLE_RATE,
output=True,
)
while True:
try:
flush_id = self.replies.get(timeout=30)
except queue.Empty:
flush_id = None
try:
if flush_id is None:
# An idle session closes after 2 minutes; this keeps it open
self.tts_session.keep_alive()
continue
# Raw PCM, played as it arrives
for chunk in self.tts_session.audio(flush_id):
speaker.write(chunk)
except Exception as e:
print(f"Voice error: {e}")
def start(self):
# Open the Text-to-Speech session and the playback thread first
self.tts_session.connect()
player = threading.Thread(target=self.play_replies, daemon=True)
player.start()
self.client.connect(
RealTimeParameters(
sample_rate=16000,
speech_model="assemblyai-stt-3-7-realtime-agent",
)
)
# Capture microphone audio with PyAudio and feed raw PCM16 chunks
# to the transcriber via client.stream().
pa = pyaudio.PyAudio()
mic = pa.open(
format=pyaudio.paInt16,
channels=1,
rate=16000,
input=True,
frames_per_buffer=FRAMES_PER_BUFFER,
)
try:
while True:
self.client.stream(mic.read(FRAMES_PER_BUFFER, exception_on_overflow=False))
except KeyboardInterrupt:
pass
finally:
mic.stop_stream()
mic.close()
pa.terminate()
self.stop()
def stop(self):
print("\nStopping voice agent...")
self.client.disconnect(terminate=True)
self.tts_session.terminate()
sys.exit(0)
if __name__ == "__main__":
agent = VoiceAgent()
agent.start()This complete implementation includes error handling, conversation memory, and clean shutdown. "The agent remembers what you've talked about during each session, enabling natural back-and-forth conversations."
Run your voice agent
Execute your voice agent using this command:
python voice_agent.pyWhen "Voice Agent Ready! Start speaking..." displays, the agent is listening. Speak into your microphone and the agent responds with both text and voice output. Use headphones, so the microphone doesn't pick up the agent's own voice and treat it as your next turn.
Test your agent with these prompts:
- "What's the weather like today?"
- "Tell me a quick joke"
- "Help me plan dinner"
- "Explain how WiFi works simply"
Common issues and fixes:
- No microphone input: Verify system permissions and microphone settings
- Slow responses: Check internet connectivity and consider swapping gpt-3.5-turbo for faster processing
- Agent responds to itself: The microphone is hearing the speakers. Wear headphones or lower the speaker volume
- No voice output: Check that your output device is set as the system default and that the "Voice error" message, if any, doesn't report an invalid API key
Final words
You've constructed a complete realtime voice agent that processes audio in real-time and participates in natural dialogue. This implementation integrates speech recognition, artificial intelligence processing, and voice synthesis into a unified application showcasing contemporary Voice AI capabilities, with both speech layers on AssemblyAI.
"AssemblyAI-STT-3.7-Realtime-Agent makes this possible by providing the accuracy and speed that voice agents require." The SDK manages intricate WebSocket connectivity and audio handling, freeing you to concentrate on application development rather than infrastructure details.
To expand functionality, review the AssemblyAI-STT-3.7-Realtime-Agent documentation for capabilities including keyterm prompting, realtime speaker labels, and live configuration adjustments—all functioning without agent restart.
On the speaking side, the next step is to start talking before GPT-4 finishes the reply: send each piece of text with generate() as it streams in, and call flush() at the end of each sentence. Optimizing accuracy and latency explains that pattern, and the Text-to-Speech quickstart covers the rest of the SDK.
Get your API key and the streaming model featured in this guide. Keyterm prompting, speaker diarization, and live config changes are all available on the same connection.
Frequently asked questions
Do I need WebSocket knowledge to build this voice agent?
No. The AssemblyAI Python SDK manages WebSocket connection handling, reconnection procedures, and audio realtime protocol automatically. You implement event handlers while the SDK manages remaining details.
How much does running this voice agent cost per hour?
The cost depends on how long the agent listens and how much it says. AssemblyAI charges $0.45/hr for AssemblyAI-STT-3.7-Realtime-Agent, OpenAI costs roughly $0.30/hour for GPT-4 responses, and AssemblyAI's Text-to-Speech API costs $15 per 1M characters, billed on the characters of text you send it. See AssemblyAI pricing for current rates.
Can I replace AssemblyAI with a different speech-to-text service?
While technically possible, switching providers demands manually implementing WebSocket coordination, audio realtime protocols, turn detection procedures, and connection restoration. You'd sacrifice AssemblyAI's built-in turn detection and the SDK ease enabling rapid implementation.
Can I use this pattern inside a framework like Pipecat or LiveKit?
Yes — AssemblyAI maintains first-party integrations for Pipecat, LiveKit, Vapi, and Twilio. These frameworks handle telephony, workflow coordination, and turn-taking, letting you emphasize agent capabilities.
Does this work with languages other than English?
Yes. The model covers 32 languages with native code switching. To steer it toward the languages you expect, pass language_codes, for instance: RealTimeParameters(speech_model="assemblyai-stt-3-7-realtime-agent", sample_rate=16_000, language_codes=["es"]). Consult the Supported Languages documentation for comprehensive language coverage.