Build an AI Voice Agent with DeepSeek R1, AssemblyAI, and ElevenLabs

AI voice agents are revolutionizing how businesses interact with users, enabling real-time conversations with intelligent and human-like responses. According to a16z, financial services alone make up 25% of total global contact center spending and account for over $100 billion annually in business process outsourcing (BPO). With voice agents increasingly completing complex tasks successfully, businesses are rapidly adopting automated voice interactions to improve efficiency and customer experience.

In this tutorial, you'll learn how to build an AI voice agent that transcribes speech in real time using AssemblyAI's Universal-Streaming, generates intelligent responses using DeepSeek R1 (7B model) via Ollama, and converts those responses into speech with ElevenLabs. The entire process happens with ultra-low latency, allowing for smooth and interactive voice-based applications that feel natural and responsive.

By the end of this tutorial, you'll have a fully functional AI voice agent that can transcribe, process, and respond to spoken queries with industry-leading accuracy and speed. If you prefer watching the video, check it out below:

https://www.youtube.com/embed/SZYpRkoazb0

Key components of the AI voice agent

The AI voice agent consists of three major components:

• Real-time speech-to-text: AssemblyAI's Universal-Streaming model converts spoken words into text with ultra-low latency (~300ms) and superior accuracy, specifically designed for voice agents.
• AI-powered responses: DeepSeek R1 (7B model) via Ollama generates context-aware responses with intelligent conversation flow.
• Text-to-speech synthesis: ElevenLabs converts the AI-generated text response into natural-sounding audio.

In this step-by-step guide, we'll install the necessary dependencies, configure the AI agent, and run the complete system using AssemblyAI's cutting-edge Universal-Streaming technology that delivers the accuracy and speed voice agents need.

Step 1: Install required dependencies

Before you begin, ensure you have all the required dependencies installed.

1.1 Get API Keys

To use AssemblyAI and ElevenLabs, sign up for their API keys:

• AssemblyAI (Speech-to-Text): Sign up for a free API key - Get $50 in free credits to get started
• ElevenLabs (Text-to-Speech): Sign up for an account

1.2 Install Ollama

DeepSeek R1 is accessed via Ollama. Install Ollama by following the instructions here.

1.3 Install PortAudio (Required for real-time transcription)

For Debian/Ubuntu:

apt install portaudio19-dev

For MacOS:

brew install portaudio

1.4 Install Python Libraries

Run the following command to install the required Python dependencies:

pip install assemblyai pyaudio ollama elevenlabs

1.5 (MacOS Only) Install MPV for Audio Streaming

brew install mpv

Step 2: Download the DeepSeek R1 model

Since this script uses DeepSeek R1 via Ollama, download the model locally by running:

ollama pull deepseek-r1:7b

Step 3: Implement the AI voice agent

Create a new Python file called AIVoiceAgent.py and add the following code.

3.1 Import required libraries

import assemblyai as aai
import pyaudio
from assemblyai.streaming.v3 import (
    BeginEvent,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    TerminationEvent,
    TurnEvent,
)
from elevenlabs.client import ElevenLabs
from elevenlabs import stream
import ollama

FRAMES_PER_BUFFER = 800  # 50ms of audio at 16kHz

The first step is to import the necessary libraries. AssemblyAI's Universal-Streaming is used for real-time speech-to-text transcription with superior accuracy and ultra-low latency, PyAudio captures audio from the microphone, Ollama allows us to interact with DeepSeek R1, and ElevenLabs is responsible for converting AI-generated text into speech. These tools together form the core of our AI voice agent, providing the developer-friendly integration that makes building production-ready voice applications straightforward.

3.2 Define the AI voice agent class

Next, we define the AIVoiceAgent class, which initializes our API keys and sets up the conversation state.

class AIVoiceAgent:
    def __init__(self):
        aai.settings.api_key = "ASSEMBLYAI_API_KEY"
        self.elevenlabs_client = ElevenLabs(api_key="ELEVENLABS_API_KEY")
        self.streaming_client = None
        self.pyaudio_instance = None
        self.microphone_stream = None
        self.is_streaming = False
        self.full_transcript = [
            {"role": "system", "content": "You are a language model called R1 created by DeepSeek. Answer in less than 300 characters."}
        ]

This class stores API keys for AssemblyAI and ElevenLabs and initializes full_transcript, which keeps track of the conversation history. The system message defines the behavior of the AI model, ensuring that responses are concise and relevant. AssemblyAI's Universal-Streaming provides the reliable foundation for accurate speech recognition that powers the entire conversation flow.

3.3 Set up real-time transcription with Universal-Streaming

We then define the function that handles streaming real-time transcription using AssemblyAI's latest Universal-Streaming model.

    def start_transcription(self):
        print("\nReal-time transcription with Universal-Streaming:", end="\r\n")

        # Configure streaming parameters for optimal voice agent performance
        streaming_params = RealTimeParameters(
            sample_rate=16000,
            speech_model="universal-3-5-pro",  # flagship streaming model
            min_end_of_turn_silence_when_confident=160  # Optimized for voice agents
        )

        self.streaming_client = RealTimeTranscriber(
            RealTimeTranscriberOptions(),
            api_key=aai.settings.api_key,
        )

        # Set up event handlers for Universal-Streaming
        self.streaming_client.on(RealTimeEvents.Begin, self.on_begin)
        self.streaming_client.on(RealTimeEvents.Turn, self.on_turn)
        self.streaming_client.on(RealTimeEvents.Termination, self.on_terminated)
        self.streaming_client.on(RealTimeEvents.Error, self.on_error)
        # Start the streaming session
        self.streaming_client.connect(streaming_params)

        # Stream microphone audio with PyAudio
        self.pyaudio_instance = pyaudio.PyAudio()
        self.microphone_stream = self.pyaudio_instance.open(
            format=pyaudio.paInt16,
            channels=1,
            rate=16000,
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
        )
        self.is_streaming = True
        try:
            while self.is_streaming:
                self.streaming_client.stream(
                    self.microphone_stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                )
        except OSError:
            pass  # Microphone stream was closed while stopping

This method starts the real-time transcription process using Universal-Streaming, capturing audio from the microphone with PyAudio and streaming it to AssemblyAI for conversion into text with ~300ms latency. Universal-Streaming's intelligent endpointing ensures natural conversation flow without awkward pauses or premature interruptions.

3.4 Handling Universal-Streaming events

Once the transcription starts, we need to process the incoming text using Universal-Streaming's event-driven architecture.

    def on_begin(self, client, event: BeginEvent):
        print(f"Universal-Streaming session started with ID: {event.id}")
    def on_turn(self, client, event: TurnEvent):
        if not event.transcript:
            return

        if event.end_of_turn:
            print(f"\nFinal: {event.transcript}")
            self.generate_ai_response(event.transcript)
        else:
            print(f"Partial: {event.transcript}", end="\r")
    def on_terminated(self, client, event: TerminationEvent):
        print(f"\nSession terminated: {event.audio_duration_seconds} seconds processed")
    def on_error(self, client, error: RealTimeError):
        if not str(error):
            return
        print(f"Streaming error: {error}")

Universal-Streaming provides immutable transcripts, meaning the text won't change once delivered, enabling faster downstream processing. When an end-of-turn is detected using intelligent endpointing, the final transcript is sent to the AI model for response generation.

3.5 Generate AI responses with DeepSeek R1

Once transcription is complete, the system sends the conversation history to DeepSeek R1 via Ollama to generate a response.

    def generate_ai_response(self, transcript_text):
        self.stop_transcription()
        self.full_transcript.append({"role": "user", "content": transcript_text})
        print(f"\nUser: {transcript_text}", end="\r\n")

        ollama_stream = ollama.chat(
            model="deepseek-r1:7b",
            messages=self.full_transcript,
            stream=True,
        )

        response_text = "".join(chunk['message']['content'] for chunk in ollama_stream)
        print("DeepSeek R1:", response_text)
        self.speak_response(response_text)
        self.full_transcript.append({"role": "assistant", "content": response_text})
        self.start_transcription()

    def speak_response(self, text):
        """Convert text to speech using ElevenLabs (new SDK layout)"""
        try:
            # Preferred path for current SDKs (streams chunks as they're produced)
            audio_generator = self.elevenlabs_client.text_to_speech.stream(
                text=text,
                voice_id="pNInz6obpgDQGcFmaJgB",        # Adam's voice ID
                model_id="eleven_monolingual_v1",
                output_format="mp3_44100_128",
            )
        except AttributeError:
            # Fall back to older SDKs that still expose `.generate()`
            audio_generator = self.elevenlabs_client.generate(
                text=text,
                voice="Adam",
                model="eleven_monolingual_v1",
            )
        # Plays the bytes (or generator) through your system audio
        stream(audio_generator)
    def stop_transcription(self):
        self.is_streaming = False
        if self.microphone_stream:
            self.microphone_stream.stop_stream()
            self.microphone_stream.close()
            self.microphone_stream = None
        if self.pyaudio_instance:
            self.pyaudio_instance.terminate()
            self.pyaudio_instance = None
        if self.streaming_client:
            self.streaming_client.disconnect(terminate=True)
            self.streaming_client = None

This method pauses transcription, sends the user's message to DeepSeek R1, and streams back the AI-generated response. The response is then converted to speech using ElevenLabs, creating a seamless conversation experience powered by AssemblyAI's industry-leading accuracy.

3.6 Main execution

    def run(self):
        print("Starting AI Voice Agent with Universal-Streaming...")
        print("Pricing: Universal-Streaming at just $0.15/hour with unlimited concurrency")
        try:
            self.start_transcription()
            input("Press Enter to stop...")
        except KeyboardInterrupt:
            print("\nShutting down...")
        finally:
            self.stop_transcription()

if __name__ == "__main__":
    agent = AIVoiceAgent()
    agent.run()

Step 4: Running the AI voice agent

Once all dependencies are installed and the model is downloaded, simply run:

python AIVoiceAgent.py

Why Universal-Streaming for voice agents

AssemblyAI's Universal-Streaming delivers several key advantages for voice agent applications:

• Ultra-low latency: ~300ms immutable transcripts enable faster response times
• Superior accuracy: Substantial improvements in capturing emails, codes, and proper nouns
• Intelligent endpointing: Combines acoustic and semantic features for natural conversation flow
• Transparent pricing: $0.15/hour with unlimited concurrency and no hidden fees
• Production-ready: Designed specifically for voice agents with enterprise reliability

This makes Universal-Streaming the ideal choice for developers building voice agents that need to feel natural, complete tasks successfully, and scale without compromise.

Final words

By integrating AssemblyAI's Universal-Streaming, DeepSeek R1, and ElevenLabs, this tutorial demonstrates how to build a production-ready AI voice agent with ultra-low latency transcription, intelligent conversation flow, and natural speech synthesis—enabling voice agents that feel more responsive and complete tasks successfully.

Universal-Streaming's developer-friendly API and cutting-edge technology provide the reliable foundation needed for building voice agents that deliver exceptional user experiences. With industry-leading accuracy and transparent pricing, you can focus on building innovative features while AssemblyAI handles the complexities of speech recognition.

Learn more about AssemblyAI's Universal-Streaming: Check the Docs

Frequently asked questions

What do you need to build an AI voice agent?

An AI voice agent needs three components wired together: speech-to-text to transcribe the caller, a large language model to decide what to say, and text-to-speech to speak the response. This tutorial uses AssemblyAI's Universal-Streaming for transcription, DeepSeek R1 (7B) running locally through Ollama for reasoning, and ElevenLabs for speech synthesis. The transcription layer sets the accuracy ceiling for everything downstream — if the agent mishears an email address or order number, the LLM never gets a chance to answer correctly.

Can Ollama transcribe audio, or do I need a separate speech-to-text API?

Ollama does not do real-time speech-to-text — it runs local language models, so you still need a dedicated streaming transcription layer in front of it. In this build, AssemblyAI's Universal-Streaming handles the microphone audio and hands finalized turns to DeepSeek R1 through Ollama. Splitting the two is also what keeps latency low: transcripts arrive in roughly 300ms and are immutable, so the LLM can start reasoning without waiting for text to be revised.

Can I build an offline voice assistant in Python?

Partly. The language model in this tutorial runs entirely on your machine through Ollama, so no prompt or conversation history leaves your hardware — but transcription and speech synthesis call cloud APIs. A fully offline build is possible with local STT and TTS models, though you'll trade meaningful accuracy on the things voice agents fumble most: names, emails, order numbers, and accented speech.

How much does it cost to build an AI voice agent?

Running DeepSeek R1 locally through Ollama is free apart from your own hardware, so the cost comes from the speech layers. AssemblyAI streaming starts at $0.15/hour with unlimited concurrency and no minimum commitment, and ElevenLabs charges by characters synthesized. If you'd rather not price three components separately, the Voice Agent API bundles speech-to-text, an LLM, and text-to-speech behind one WebSocket at a flat $4.50/hour.

Should I build my own STT-LLM-TTS pipeline or use a single voice agent API?

Build the pipeline yourself when you need control over the specific components — a local model like DeepSeek R1 for privacy, a particular voice, or custom logic between turns, as in this tutorial. Use a managed voice agent API when you want turn detection, barge-in, and orchestration handled for you and don't need to swap providers per request. Both approaches run on the same AssemblyAI speech foundation, so accuracy doesn't change with the choice.

What is the best speech-to-text model for a voice agent?

The right model for a voice agent is judged on streaming latency and accuracy on messy real-world audio, not on clean benchmark recordings. AssemblyAI's Universal-Streaming returns immutable transcripts in roughly 300ms and uses intelligent endpointing that combines acoustic and semantic signals, so the agent replies at the right moment instead of interrupting. For agent workloads specifically, Universal-3.5 Pro Realtime posts a 6.99% word error rate on the open Pipecat voice-agent benchmark, ahead of Google Chirp3 (9.04%), ElevenLabs Scribe v2 (9.76%), and Deepgram Flux (15.58%).