Build an AI voice agent with DeepSeek and AssemblyAI (Python tutorial)

DeepSeek is one of the most popular open-weights LLM families, and "can I talk to it?" is the obvious next question. In this tutorial you'll build a Python voice agent that listens through your microphone, thinks with DeepSeek, and talks back, all on one AssemblyAI API key:

• Speech-to-text: Universal-3.6 Pro Realtime, AssemblyAI's streaming flagship, transcribes you as you speak and detects when you've finished a turn.
• LLM: DeepSeek V4.1 Flash through LLM Gateway, AssemblyAI's OpenAI-compatible endpoint for 40+ LLMs. Prefer to keep the model on your own machine? Flip one setting and the same code runs DeepSeek-R1 locally through Ollama.
• Text-to-speech: AssemblyAI's Text-to-Speech API speaks the reply, streaming audio while the rest is still being generated.

The whole agent is about 180 lines of Python. If you'd like a walkthrough first, the video below shows the earlier version of this build, before AssemblyAI text-to-speech; the code in this post replaces it.

How the DeepSeek voice agent works

Every turn runs through three stages:

1. Listen. PyAudio captures 16 kHz microphone audio in 50 ms chunks and streams it over a WebSocket to Universal-3.6 Pro Realtime. The model returns partial transcripts as you speak and a final transcript with end_of_turn set when you're done. Its turn detection combines semantic context with voice activity, so a pause in the middle of a phone number isn't treated as the end of your sentence.
2. Think. The finished turn, plus the conversation so far, goes to DeepSeek as an OpenAI-style chat request.
3. Speak. The reply goes to AssemblyAI's Text-to-Speech API, which streams back 24 kHz PCM audio that plays as it arrives.

The agent stops listening while it speaks, so it never transcribes its own voice. That's the simplest reliable pattern for a laptop demo. Production agents usually keep listening and support barge-in; frameworks like LiveKit and Pipecat handle that for you (more on that at the end).

Why this split? A cascaded pipeline gives you a transcript at every step. When the agent gives a wrong answer, you can see whether it misheard you or whether the LLM got it wrong, and fix the layer that failed. Speech recognition sets the ceiling: on AssemblyAI's English voice-agent benchmark, Universal-3.6 Pro Realtime posted a 5.19% mean word error rate and a 14.4% entity error rate, the lowest of the 22 systems tested. If the agent mishears an order number, no LLM can recover it.

Which DeepSeek model should you use?

DeepSeek's current API lineup includes DeepSeek-V4.1-Flash, first released on September 10, 2026, and DeepSeek-V4-Pro. The latest Flash update, DeepSeek-V4.1-Flash-1017, shipped on October 17, 2026 with open weights. DeepSeek-R1 is the older reasoning model that made the family famous, and it's still easy to run locally.

A note on latency: R1 is a reasoning model, so it "thinks" before it answers. That's great for hard questions and noticeably slower for conversation, especially on a laptop. For a voice agent, start with V4.1 Flash and switch to R1 when privacy matters more than snappy replies.

On LLM Gateway, deepseek-v4.1-flash costs $0.30 per 1M input tokens and $1.20 per 1M output tokens, supports streaming and tool calling, and has a 1M-token context window. See the available models page for the current list and prices.

Step 1: Install dependencies

You'll need Python 3.9 or newer and a working microphone and speakers.

1.1 Get your API key

Sign up for an AssemblyAI account and copy your API key from the dashboard. The same key works for streaming speech-to-text, LLM Gateway and text-to-speech. The free tier includes up to 333 hours of streaming transcription with no credit card. LLM Gateway requires a paid AssemblyAI account; if you're on the free tier, use the Ollama option in Step 5 for the LLM.

1.2 Install PortAudio

PyAudio needs PortAudio for microphone access.

# macOS
brew install portaudio

# Debian/Ubuntu
sudo apt install portaudio19-dev

1.3 Install the Python libraries

pip install "assemblyai>=1.8.2" pyaudio sounddevice openai python-dotenv

assemblyai 1.8.2 or newer includes the Text-to-Speech client. sounddevice plays the audio, and openai is the client for LLM Gateway (and for Ollama, which speaks the same API).

Step 2: Set your API key

Create a .env file next to your script. Never hardcode keys in source files.

ASSEMBLYAI_API_KEY=your_assemblyai_api_key_here
USE_OLLAMA=false

Step 3: Build the voice agent

Create a file called voice_agent.py. The code blocks below are the whole file, in order.

3.1 Imports and configuration

import os
import re

from dotenv import load_dotenv
import pyaudio
import sounddevice as sd
import assemblyai as aai
from assemblyai.streaming.v3 import (
    BeginEvent,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    TerminationEvent,
    TurnEvent,
)
from openai import OpenAI

load_dotenv()

FRAMES_PER_BUFFER = 800  # 50 ms of audio at 16 kHz
TTS_SAMPLE_RATE = 24000  # Text-to-Speech output: 24 kHz, 16-bit mono PCM

# Set USE_OLLAMA=true in .env to run DeepSeek-R1 locally instead of through LLM Gateway
USE_OLLAMA = os.getenv("USE_OLLAMA", "false").lower() == "true"

SYSTEM_PROMPT = (
    "You are a friendly voice assistant. Your replies are spoken aloud, "
    "so answer in one or two short sentences with no markdown, lists or emoji."
)

The system prompt matters more for voice than for chat. Markdown, bullet lists and emoji sound terrible when read aloud, so the prompt asks for one or two short spoken sentences.

3.2 Set up the agent and pick the LLM

class AIVoiceAgent:
    def __init__(self):
        self.assemblyai_api_key = os.environ["ASSEMBLYAI_API_KEY"]

        if USE_OLLAMA:
            # Ollama's OpenAI-compatible endpoint; the API key is required but ignored
            self.llm = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")
            self.llm_model = "deepseek-r1:7b"
        else:
            # LLM Gateway is OpenAI-compatible and uses your AssemblyAI API key
            self.llm = OpenAI(
                base_url="https://llm-gateway.assemblyai.com/v1",
                api_key=self.assemblyai_api_key,
            )
            self.llm_model = "deepseek-v4.1-flash"

        # Text-to-Speech settings, reused for every reply
        self.tts_config = aai.TTSConfig(
            voice_id="aai-kyla",  # Kyla
            language="english",
            sample_rate=TTS_SAMPLE_RATE,
        )

        self.streaming_client = None
        self.pyaudio_instance = None
        self.microphone_stream = None
        self.is_streaming = False
        self.is_processing = False
        self.messages = [{"role": "system", "content": SYSTEM_PROMPT}]

Both LLM options use the same OpenAI client. LLM Gateway lives at https://llm-gateway.assemblyai.com/v1 and takes your AssemblyAI key. Ollama exposes an OpenAI-compatible endpoint on localhost:11434. Swapping models is a change to one string.

The Text-to-Speech config sets the voice explicitly. Kyla (aai-kyla) is the voice the voice library docs use in their examples and the default English voice, and assemblyai-tts-1 speaks every voice in all six supported languages.

3.3 Stream your microphone to Universal-3.6 Pro Realtime

    def start_transcription(self):
        self.streaming_client = RealTimeTranscriber(
            RealTimeTranscriberOptions(),
            api_key=self.assemblyai_api_key,
        )

        self.streaming_client.on(RealTimeEvents.Begin, self.on_begin)
        self.streaming_client.on(RealTimeEvents.Turn, self.on_turn)
        self.streaming_client.on(RealTimeEvents.Termination, self.on_terminated)
        self.streaming_client.on(RealTimeEvents.Error, self.on_error)

        self.streaming_client.connect(
            RealTimeParameters(
                sample_rate=16000,
                speech_model="universal-3-6-pro",
            )
        )

        self.pyaudio_instance = pyaudio.PyAudio()
        self.microphone_stream = self.pyaudio_instance.open(
            format=pyaudio.paInt16,
            channels=1,
            rate=16000,
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
        )

        self.is_streaming = True
        try:
            while self.is_streaming:
                self.streaming_client.stream(
                    self.microphone_stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                )
        except OSError:
            pass  # Microphone stream was closed while stopping

    def stop_transcription(self):
        self.is_streaming = False
        if self.streaming_client:
            self.streaming_client.disconnect(terminate=True)
            self.streaming_client = None
        if self.microphone_stream:
            self.microphone_stream.stop_stream()
            self.microphone_stream.close()
            self.microphone_stream = None
        if self.pyaudio_instance:
            self.pyaudio_instance.terminate()
            self.pyaudio_instance = None

speech_model="universal-3-6-pro" selects Universal-3.6 Pro Realtime. Turn detection runs at its defaults, which adapt to what's being said; hardcoding silence thresholds turns that off, so leave them alone unless you've measured a problem.

3.4 Handle streaming events

    def on_begin(self, client, event: BeginEvent):
        print(f"\n[Listening... session {event.id}]")

    def on_turn(self, client, event: TurnEvent):
        if not event.transcript or self.is_processing:
            return

        if event.end_of_turn:
            print(f"\nYou: {event.transcript}")
            self.is_processing = True
            self.generate_ai_response(event.transcript)
        else:
            print(f"\r{event.transcript}", end="", flush=True)

    def on_terminated(self, client, event: TerminationEvent):
        print(f"\n[Session ended: {event.audio_duration_seconds} s of audio]")

    def on_error(self, client, error: RealTimeError):
        print(f"\nStreaming error: {error}")

Partial transcripts print in place as you talk. When a TurnEvent arrives with end_of_turn set, the finished turn goes to the LLM. The is_processing flag drops any events that arrive while the agent is busy.

3.5 Generate a reply with DeepSeek

    def generate_ai_response(self, transcript_text):
        # Stop listening while the agent thinks and speaks, so it doesn't hear itself
        self.stop_transcription()
        self.messages.append({"role": "user", "content": transcript_text})

        stream = self.llm.chat.completions.create(
            model=self.llm_model,
            messages=self.messages,
            stream=True,
        )

        reply = ""
        for chunk in stream:
            if chunk.choices and chunk.choices[0].delta.content:
                reply += chunk.choices[0].delta.content

        # DeepSeek-R1 can return its reasoning inside <think> tags; never speak it
        reply = re.sub(r"<think>.*?</think>", "", reply, flags=re.DOTALL).strip()
        print(f"Agent: {reply}")

        self.messages.append({"role": "assistant", "content": reply})
        self.speak(reply)

        self.is_processing = False
        self.start_transcription()

The agent stops the transcription session, sends the conversation history to DeepSeek, and streams the reply. DeepSeek-R1 running in Ollama can include its chain of thought inside <think> tags, so the regex strips it before anything is spoken. Then the reply goes to text-to-speech, and listening resumes.

3.6 Speak the reply with AssemblyAI text-to-speech

    def speak(self, text):
        with aai.TTSSession(self.tts_config, api_key=self.assemblyai_api_key) as session:
            with sd.RawOutputStream(
                samplerate=TTS_SAMPLE_RATE, channels=1, dtype="int16"
            ) as speaker:
                for chunk in session.speak(text):  # raw 24 kHz 16-bit mono PCM
                    speaker.write(chunk)  # plays while the rest is still arriving

TTSSession opens a WebSocket to the Text-to-Speech API and speak() yields raw 24 kHz, 16-bit mono PCM chunks as they're generated. Writing each chunk straight to a sounddevice output stream means the first words play while the rest of the sentence is still arriving.

3.7 Run the agent

    def run(self):
        backend = "Ollama (local)" if USE_OLLAMA else "LLM Gateway"
        print(f"Starting voice agent: {self.llm_model} via {backend}. Press Enter to stop.")
        greeting = "Hi! I'm your DeepSeek voice assistant. What can I help you with?"
        print(f"Agent: {greeting}")
        self.speak(greeting)
        try:
            self.start_transcription()
            # After the first reply, listening restarts on a background thread,
            # so keep the main thread alive until you press Enter
            input()
        except KeyboardInterrupt:
            print("\nShutting down...")
        finally:
            self.stop_transcription()


if __name__ == "__main__":
    AIVoiceAgent().run()

Step 4: Talk to your agent

python voice_agent.py

The agent greets you, then starts listening. Ask it something, pause, and it answers. Press Enter to stop.

Step 5 (optional): Run DeepSeek-R1 locally with Ollama

Want the LLM on your own machine? Install Ollama, then pull DeepSeek-R1:

ollama pull deepseek-r1:7b

Set USE_OLLAMA=true in your .env file and run the script again:

USE_OLLAMA=true

Nothing else changes. Prompts and replies stay on your hardware, while transcription and speech still run through AssemblyAI. Expect slower replies than V4.1 Flash, because R1 reasons before it answers and a 7B model on a laptop is far less powerful than a hosted one. Larger R1 variants answer better if your GPU can hold them.

Why AssemblyAI for the speech layers

• Accuracy where agents fail. Universal-3.6 Pro Realtime ranked 1st of 22 systems on AssemblyAI's English voice-agent benchmark, with 5.19% mean WER and the lowest entity error rate, 14.4%. Names, emails and order numbers are where voice agents usually break. Read the Universal-3.6 Pro Realtime launch post for the full results.
• 32 languages with code-switching. The same model handles automatic language detection and speakers who switch languages mid-sentence. See supported models and languages.
• Text-to-speech built for agents. AssemblyAI-TTS-1 is trained on millions of hours of audio, including real conversational data. It delivers frontier-level quality, it's faster than other production voice agent TTS models, it speaks six languages, and it costs $15 per 1M characters.
• One key, any LLM. LLM Gateway puts DeepSeek, Claude, GPT, Gemini and other models behind one OpenAI-compatible endpoint, so you can A/B your agent's brain by changing a model string.
• Simple pricing. Universal-3.6 Pro Realtime streaming costs $0.45/hr, billed per second. Sessions already open aren't capped; only new sessions per minute are rate-limited, and that limit grows with use on a paid plan. See pricing.

Where to go next

This agent is deliberately simple: it stops listening while it talks, and the LLM has no tools. To go further:

• Compare with an OpenAI-powered build. The real-time AI voice bot in Python uses the same skeleton with an OpenAI model and a receptionist persona.
• Add barge-in and production turn handling. Start from AssemblyAI's open-source reference agents for LiveKit or Pipecat, or see how to build a voice agent without Pipecat or LiveKit.
• Go deeper on speech output. The text-to-speech in Python guide and the Text-to-Speech quickstart cover voices, encodings and word timings.
• Understand the architecture. Our voice agent architecture guide explains the STT-LLM-TTS pipeline and where latency comes from. For the fastest possible start, try the 5-minute Python voice agent tutorial.

Final words

The interesting part of this build isn't DeepSeek, and it isn't the voice. It's that the LLM became a one-line choice. The same agent runs a hosted DeepSeek model, a local R1, or anything else on LLM Gateway, and you can swap between them while the speech layers stay put. That's the real argument for a cascaded agent: you can upgrade the brain every time a better open-weights model ships, without touching the ears or the mouth.

Frequently Asked Questions (FAQs)

Can I build a voice agent with DeepSeek?

Yes. DeepSeek handles the reasoning, and you pair it with streaming speech-to-text for input and text-to-speech for output. In this tutorial, Universal-3.6 Pro Realtime transcribes the user, DeepSeek V4.1 Flash runs through AssemblyAI's LLM Gateway (or DeepSeek-R1 runs locally in Ollama), and AssemblyAI's Text-to-Speech API speaks the reply.

Does DeepSeek have voice chat or a speech API?

DeepSeek's API is text-in, text-out, so it doesn't listen or speak on its own. To give it a voice, put a streaming speech-to-text model in front of it and a text-to-speech model after it. That's the cascaded pipeline this tutorial builds.

Can Ollama transcribe audio, or do I need a separate speech-to-text API?

Ollama runs language models; it doesn't do real-time speech-to-text. You still need a streaming transcription layer in front of it. Here, Universal-3.6 Pro Realtime transcribes your microphone audio and hands each finished turn to DeepSeek-R1 in Ollama.

Can I build an offline voice assistant in Python with DeepSeek?

Partly. With the Ollama option, DeepSeek-R1 runs on your machine, so prompts and conversation history never leave your hardware. Transcription and speech synthesis still call AssemblyAI's APIs; fully offline speech models exist, but they trade away accuracy on names, numbers and accented speech.

Should I use DeepSeek-R1 or DeepSeek V4.1 Flash for a voice agent?

Start with V4.1 Flash. R1 is a reasoning model that thinks before it answers, which adds noticeable delay in conversation, while V4.1 Flash is DeepSeek's current fast model and supports tool calling on LLM Gateway. Use R1 locally when keeping data on your own hardware matters more than response time.

How much does a DeepSeek voice agent cost to run?

Universal-3.6 Pro Realtime streaming costs $0.45/hr, AssemblyAI text-to-speech costs $15 per 1M characters, and DeepSeek V4.1 Flash on LLM Gateway costs $0.30 per 1M input tokens and $1.20 per 1M output tokens. Running DeepSeek-R1 in Ollama has no LLM cost beyond your own hardware.