AI voice agents are transforming how users interact with applications, enabling natural conversations that feel remarkably human. Instead of clicking through menus or typing commands, users can simply speak and receive intelligent responses in real-time.
In this tutorial, you'll build a complete AI voice agent using LiveKit as the orchestrating framework, with AssemblyAI handling speech-to-text conversion, Cerebras powering the language model responses, and Rime providing natural-sounding text-to-speech.
AI voice agents follow a three-step pipeline when processing user interactions:
- Speech-to-Text (STT): Converts the user's spoken words into text
- Large Language Model (LLM): Processes the text and generates an appropriate response
- Text-to-Speech (TTS): Converts the LLM's text response back into spoken audio
Here's what this looks like in practice:
LiveKit handles the orchestration between these components, managing the real-time audio streams and coordinating the different services.
Setting up your development environment
First, create a new project directory and navigate into it:
We'll use UV as our virtual environment manager. Create a new project and add the required dependencies:
These packages provide:
- livekit-agents: Core LiveKit agents framework
- livekit-plugins-assemblyai: AssemblyAI integration for speech recognition
- livekit-plugins-openai: OpenAI-compatible LLM integration (works with Cerebras)
- livekit-plugins-rime: Rime text-to-speech integration
- livekit-plugins-silero: Voice activity detection
- livekit-plugins-turn-detector: Conversation turn management
- python-dotenv: Environment variable management
Configuring API keys
Create a .env file in your project root to store your API credentials:
Getting your API keys
AssemblyAI: Log into your AssemblyAI dashboard, click "API Keys" in the left sidebar, and copy your existing key or create a new one.
Cerebras: Access your Cerebras Cloud Dashboard, select "API Keys" from the left navigation, and copy an existing key or generate a new one.
Rime: In your Rime dashboard, navigate to Settings → API Tokens, then copy an existing token or create a new one.
LiveKit: From your LiveKit dashboard, go to Settings → API Keys, select your key, and copy the connection details from the dialog box.
Building the voice agent
Rename the auto-generated main file to agent.py and replace its contents with this implementation:
This code sets up:
- AssemblyAI for speech-to-text conversion
- Cerebras (via OpenAI-compatible API) for language model processing using the Llama 3.1 70B model
- Rime for text-to-speech with the "mistv2" model and "marsh" voice
- Silero for voice activity detection
Testing your voice agent locally
Before running the application, download the required plugins:
Once the plugins are loaded, start your voice agent:
The agent will initialize and you'll hear: "Hello, I'm here to help with any questions or tasks you have. What do you need assistance with?"
Try asking questions like:
- "Can you tell me about AI voice agents?"
- "What amazing apps can I make with voice technology?"
The console will display transcriptions of your speech and the agent's responses as they're processed.
Customizing agent behavior
Print user input to the terminal
Output the user’s transcripts by overriding the on_user_turn_completed function:
Monitoring agent responses
Create a wrapper around the TTS function to intercept what the agent is about to speak:
Improving transcription formatting
Enable formatted turns in AssemblyAI to get more readable transcriptions with proper punctuation:
Changing the voice
Rime offers various voice options. Browse available voices in your Rime dashboard under the Pronunciation tab, or check their voice documentation for the complete list.
Popular voice options include:
- marsh (male)
- rainforest (female)
You can also explore newer Arcana model voices for even more realistic synthesis.
Update the voice parameter in your TTS configuration:
Deploying to LiveKit Cloud
Once your agent works locally, deploy it to LiveKit Cloud for global access:
After successful deployment, visit agents-playground.livekit.io and log in with your LiveKit credentials. Select your project to access a web-based interface that's much more user-friendly than the command line.
The playground provides:
- Real-time audio visualization
- Conversation history
- Easy sharing capabilities
- Better debugging tools
What you've built
Your AI voice agent now handles the complete conversation loop:
- Listens for user speech with voice activity detection
- Converts speech to text using AssemblyAI's accurate transcription
- Processes the text with Cerebras's Llama 3.1 model for intelligent responses
- Converts responses back to natural speech using Rime's TTS
- Manages conversation flow and turn-taking automatically
You now have a responsive voice agent that handles complex conversations while maintaining context and providing helpful responses.
Next steps
With your foundation in place, consider these enhancements:
- Custom knowledge bases: Integrate your own data sources for domain-specific responses
- Multi-language support: AssemblyAI supports multiple languages for global applications
- Advanced voice features: Experiment with different Rime models and voice styles
- Integration workflows: Connect your agent to external APIs and databases
- Performance monitoring: Track conversation quality and response times
LiveKit's orchestration, AssemblyAI's Voice AI, Cerebras's language processing, and Rime's voice synthesis provide a solid foundation for building sophisticated voice applications.
You can find the complete code for this tutorial in the AssemblyAI Community GitHub repository.
If you’d prefer to follow along via video, check out our YouTube tutorial on here:
