Twilio phone agent with AssemblyAI Universal-3.6 Pro Realtime

Build an AI phone agent that handles real calls using Twilio Voice + Media Streams and the AssemblyAI Universal-3.6 Pro Realtime model for real-time speech-to-text.

The key detail: Twilio streams 8 kHz μ-law (mulaw) audio, and Universal-3.6 Pro Realtime accepts pcm_mulaw at sample_rate=8000 natively — no resampling, no format conversion.

Architecture

    Incoming call
         │
      Twilio Voice
         │ TwiML → open WebSocket
         ▼
    Your server (/media-stream WebSocket)
         │                        │
         │ mulaw 8kHz audio       │ synthesized mulaw audio
         ▼                        ▲
    AssemblyAI Universal-3.6      ElevenLabs TTS
    Pro Realtime
         │ transcript + turn signal
         ▼
      OpenAI GPT-4o

Prerequisites

• Python 3.11+
• AssemblyAI API key (free account)
• Twilio account with a phone number
• OpenAI API key
• ElevenLabs API key
• ngrok (for local development)

Quick start

git clone https://github.com/kelsey-aai/voice-agent-twilio-universal-3-5-pro
cd voice-agent-twilio-universal-3-5-pro

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env
# Edit .env with your API keys

uvicorn server:app --host 0.0.0.0 --port 8000
ngrok http 8000

Configure Twilio

1. Go to Twilio Console → Phone Numbers
2. Select your number → Voice & Fax
3. Set A Call Comes In to Webhook: https://your-ngrok-url.ngrok.io/incoming-call
4. Call your Twilio number

AssemblyAI WebSocket parameters for Twilio

ASSEMBLYAI_WS_URL = (
    "wss://streaming.assemblyai.com/v3/ws"
    "?speech_model=universal-3-6-pro"
    "&encoding=pcm_mulaw"      # must match Twilio's audio format
    "&sample_rate=8000"        # must match Twilio's 8kHz stream

    "&min_turn_silence=400"    # phone audio: wait a beat longer before ending the turn
    "&max_turn_silence=2000"   # hard ceiling so deliberate callers aren't cut off
)

Phone calls carry more background noise than browser audio, so a slightly longer min_turn_silence reduces premature turn endings. Universal-3.6 Pro Realtime uses end-of-turn detection that combines semantic context with voice activity, and entity-aware endpointing holds the turn open while a caller reads out a phone number, code, or email. You can also set the high-level mode preset (min_latency, balanced, max_accuracy) to shift the whole accuracy/latency balance at once.

Note: end_of_turn_confidence_threshold does not apply to Universal-3.6 Pro Realtime — that parameter belongs to the older universal-streaming models. Use min_turn_silence / max_turn_silence here.Heads up on model IDs: this tutorial uses universal-3-6-pro, the current streaming default. universal-3-5-pro stays available if you need to pin the previous model; if you're still on the legacy u3-rt-pro ID, switch to universal-3-6-pro.

Extending the agent

Sharpen accuracy with conversation context

Universal-3.6 Pro Realtime can transcribe each caller turn in the context of what your agent just said — after your agent asks *"What's your account number?"*, the model is primed for the answer. On a raw WebSocket, push your agent's last reply mid-stream with an UpdateConfiguration message after each agent turn:

await aai_ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "agent_context": "Thanks for calling Acme. What's the account number on the 
policy?",
}))

Handle noisy lines with Voice Focus

For calls from cars, speakerphones, or noisy rooms, add Voice Focus to isolate the caller's voice server-side:

ASSEMBLYAI_WS_URL += "&voice_focus=far-field"

Add keyterm prompting

ASSEMBLYAI_WS_URL += "&keyterms_prompt=YourBrand&keyterms_prompt=SpecialTerm"

Add post-call transcription

import assemblyai as aai
transcriber = aai.Transcriber()
transcript = transcriber.transcribe(recording_url)
print(transcript.text)

Deploy to Railway or Render

# Railway
railway login && railway init && railway up

# Render — create a Web Service pointing to this repo
# Start: uvicorn server:app --host 0.0.0.0 --port $PORT

Resources

• Universal-3.6 Pro Realtime streaming API reference
• AssemblyAI streaming getting started
• Twilio Media Streams docs

Frequently asked questions

What is the best speech-to-text model for a Twilio phone agent?

AssemblyAI's Universal-3.6 Pro Realtime is well suited to Twilio phone agents because it accepts Twilio's native 8 kHz pcm_mulaw stream directly — no resampling or transcoding — and posts a 5.19% word error rate on AssemblyAI's English voice-agent benchmark, with a 0.96% pooled semantic WER on Pipecat's open benchmark. Its turn detection, which combines semantic context with voice activity, and Context Carryover are built for the back-and-forth of phone conversations, and it handles the account numbers, phone numbers, and spelled-out entities that come up constantly on support calls.

How do I connect Twilio Media Streams to AssemblyAI?

Point your Twilio number's incoming-call webhook at a server that returns TwiML opening a Media Stream to your WebSocket endpoint. In that endpoint, open a WebSocket to wss://streaming.assemblyai.com/v3/ws?speech_model=universal-3-6-pro&encoding=pcm_mulaw&sample_rate=8000, forward Twilio's base64 media payloads as audio, and act on AssemblyAI's Turn messages. Because both sides speak 8 kHz μ-law, no format conversion is needed.

Do I need to convert Twilio's audio format for AssemblyAI?

No. Twilio streams 8 kHz μ-law and Universal-3.6 Pro Realtime accepts encoding=pcm_mulaw at sample_rate=8000 natively, so you can pass Twilio's audio straight through. This removes a whole class of latency and quality bugs that come from resampling on the fly.

How do I reduce false turn endings on noisy phone calls?

Raise min_turn_silence (e.g. to 400 ms) so brief line noise doesn't trigger an early turn end, keep a max_turn_silence ceiling (e.g. 2000 ms) so deliberate callers aren't cut off, and enable voice_focus=far-field to suppress background noise server-side. For a one-line adjustment, set mode="max_accuracy" to bias the whole pipeline toward cleaner transcripts.

How much does a Twilio + AssemblyAI phone agent cost?

AssemblyAI's Universal-3.6 Pro Realtime is $0.45/hr for speech-to-text, billed on session duration with keyterm prompting included. Twilio Voice, your LLM (e.g. OpenAI), and your TTS provider (e.g. ElevenLabs) bill separately through their own accounts. AssemblyAI offers a free tier to build and test before going live.