Universal-3.5 Pro Realtime vs. Voice Agent API: Which one should you actually build on?

The core decision teams face: should you use a managed voice agent service or build your own orchestration on top of a speech-to-text model?

AssemblyAI offers both paths running on the same underlying speech model. One provides a single WebSocket bundling speech-to-text, LLM, and text-to-speech. The other delivers raw transcription and lets you own every layer above it.

"A voice agent API is a service that handles the full back-and-forth of a spoken conversation"

so you don't need to integrate separate services yourself.

What is a voice agent API, anyway?

A real-time voice agent must listen, detect when the speaker finishes, transcribe speech, query an LLM, convert responses to audio, and play them back—all with minimal latency. A managed voice agent API collapses this into one connection rather than three separate services with three separate bills.

AssemblyAI's Voice Agent API is not a single speech-to-speech model but rather orchestrates speech-to-text, an LLM, and text-to-speech behind one interface. This cascading architecture preserves model-level control—you can swap the LLM, adjust the prompt, or select from available voices—instead of being locked into one opaque end-to-end model.

How the Voice Agent API works

The entire system runs over one WebSocket at wss://agents.assemblyai.com/v1/ws. You open the connection, configure the session, stream audio in, and receive audio out.

Key capabilities:

Turn detection — The agent determines when the caller has finished speaking rather than interrupting on the first silence.

Interruption handling — When a caller speaks while the agent responds, the API detects this and stops the agent, mimicking natural human conversation.

Tool calling — The LLM can invoke your APIs mid-conversation to look up orders, check availability, or create tickets.

Session resumption — A 30-second window allows reconnection with context intact if the socket drops.

The service delivers roughly 1-second end-to-end latency at $4.50/hr with unlimited concurrency, 34 voices, turn detection, interruption handling, and tool calling built in. For comparison, OpenAI's Realtime API costs around $18/hr.

The code

// Configure the agent once the socket is open
ws.onopen = () => ws.send(JSON.stringify({
  type: "session.update",
  session: {
    system_prompt:
      "You are a friendly support agent for an online bookstore. " +
      "Keep replies short and conversational. Use tools to look up orders.",
    greeting: "Hi! How can I help with your order today?",
    output: { voice: "ivy" }, // pick from 34 voices
    tools: [
      {
        name: "lookup_order",
        description: "Look up an order by its ID. Use when the caller references an order.",
        parameters: {
          type: "object",
          properties: { order_id: { type: "string" } },
          required: ["order_id"],
        },
      },
    ],
  },
}));

ws.onmessage = ({ data }) => {
  const m = JSON.parse(data);
  switch (m.type) {
    case "session.ready":
      // Save m.session_id. Now it's safe to start streaming input.audio.
      break;

    case "reply.audio":
      // Base64 PCM of the agent speaking—pipe this to your speaker/telephony layer.
      playReplyAudio(m.data);
      break;

    case "reply.done":
      // On barge-in, flush the buffered audio immediately.
      if (m.status === "interrupted") flushPlayback();
      break;

    case "tool.call":
      // The LLM wants to run a tool. Execute it and send tool.result back.
      handleToolCall(m);
      break;

    case "transcript.user":
    case "transcript.agent":
      // Live transcripts of both sides of the conversation.
      break;

    case "session.error":
      console.error(m.message);
      break;
  }
};

Audio is PCM16 mono at 24 kHz, base64-encoded. After receiving session.ready, stream the caller's audio:

// pcmString is the raw PCM16 bytes as a binary string
ws.send(JSON.stringify({ type: "input.audio", audio: btoa(pcmString) }));

Tool calling follows a tight round-trip pattern where arguments arrive pre-parsed and results must be JSON-encoded strings with the matching call_id echoed:

Code — Plain text

→ { type:"tool.call",   call_id:"c_123", name:"get_weather", arguments:{ location:"London" } }
← (run your tool)
→ { type:"tool.result", call_id:"c_123", result:"{\"temp_c\":22}" }

What you can build with it

Support and service agents

Voice agents answering calls, authenticating callers, looking up account information with tool calls, and resolving common cases without human escalation. Tool calling transforms this from a scripted IVR into an agent that can actually perform actions. If you're weighing the architecture for a deployment like this, our guide on building AI voice agents walks through the moving parts.

Companions and coaching

Language tutors, interview practice, wellness check-ins, and sales-call coaching benefit from natural turn-taking and interruption handling. Low latency and native barge-in make conversations feel natural rather than stilted.

Clinical intake and screening

Pre-visit intake, symptom screening, appointment scheduling, and follow-up calls. Healthcare deployments require high accuracy on clinical terminology and defensible data handling.

Medical Mode significantly improves recognition of drug names, procedures, and anatomical terms. AssemblyAI operates as a HIPAA business associate and offers a Business Associate Addendum (BAA) for customers processing protected health information (PHI).

When you'd reach for Universal-3.5 Pro Realtime instead

Universal-3.5 Pro Realtime provides speech-to-text only—you bring your own LLM and text-to-speech. Choose this path when you need control and specialization:

• Teams with existing investments in specific LLM stacks, fine-tuned models, custom RAG pipelines, or licensed TTS voices
• Non-conversational use cases like live captioning, real-time meeting notes, voice search, or command parsing
• Applications where paying for a full voice-agent orchestrator would include unnecessary overhead

If you're building a voice agent from streaming STT plus your own LLM and TTS, our breakdown on choosing an STT API for voice agents covers what to actually evaluate, and the real-time speech-to-text primer covers the streaming fundamentals.

Key features

Context, not just transcription — Pass the agent's question via agent_context so the model hears replies through the lens of that question. "User at assembly a i dot com" resolves to the correct email address. Across 20,000 voice agent audio files, agent context reduced word error rate by 10.2%, with largest gains on fabrications and short utterances.

The model also maintains a rolling conversation memory automatically, using it as context for each new turn.

Speaker isolation — voice_focus isolates the primary speaker and suppresses background speech. Use near-field for headsets and phones, far-field for rooms and kiosks. Speaker labels run live during calls then re-cluster at stream end, sending a single revision for up to 10 speakers—live labels during the call with accuracy-grade results within about half a second of ending.

18 languages with steering — Runs at flagship accuracy in 18 languages with mid-sentence code-switching for bilingual calls. The language_code parameter commits the model to one language when known, preventing wrong-language slips on short audio. Use keyterm prompting to inject domain vocabulary.

Three performance modes — Open a WebSocket and choose: min_latency for fastest transcripts, balanced (default) for strong all-around performance, or max_accuracy for noisy, far-field audio. End-of-turn detection reads tonality, pacing, and rhythm—not just silence—achieving around 300ms detection. On Pipecat's open STT benchmark of real agent conversations, Universal-3.5 Pro Realtime achieves 6.99% pooled word error rate.

The code

import pyaudio
from assemblyai.streaming.v3 import (
    BeginEvent,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    TerminationEvent,
    TurnEvent,
)

client = RealTimeTranscriber(
    RealTimeTranscriberOptions(terminate_timeout=30.0),
    api_key=api_key,
)

client.on(RealTimeEvents.Begin, on_begin)
client.on(RealTimeEvents.Turn, on_turn)
client.on(RealTimeEvents.Termination, on_terminated)
client.on(RealTimeEvents.Error, on_error)

client.connect(
    RealTimeParameters(
        speech_model="u3-rt-pro",
        sample_rate=16000,
    )
)

# Capture mic audio with PyAudio and feed it to the transcriber
FRAMES_PER_BUFFER = 800  # 50ms at 16kHz
pa = pyaudio.PyAudio()
mic = pa.open(format=pyaudio.paInt16, channels=1, rate=16000,
              input=True, frames_per_buffer=FRAMES_PER_BUFFER)

try:
    while True:
        client.stream(mic.read(FRAMES_PER_BUFFER, exception_on_overflow=False))
except KeyboardInterrupt:
    pass
finally:
    mic.stop_stream()
    mic.close()
    pa.terminate()
    client.disconnect(terminate=True)

u3-rt-pro is the streaming Pro model; Universal-3.5 Pro Realtime is its new default. Most teams receive the upgrade automatically. Sessions bill by duration, so disconnect explicitly when done or they auto-close after 3 hours.

How to choose: the actual rule

Do you need to own the LLM and TTS layers, or do you just need a working conversation?

Use the Voice Agent API for two-way spoken conversations where you're satisfied letting AssemblyAI orchestrate the stack. You'll ship faster with one bill, one log stream, and no custom turn-detection or barge-in code—and you still get Universal-3.5 Pro Realtime under the hood.

Use the streaming model when you need a custom LLM, specific TTS voice, non-conversational use case, or surgical control over every layer including direct access to agent_context, voice_focus, language_code, and latency modes.

Pricing

Universal-3.5 Pro Realtime: $0.45/hr ($0.0075/min) base rate, unchanged from the previous Pro model. Includes rolling memory, agent_context, and keyterm prompting. Add-ons layer only as used: diarization with revision (+$0.12/hr), prompting (+$0.05/hr), voice isolation (+$0.10/hr). Unlimited concurrency, no rate limits, no upfront commitments, with volume discounts at scale.

Voice Agent API: Flat $4.50/hr. Refer to the pricing page for current rates.

Getting started

For the managed path: Grab an API key, connect to wss://agents.assemblyai.com/v1/ws, send a session.update, and start streaming audio. The example code above represents most of the work. For telephony or media server integration, AssemblyAI partners with Twilio, LiveKit, and Daily.

For the bring-your-own-stack path: Start with the streaming getting-started guide, wire transcripts into your LLM, and pipe responses to your TTS. You trade build time for total control and direct access to every context feature.

For recorded audio: Use the async speech-to-text API running on Universal-3 Pro for highest accuracy on call recordings, podcasts, and meeting archives.

Frequently asked questions

What is Universal-3.5 Pro Realtime?

Universal-3.5 Pro Realtime is AssemblyAI's flagship realtime speech-to-text model and new streaming default. It runs over WebSocket, delivers around 300ms end-of-turn detection, and posts 6.99% pooled word error rate on Pipecat's open STT benchmark. It features context via agent_context, maintains rolling conversation memory, isolates speakers with voice_focus, and supports 18 languages with mid-sentence code-switching. It also powers the Voice Agent API's transcription layer.

How much does Universal-3.5 Pro Realtime cost?

Base price is $0.45/hr with rolling memory, agent_context, and keyterm prompting included. Add-ons: diarization with revision (+$0.12/hr), prompting (+$0.05/hr), voice isolation (+$0.10/hr). Unlimited concurrency, no rate limits. The Voice Agent API costs a separate flat $4.50/hr. Check the pricing page for current rates.

What is the best speech-to-text API for voice agents?

The best option has low latency, accurate real-time transcription, and conversation-designed features like turn detection, speaker diarization, and context. Universal-3.5 Pro Realtime delivers around 300ms end-of-turn detection with unlimited concurrency, keyterm prompting, voice isolation, and the ability to accept the agent's question as input—making it a common foundation for custom voice-agent stacks. For faster deployment, the Voice Agent API bundles the same transcription model with LLM and TTS layers.

When should I use the managed Voice Agent API vs. just streaming STT?

Use the managed Voice Agent API for two-way spoken conversations when you want to ship fast without owning LLM and TTS layers. Use Universal-3.5 Pro Realtime directly when you need a custom LLM, specific TTS voice, surgical layer control, direct access to context features like agent_context and voice_focus, or a non-conversational use case like live captioning or voice search. The deciding factor is whether you need to own the layers above transcription or simply need a working conversation. Either way, the same speech model handles transcription.

AssemblyAI Voice Agent API vs OpenAI/Vapi: how do they compare?

AssemblyAI's Voice Agent API is a cascading orchestration model on one WebSocket at a flat $4.50/hr with built-in turn detection, interruption handling, tool calling, and session resumption, running on Universal-3.5 Pro Realtime underneath. OpenAI's Realtime API is a speech-to-speech model costing around $18/hr that locks you into the OpenAI stack, while platforms like Vapi sit atop separate STT, LLM, and TTS providers you assemble. Main tradeoffs are cost, model-level control you retain, and whether the architecture cascades or runs end-to-end speech-to-speech.

‍