Managed voice agent platform vs. streaming speech-to-text plus your own stack: how to choose

If you need a working phone or web agent this week and your call flow is standard, a managed voice agent platform such as Vapi, Retell, or OpenAI Realtime gets you there fastest. If you need control over the LLM, the voice, turn-taking, or cost per call, compose your own stack: streaming speech-to-text and text-to-speech from a speech provider, your LLM, and an orchestrator like LiveKit, Pipecat, or your own loop. The deciding question is whether you need to own the layers above transcription, or whether you just need a working conversation.

This guide is vendor-neutral on the orchestration question. AssemblyAI doesn’t sell a managed agent; we’re the speech layer you build on, with Universal-3.6 Pro Realtime for what the agent hears and the Text-to-Speech API for what it says. That speech layer also runs inside managed platforms: Universal-3.6 Pro Realtime drops into Vapi as well as LiveKit and Pipecat. So the choice below isn’t about whose speech model you use. It’s about who owns the loop.

What is a voice agent platform, anyway?

A real-time voice agent must listen, detect when the speaker finishes, transcribe speech, query an LLM, convert the response to audio, and play it back, all with minimal latency. A voice agent platform is a service that handles that full back-and-forth of a spoken conversation, so you don’t integrate separate services yourself. A managed platform collapses the stack into one product and one bill rather than three separate services with three separate bills.

Most managed platforms are cascades under the hood: speech-to-text, an LLM, and text-to-speech behind one interface. Some, like OpenAI Realtime, use a single multimodal model for speech in and speech out. Either way, the platform owns the orchestration, and you configure it.

What a managed platform gives you

The appeal is that the hard conversational plumbing comes built in:

Turn detection. The agent determines when the caller has finished speaking rather than interrupting on the first silence.

Interruption handling. When a caller speaks while the agent responds, the platform detects it and stops the agent, mimicking natural conversation.

Tool calling. The LLM can invoke your APIs mid-conversation to look up orders, check availability, or create tickets.

The trade is control. You get the platform’s turn-taking, the platform’s voices, and the platform’s view of the call. For many teams that’s exactly right. For others it becomes the ceiling: a call flow that needs custom logic between transcript and LLM, a licensed voice, or a cost-per-call target the platform pricing can’t hit.

What composing your own stack gives you

Here you take speech-to-text and text-to-speech as APIs, bring your own LLM (or reach many through LLM Gateway), and let an orchestrator wire them together. LiveKit and Pipecat are the common open choices; the LiveKit and Pipecat guides show Universal-3.6 Pro Realtime as a drop-in. You can also write the loop yourself against the streaming WebSocket.

Choose this path when you need control and specialization:

• Teams with existing investments in specific LLM stacks, fine-tuned models, custom RAG pipelines, or licensed TTS voices
• Call flows that need logic between the transcript and the LLM, or a bespoke turn-taking policy
• Non-conversational use cases like live captioning, real-time meeting notes, voice search, or command parsing, where a full agent platform is unnecessary overhead

If you’re going this route, our breakdown on choosing an STT API for voice agents covers what to evaluate, and our real-time speech-to-text primer covers the streaming fundamentals.

What the speech layer brings to your stack

Context, not just transcription. Pass the agent’s last reply via agent_context so the model hears the caller’s answer through the lens of the question. “User at assembly a i dot com” resolves to the correct email address. Across 20,000 voice agent audio files, agent_context reduced word error rate by 10.2%, with the largest gains on fabrications and short utterances. The model also keeps a rolling conversation memory and uses it as context for each new turn.

Speaker isolation. voice_focus isolates the primary speaker and suppresses background speech. Use near-field for headsets and phones, far-field for rooms and kiosks. It hears the caller, not the room. Streaming speaker labels support up to 10 speakers, with revisions that only ever change speaker assignments, never the text.

32 languages with steering. Universal-3.6 Pro Realtime runs in 32 languages with automatic language detection and mid-sentence code-switching for bilingual calls. The language_codes parameter steers the model when you know the caller’s language, preventing wrong-language slips on short audio. Keyterms prompting injects domain vocabulary at no extra charge.

Turn detection you can tune. End-of-turn detection combines semantic context with voice activity, not silence alone, and holds the turn open when a caller is partway through an entity like a phone number. Median endpoint latency is 537 ms, and on Pipecat’s open STT benchmark the final transcript lands a median 307 ms after the speaker stops. You can adjust min_turn_silence and max_turn_silence mid-stream, or force an endpoint.

The code

With LiveKit or Pipecat, you set the model in the plugin and the framework handles the socket. Underneath, it’s a WebSocket with an Authorization header, binary audio frames in, and JSON Turn messages out. Here’s the speech side of a hand-rolled loop in Node.js; the LLM, TTS, and barge-in steps are labelled stubs for your own providers and policy.

import WebSocket from "ws";

const params = new URLSearchParams({
  speech_model: "universal-3-6-pro",
  sample_rate: "16000",
  encoding: "pcm_s16le",
});

const ws = new WebSocket(`wss://streaming.assemblyai.com/v3/ws?${params}`, {
  headers: { Authorization: process.env.ASSEMBLYAI_API_KEY },
});

ws.on("message", async (data) => {
  const msg = JSON.parse(data);
  switch (msg.type) {
    case "Begin":
      // Unrecognized parameters are ignored silently, so confirm the model
      console.log("session", msg.id, msg.configuration.model);
      break;

    case "SpeechStarted":
      handlePossibleBargeIn(); // your interruption policy
      break;

    case "Turn": {
      // Partial turns arrive with end_of_turn: false; act on the final one
      if (!msg.end_of_turn) break;
      const reply = await callLLM(msg.transcript); // your LLM, or LLM Gateway
      // The agent's reply becomes context for the caller's next turn
      ws.send(JSON.stringify({ type: "UpdateConfiguration", agent_context: reply }));
      synthesize(reply); // your TTS provider
      break;
    }

    case "Termination":
      ws.close();
      break;
  }
});

// micStream emits 16 kHz PCM16 mono audio in 50 ms chunks
micStream.on("data", (chunk) => ws.send(chunk));

// When the call ends, close the session so billing stops:
// ws.send(JSON.stringify({ type: "Terminate" }));

The code uses universal-3-6-pro, the Universal-3.6 Pro Realtime model ID and the streaming default if you omit speech_model. universal-3-5-pro stays supported if you need to pin the previous model. Each Turn message re-transcribes the full turn, so replace earlier partial text instead of appending. Sessions bill by duration, so send Terminate when a call ends; sessions close on their own after 3 hours. Browsers can’t set headers, so browser clients connect with a temporary token generated on your server.

How to choose: the decision table

Do you need to own the LLM, TTS, and turn-taking layers, or do you just need a working conversation? Start there, then check the criteria below.

Control

If your product’s differentiation lives in how the agent converses (when it yields the floor, what it does between a transcript and a tool call, which voice it uses), you want to own the loop. If differentiation lives in your backend and the conversation is a standard intake or lookup, a platform’s defaults are fine.

Cost shape

Managed pricing bundles everything into one rate, which is easy to forecast but hard to optimize. A composed stack itemizes: Universal-3.6 Pro Realtime is $0.45/hr base, pay-as-you-go with no minimums, and each other line is a separate decision. At high call volume, being able to swap one stage for a cheaper one without migrating the whole agent is usually worth the integration work.

Latency budget

A voice agent turn is a relay race, and latency is the sum of the handoffs. On a platform, you get its total. In your own stack, you can see each stage’s contribution and spend the budget where it matters: stream transcripts into the LLM, start synthesis on the first sentence, and keep turn detection responsive without cutting callers off.

Debuggability

When an agent gives a wrong answer, you need to know whether the transcript was wrong, the model reasoned badly, or the tool returned bad data. Own the stack and you can log every Turn message, LLM request, and tool result under one call ID.

Compliance

AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI. In a composed stack, you sign a BAA with each provider that touches PHI. On a managed platform, confirm the BAA covers every provider the platform routes audio through.

What you can build either way

Support and service agents

Voice agents answering calls, authenticating callers, looking up account information with tool calls, and resolving common cases without human escalation. Tool calling turns this from a scripted IVR into an agent that can act. Entity accuracy on names, account codes, and phone numbers decides whether those tool calls hit the right record. Our guide on building AI voice agents walks through the moving parts.

Clinical intake and screening

Pre-visit intake, symptom screening, appointment scheduling, and follow-up calls. Healthcare deployments need high accuracy on clinical terminology and defensible data handling. Medical Mode (+$0.15/hr on streaming) improves recognition of drug names, procedures, and anatomical terms. Pair it with the BAA scoping above.

Pricing

Universal-3.6 Pro Realtime: $0.45/hr base, billed per second, pay-as-you-go with no minimums. Keyterms prompting and agent_context are included. Add-ons apply only when used: streaming diarization +$0.12/hr, Voice Focus +$0.10/hr, Medical Mode +$0.15/hr. There’s no cap on open sessions; new sessions per minute are rate limited (5 on the free tier, 100+ on paid) and scale automatically. The free tier includes up to 333 hours of streaming with no credit card.

Text-to-Speech API: launching October 28, 2026, so speech in and speech out come from one vendor. LLM: your provider’s rates, directly or through LLM Gateway. Current rates for every AssemblyAI product are on the pricing page.

Getting started

For the managed path: pick a platform, and check whether it lets you choose the speech-to-text model. Vapi supports Universal-3.6 Pro Realtime, so you can keep the platform’s speed and still choose the speech layer.

For the bring-your-own-stack path: start with the LiveKit or Pipecat guide above, or the streaming WebSocket code in this post. Wire transcripts into your LLM, pipe responses to your TTS, and send each reply back as agent_context. You trade build time for total control.

For recorded audio: use the async speech-to-text API running on Universal-3.5 Pro ($0.21/hr, native code-switching across 18 languages) for call recordings, podcasts, and meeting archives.

Frequently asked questions

What is the difference between a managed voice agent platform and building your own stack?

A managed voice agent platform, such as Vapi, Retell, or OpenAI Realtime, runs speech-to-text, the LLM, text-to-speech, and orchestration for you behind one product. Building your own stack means composing streaming speech-to-text and text-to-speech APIs with your own LLM and an orchestrator like LiveKit, Pipecat, or a custom loop. The managed path ships faster; the composed path gives you control over every layer and an itemized cost per call.

When should I build my own voice agent stack instead of using a platform?

Build your own stack when you need a custom or fine-tuned LLM, a specific TTS voice, logic between the transcript and the LLM, a bespoke turn-taking policy, or tight control over cost per call. It also fits non-conversational use cases like live captioning or voice search. Use a managed platform when the call flow is standard and speed to launch matters most.

What is Universal-3.6 Pro Realtime?

Universal-3.6 Pro Realtime is AssemblyAI’s flagship streaming speech-to-text model and the streaming default. It runs over WebSocket, posts a 5.19% word error rate on AssemblyAI’s English voice-agent benchmark, and on Pipecat’s open STT benchmark posts a 0.96% pooled semantic WER with the final transcript landing a median 307 ms after the speaker stops. It supports agent_context, Voice Focus, keyterms prompting, and 32 languages with mid-sentence code-switching.

How much does Universal-3.6 Pro Realtime cost?

Universal-3.6 Pro Realtime costs $0.45/hr base, billed per second with no minimums, and includes keyterms prompting and agent_context. Optional add-ons are streaming diarization (+$0.12/hr), Voice Focus (+$0.10/hr), and Medical Mode (+$0.15/hr). There is no cap on open sessions, and the free tier includes up to 333 hours of streaming.

Can I use AssemblyAI with Vapi, LiveKit, or Pipecat?

Yes. Universal-3.6 Pro Realtime drops into LiveKit and Pipecat as a speech-to-text plugin and is available in Vapi, so you can pick AssemblyAI’s speech layer whether you compose your own stack or use a managed platform. AssemblyAI’s documentation includes setup guides for each.

What is the best speech-to-text API for voice agents?

The best option combines low latency, accurate real-time transcription on entities like names and numbers, and conversation-aware features such as semantic turn detection and context from the agent’s last reply. Universal-3.6 Pro Realtime ranks 1st of 22 systems on AssemblyAI’s voice-agent benchmark suite with a 5.19% WER averaged over noise conditions and the lowest entity error rate at 14.41%, making it a common foundation for composed voice agent stacks.