Every AI voice agent is three models wearing a trench coat.
Speech-to-text hears the caller. An LLM figures out what they want. Text-to-speech says something back. That’s the whole architecture, and on a whiteboard it takes about nine seconds to draw.
Then you try to ship it, and you discover the drawing left out everything hard. Who decides the caller is finished talking? What happens when they interrupt the agent mid-sentence? Where does the conversation history live? What calls your CRM, and what happens when the CRM takes four seconds to answer?
That layer (timing, turn-taking, interruptions, tool calls and state) is orchestration. It’s the part of a voice agent nobody demos and everybody rebuilds. That’s why a category of tools now exists to hand it to you.
The eight orchestration tools delivering results in production today are LiveKit, Pipecat, Telnyx, Vision Agents, Vapi, Retell, Synthflow and Bland. Three of them (LiveKit, Pipecat and Vision Agents) are open source and you can self-host them. The other five are managed platforms. Here’s how to tell which bet matches your team, starting with the open-source vs managed split most lists skip.
What are AI voice agents?
The difference between a voice agent and an IVR tree is the difference between a conversation and a flowchart.
An IVR gives you four options and waits. Press 1 for sales. Press 2 for billing. Miss the window and it starts over.
A voice agent takes “I need to check my recent order and also ask about your return policy” and handles both, in the order the caller raised them. It holds context across turns, so “and how long does that take?” three sentences later still refers to the return. It handles being interrupted. It calls your order system mid-conversation and reads back a real tracking number.
The caller no longer has to translate what they want into the system’s vocabulary. Our primer on what AI voice agents are and how they work covers the ground floor.
How do AI voice agents work?
Three models in a loop, plus the machinery that keeps the loop honest.
| Component | Function | Key requirement |
|---|---|---|
| Speech-to-text (STT) | Converts caller speech to text | Low latency and entity accuracy |
| Large language model (LLM) | Understands intent, generates the response | Context awareness and tool calling |
| Text-to-speech (TTS) | Converts the response to natural audio | Human-like quality, fast first byte |
Around that loop, a production system also needs turn detection that knows a pause from a finished thought, and interruption handling that stops the agent mid-word. It also needs context management across turns, tool calling into your own APIs, and error recovery when any of the above misfires.
The three models are components you can swap. The machinery around them is where orchestration tools earn their keep.
One asymmetry is worth naming first: errors at the STT layer are the only ones you can’t recover downstream. A mediocre LLM response can be fixed with a better prompt. A flat TTS voice can be swapped in an afternoon. A misheard account number is simply gone, and every layer after it reasons confidently from the wrong input. That’s why entity accuracy matters more than pooled word error rate when you evaluate a transcriber for agents.
Use your own call audio and check latency and entity accuracy before you wire anything into Vapi, LiveKit, Pipecat or your own stack.
Open-source vs managed: the first decision
Before comparing features, decide whether you want to run the orchestrator yourself.
Open-source frameworks give you the code. You self-host, modify anything and pay only for infrastructure and model providers. You also own uptime, scaling, monitoring and telephony.
Managed platforms run the orchestration for you, usually with a per-minute fee on top of provider costs. You ship faster, but you work inside the platform’s conversation model.
Note that “developer-friendly” isn’t the same as open source. Vapi is API-first and very developer-oriented, but it’s a proprietary managed service. So are Retell, Synthflow, Bland and Telnyx.
Open-source voice agent frameworks compared
| Framework | Repo | License | Language | Transport | AssemblyAI integration |
|---|---|---|---|---|---|
| Pipecat | pipecat-ai/pipecat | BSD-2-Clause | Python (client SDKs for JS, React, iOS, Android) | Transport-agnostic (WebRTC, WebSocket, telephony) | Official plugin |
| LiveKit Agents | livekit/agents (Node: agents-js) | Apache-2.0 | Python, Node.js | WebRTC SFU, SIP | Official plugin |
| Vision Agents (Stream) | GetStream/Vision-Agents | Apache-2.0 | Python | Stream edge WebRTC | Listed STT integration |
| TEN Framework | TEN-framework/ten-framework | Apache 2.0 with conditions | Multi-language (C++, Python, Go, TypeScript) | RTC, WebSocket, SIP extension | Bring your own STT extension |
TEN Framework isn’t in our top eight because its sweet spot is multimodal and hardware experiments rather than production phone agents. It’s worth a look if you’re building on devices. For a deeper head-to-head on the two most popular frameworks and the two most popular managed platforms, see Vapi vs LiveKit vs Retell vs Pipecat.
What to consider when choosing an orchestration tool
Technical fit. Do you want code-level control, or a no-code builder? Cloud, or self-hosted behind your own firewall? These are constraints, not preferences. A data residency requirement can eliminate half this list.
Performance. Latency low enough that the conversation doesn’t feel laggy. Integration with the CRM and telephony you already run. Headroom for your peak hour.
Implementation capacity. This is the one teams get wrong. Customizable frameworks demand more engineering to run, and teams almost always underestimate that maintenance load. An open-source framework you can’t staff is worse than a managed platform you find slightly constraining.
Pricing shape. Managed tools charge on some mix of minutes, API calls and feature tiers. For example, Vapi lists a $0.05/min hosting fee with providers billed at cost, and LiveKit Cloud has a free tier with 1,000 agent session minutes a month. Model the bill at 10x your pilot volume, and remember you’ll pay the orchestrator plus your STT, LLM and TTS providers.
That last point is the argument for skipping orchestration entirely. Our Voice Agent API replaces separate STT, LLM and TTS providers with a single WebSocket at a flat $4.50/hr. The orchestration still happens; you just don’t configure it.
Top 8 orchestration tools for building AI voice agents
1. LiveKit: open-source with maximum control
Open source: Yes (Apache-2.0). Managed option: LiveKit Cloud.
LiveKit Agents is fully open source, so there’s no ceiling on what you can change. Teams modify the codebase directly and tailor agents to use cases a hosted platform would never prioritize.
It handles voice, video and text in one framework, with function calling, turn detection and telephony through SIP. AssemblyAI ships a drop-in plugin, so adding Universal-3.6 Pro Realtime for real-time transcription is a one-line change. Our LiveKit voice agent tutorial shows the setup, and agent context carryover in LiveKit shows how to pass the agent’s question to the transcriber.
Best for: teams with real engineering capacity who need control over every layer and expect to keep customizing after launch.
2. Pipecat: flexible open-source orchestration
Open source: Yes (BSD-2-Clause). Maintained by: Daily and the community.
Pipecat is a vendor-neutral Python framework built for mixing and matching. Every component is swappable, so you choose each model on performance, cost and fit rather than taking what a platform bundles.
It manages multi-turn context and real-time media transport natively, and it ships client SDKs for JavaScript, React, iOS and Android. AssemblyAI has a dedicated plugin. Pipecat’s open STT benchmark is the one we publish Universal-3.6 Pro Realtime’s numbers against. See our guide to building and deploying a Pipecat voice agent.
Best for: Python teams who want full pipeline customization without vendor lock-in.
3. Telnyx: full-stack orchestration with telephony built in
Open source: No (managed).
Telnyx runs voice agent orchestration on the same private carrier network that carries the calls. That removes the third-party audio hops between telephony and inference, which helps latency when your agent lives on a phone line.
Phone numbers, SIP trunking and programmable call control sit alongside turn detection and tool calling. You choose your STT engine per call, and AssemblyAI is among the supported providers.
Best for: receptionists, appointment setting and support lines, where the phone network is the product surface.
4. Vision Agents: video-first with native voice support
Open source: Yes (Apache-2.0). Runs on: Stream’s edge network.
Vision Agents, from Stream, starts from video rather than retrofitting it onto a voice system. Agents that need to see are badly served by frameworks where video is an afterthought. Think of a coaching app watching form, or a support agent looking at what the customer sees.
It supports pluggable computer-vision processors, including YOLO, Roboflow and custom PyTorch or ONNX models. Turn-taking and speaker diarization are built in, and AssemblyAI is a listed speech-to-text integration.
Best for: multimodal agents where vision is a first-class input.
5. Vapi: developer-friendly managed orchestration
Open source: No. Vapi is a proprietary managed platform, even though it’s API-first.
Vapi lets engineers define assistants in JSON and check them into source control, with visual tooling for non-engineers. Cross-channel support (web, phone, in-app) makes it a natural fit for customer service. The integration catalog covers most CRMs and helpdesks, and tool calling and multi-language support are built in.
The AssemblyAI integration is native. Add your key under Settings → Transcriber Providers, then set the transcriber block:
{
"transcriber": {
"provider": "assembly-ai",
"speechModel": "universal-3-6-pro",
"language": "en",
"keytermsPrompt": ["YourBrand", "SpecialTerm"]
}
}That’s Universal-3.6 Pro Realtime, with keyterm prompting included at the base rate. There’s a full walkthrough in our Vapi voice agent tutorial.
Best for: developer teams that want managed infrastructure and code-level configuration.
6. Retell: managed phone agents with a focus on conversation flow
Open source: No (managed).
Retell is a managed platform for building and deploying phone agents, with a low-code builder, an API and built-in telephony. It puts a lot of engineering into turn-taking, aiming to remove the dead air that tells callers they’re talking to software.
Interruption handling, multi-language support, and web and telephony deployment are included. Retell’s engineers have described running cascading STT-LLM-TTS pipelines for most agents, with fallbacks at each layer. We covered that in our panel recap with Retell and Super. AssemblyAI is available as a transcription option in Retell.
Best for: teams that want a phone agent live quickly and value conversational flow out of the box.
7. Synthflow: no-code for faster deployment
Open source: No (managed).
Synthflow’s bet is that most voice agents are variations on a few patterns, and that a template beats a blank canvas. The drag-and-drop builder lets a business team ship a working agent without an engineer.
About 200 pre-built integrations, multi-language support and usage-based pricing round it out.
Best for: SMBs and departments that need something live this quarter without engineering capacity. You accept the platform’s conversation model in exchange.
8. Bland: self-hosted security for enterprise
Open source: No. Proprietary, but deployable on infrastructure you control.
Bland’s differentiator is architectural: the stack can run behind your firewall. That turns the compliance conversation from a vendor questionnaire into an infrastructure decision.
You still get human-like voice output, custom prompts and guardrails, an analytics dashboard and warm transfer to a human.
Best for: financial services, healthcare and government, where data governance rules out sending audio to a third party. For healthcare, whichever platform you choose, the model provider handling PHI needs a signed Business Associate Addendum (BAA). AssemblyAI offers a BAA you can sign without a sales call.
Plug Universal-3.6 Pro Realtime into LiveKit, Pipecat, Vapi or your own stack. Keyterm prompting is included at $0.45/hr.
How AssemblyAI fits in the voice AI ecosystem
We don’t make an orchestration tool. We make the layer underneath all eight.
A voice agent is only as good as its ability to understand what people say. Universal-3.6 Pro Realtime (universal-3-6-pro) is our streaming model built for agent conversations, released September 29, 2026.
The accuracy. A 5.19% word error rate and a 14.4% entity error rate on AssemblyAI’s English voice-agent benchmark of 12,460 scripted scenarios. Deepgram Flux EN posts 13.50% and 30.1% on the same set. On Pipecat’s open STT benchmark, it posts a 0.96% pooled semantic word error rate.
The turn-taking. Final transcripts land a median 307 ms after the caller stops speaking on Pipecat’s benchmark. End-of-turn detection combines semantic context with voice activity, not just silence. That keeps an agent from interrupting a caller who paused to read their bill.
The context. Pass your agent’s last question through agent_context and the model resolves short replies (spelled-out emails, account IDs, one-word confirmations) against it. Across 20,000 voice agent audio files, that cut WER by 10.2%, with fabrications down 18.3%.
The rest. Keyterm prompting at the base rate. Live speaker diarization. 32 languages with mid-sentence code-switching. $0.45/hr base, with no concurrency limits.
And if you’d rather not run an orchestrator at all, the Voice Agent API bundles STT, LLM and TTS into one WebSocket at $4.50/hr, with about 1 second end-to-end latency. It runs on the same Universal-3.6 Pro Realtime model. If you want to see that without a framework, read how to build a voice agent without Pipecat or LiveKit.
Find the right tool for your voice strategy
Quick version, if you skipped down:
- LiveKit and Pipecat (open source) when maximum customization matters and you have engineers to run them.
- Vision Agents (open source) when the agent needs to see, not just hear.
- Telnyx for agents that live on phone infrastructure.
- Vapi for developer teams that want managed infrastructure with code-level control.
- Retell for a managed phone agent with strong conversational flow.
- Synthflow for no-code speed.
- Bland when the data can’t leave your infrastructure.
- Voice Agent API if you’d rather not maintain an orchestrator at all.
Now the part these lists usually skip.
The orchestrator is the layer you’ll change. Teams outgrow no-code builders, can’t staff frameworks or switch when pricing shifts. That’s normal. Every tool here is designed to be replaceable, which makes picking one a lower-stakes decision than it feels like in week one.
What you don’t change casually is the speech layer. Your prompts, keyterm lists, confidence thresholds and escalation rules are all fitted to one model’s behavior. Swap the transcriber and you’re re-tuning the whole agent. So pick the orchestrator that fits your team today, and pick the speech model like you’ll live with it. Our take on where voice agent stacks start showing their limits goes deeper on which seams give first.
Plug Universal-3.6 Pro Realtime into any orchestration stack, or skip the wiring with the Voice Agent API at $4.50/hr.
Frequently asked questions
What is an orchestration tool for AI voice agents?
An orchestration tool connects the models in a voice agent (speech-to-text, an LLM and text-to-speech) and manages the real-time flow between them. It handles timing, turn-taking, interruptions and tool calls. Without one, you write that coordination layer yourself, and it’s the hardest part of the build.
What is the best open-source voice agent framework?
Pipecat and LiveKit Agents are the two most widely used open-source voice agent frameworks. Pipecat (BSD-2-Clause) is a transport-agnostic Python pipeline with explicit control over every step. LiveKit Agents (Apache-2.0) pairs an agent framework with WebRTC transport and is the stronger choice for multi-participant or video sessions.
Is Vapi open source?
No. Vapi is a proprietary managed platform, even though it’s API-first and built for developers. If you need open source, Pipecat and LiveKit Agents are the common alternatives, and both support AssemblyAI through official plugins.
What are open-source alternatives to Vapi?
Pipecat, LiveKit Agents and Stream’s Vision Agents are open-source alternatives you can self-host. You get full control over the pipeline and no per-minute platform fee. In exchange, you own infrastructure, telephony integration and monitoring.
Do orchestration tools work with any speech-to-text provider?
Most do. Vapi, LiveKit and Pipecat all offer native AssemblyAI integrations running Universal-3.6 Pro Realtime. In Vapi that’s a transcriber block with “provider”: “assembly-ai” and “speechModel”: “universal-3-6-pro”. In LiveKit and Pipecat it’s a drop-in plugin.
How long does it take to build and deploy a voice agent?
Simple agents can go live in days with a no-code tool, while custom systems on LiveKit or Pipecat take weeks to months including testing. Integration with your telephony and CRM usually dominates the timeline. A bundled API like AssemblyAI’s Voice Agent API shortens it by removing three model integrations from the critical path.
