The OpenAI Realtime API is a single speech-to-speech model: audio in, audio out, one vendor. The main alternative is a chained pipeline built on AssemblyAI’s speech APIs: Universal-3.6 Pro Realtime for speech-to-text, the LLM of your choice, and AssemblyAI’s Text-to-Speech API for the voice. Choose OpenAI Realtime when expressive, emotion-aware output matters most; choose the chained pipeline when your agent has to get entities right, your costs have to be forecastable, and you want to see and fix each layer.
OpenAI’s Realtime API was one of the first products to make voice agents feel accessible. Stream audio in, get audio back: simple idea, big impact. But as teams move from prototype to production, a different set of requirements kicks in: accuracy on real-world entities, cost per hour you can predict, and an agent you can debug at 2am. This is a head-to-head on exactly those five things: accuracy on entities, latency, cost per hour, control and debuggability, and vendor lock-in.
At a glance
| OpenAI Realtime API | Chained pipeline with AssemblyAI | |
|---|---|---|
| Architecture | One speech-to-speech model hears, reasons and speaks | Universal-3.6 Pro Realtime (speech-to-text) → your LLM → AssemblyAI Text-to-Speech API, wired by your orchestrator |
| Real-agent WER (AssemblyAI voice-agent benchmark) | Not published for Realtime; GPT-4o Transcribe 44.58% on async code-switching | 5.19% |
| Entity error rate | GPT-4o Transcribe ~23.3% missed entities in head-to-head English testing | 14.4% (names 10.9%, codes/IDs 10.0%, phone numbers 2.4%) |
| Latency | About 1 second per turn | Final transcript a median 307 ms after the caller stops (Pipecat); full turn depends on your LLM and TTS |
| Cost per hour | Per-token audio in and out ($32/1M in, $64/1M out on gpt-realtime-2.1); about $3–6/hr with good caching, more as context grows (our estimate) | Speech-to-text $0.45/hr, billed per second, plus your LLM’s text tokens and TTS usage |
| Languages | Broad | 32 for speech-to-text, with automatic detection and mid-sentence code-switching |
| Context input | Not a dedicated speech feature | agent_context (−10.2% WER), keyterms prompting included |
| Turn detection | VAD-based options | Semantic context + voice activity |
| Debuggability | No first-class record of what was heard before the model replied | A transcript at every step: what was heard, what the LLM got, what it said |
| LLM choice | OpenAI’s realtime model | Any LLM, including OpenAI’s GPT models through LLM Gateway |
| Lock-in | One vendor across all three layers | Swap any layer independently |
| Emotion from audio | Yes, a real speech-to-speech strength | Not a focus |
| Healthcare | Check vendor terms | BAA available; Medical Mode for clinical terms |
Architecture: speech-to-speech vs. chained
This is the foundational difference, and everything else flows from it.
OpenAI’s Realtime API uses one multimodal model. Audio goes in, audio comes out, and the same model handles speech understanding, reasoning and voice generation. It’s architecturally clean, and if you’re already building on OpenAI, it feels like a natural extension. But that model also handles text, images and video. It wasn’t built specifically for listening.
A chained pipeline splits the job. Universal-3.6 Pro Realtime turns the caller’s audio into text, an LLM decides what to say, and the Text-to-Speech API says it. AssemblyAI supplies both speech ends, built for conversation rather than narration; you own the LLM and the orchestration, whether that’s LiveKit, Pipecat, Vapi or your own loop. AssemblyAI isn’t the agent. It’s the speech layer you build the agent on.
Accuracy on entities: where the gap is real
Voice agents live or die by what they hear. If the speech layer gets an email address wrong, the agent sends a confirmation to the wrong inbox. If it misses a digit in an account number, the agent looks up the wrong record.
Universal-3.6 Pro Realtime posts a 5.19% word error rate on AssemblyAI’s English voice-agent benchmark: 12,460 scripted voice-agent scenarios, not read speech. On entity accuracy, which is what agents act on, it lands a 14.4% entity error rate, with names at 10.9%, codes and IDs at 10.0%, and phone numbers at 2.4%. Across AssemblyAI’s full voice-agent suite it ranked first of 22 systems, with the lowest entity error rate of any of them.
OpenAI doesn’t publish comparable real-agent transcription benchmarks for its Realtime model. Where OpenAI’s transcription is measured directly, the numbers are weak: GPT-4o Transcribe posts a 44.58% word error rate on async code-switching, and in head-to-head English entity testing it scored a 23.3% missed entity rate on names, emails, phone numbers and credit card numbers.
In practice, that’s the difference between an agent that captures “RX-7704132” correctly and one that hears “dash seven seven zero four one three two.” When your agent acts on that transcript, accuracy isn’t a nice-to-have. It’s the whole game.
Where OpenAI wins: its speech-to-speech model reads affect directly from the audio. That’s a real strength of the single-model approach, and a transcription-first pipeline doesn’t match it today. If reacting to a caller’s tone is core to your product, weigh that.
For clinical conversations, Medical Mode (+$0.15/hr) improves accuracy on medication names, procedures and dosages. OpenAI doesn’t have an equivalent specialized mode.
Stream a recording with names, codes and phone numbers to Universal-3.6 Pro Realtime and read exactly what it heard.
Latency: closer than you’d think
The usual argument for speech-to-speech is speed: one model, no hand-offs. In practice, a well-built chained pipeline lands in the same range, about a second per turn, because the stages stream into each other rather than waiting in line.
The speech-to-text leg is the part AssemblyAI controls, and it’s fast. On Pipecat’s open STT benchmark, Universal-3.6 Pro Realtime delivers the final transcript a median 307 ms after the speaker stops, at 0.96% pooled semantic WER. The rest of your budget goes to LLM time-to-first-token and TTS start-up, which you can tune by choosing a faster model or streaming tokens straight into speech.
Turn detection matters as much as raw speed. Universal-3.6 Pro Realtime combines semantic context with voice activity to decide when a caller is done, rather than relying on silence alone. It tells a thoughtful pause from a finished thought, so the agent doesn’t cut people off mid-sentence or sit in dead air. More on that in our guide to turn detection and endpointing.
Cost per hour: per-token vs. per-second
OpenAI’s Realtime API bills per token for both input and output audio. At $32 per 1M audio input tokens and $64 per 1M output tokens on gpt-realtime-2.1, and with the whole conversation re-sent on every response, a 50/50 conversation comes out to roughly $3–6/hr in our estimate when the history is cached. It climbs well past that as conversations get longer and cache hits drop, and it swings with how much the agent talks.
In a chained pipeline, each layer has its own meter, and the speech-to-text meter is simple: Universal-3.6 Pro Realtime is $0.45/hr, billed per second, pay-as-you-go with no minimums. The LLM runs on text tokens, which are far smaller than audio tokens, and the Text-to-Speech API puts the voice on the same AssemblyAI bill. Diarization (+$0.12/hr) and Voice Focus (+$0.10/hr) are optional add-ons; keyterms prompting is included.
Here’s how to model it at scale. At 5,000 agent hours a month, OpenAI Realtime at $3–6/hr is roughly $15,000–$30,000 before cache misses and longer calls push it up. On the chained side, speech-to-text is $2,250. Add your LLM’s text-token cost and your TTS usage at your expected talk ratio, and you have a number you can forecast line by line, because no single line depends on how long the model decides to speak in audio tokens.
Start with up to 333 hours of free streaming speech-to-text, no credit card required, and measure your real cost per call.
Control and debuggability
OpenAI’s Realtime API exposes over 30 event types. That’s fine-grained control, but it’s also a lot of surface to manage, and when the agent answers the wrong question, separating a hearing error from a reasoning error is hard because one model did both.
A chained pipeline gives you a transcript at every step. You can see what Universal-3.6 Pro Realtime heard, what your LLM received, and what it said back. A bad reply becomes a five-minute diagnosis instead of a guessing game.
You also get levers a do-everything model doesn’t expose:
- agent_context. Pass the agent’s last question to the speech model so it hears the reply through the lens of what was just asked. Across 20,000 voice-agent files, that cut WER by 10.2%.
- Keyterms prompting. Give the model the product names, drug names or account formats it should expect.
- Live configuration. Update keyterms, prompt and turn-detection settings on the open stream without reconnecting. Your system prompt and tools live in your own orchestrator code, so you change them whenever you like.
- Your own tool calling. Tools are defined in your LLM of choice with JSON Schema, so they behave the way that LLM documents them.
The streaming API itself is small: one WebSocket, binary audio in, a handful of JSON messages back, and Python and JavaScript SDKs if you want them. Most developers ship a working agent the same day.
Vendor lock-in
With OpenAI Realtime, the listening model, the reasoning model and the voice are one product. If a better LLM ships elsewhere, or a regulator asks where your audio goes, you change all three or none.
A chained pipeline makes each layer replaceable. You can keep OpenAI as the brain, routing GPT models through LLM Gateway, while moving listening to Universal-3.6 Pro Realtime and speaking to the Text-to-Speech API. Next quarter, you can switch the LLM to Claude or Gemini without touching the speech layers. And because Universal-3.6 Pro Realtime drops into LiveKit, Pipecat and Vapi, your orchestrator doesn’t lock you in either.
Migration notes: moving from OpenAI Realtime to a chained pipeline
You don’t have to rip and replace. Most teams move in stages.
Map the concepts first. OpenAI’s session instructions become your LLM’s system prompt. Its input-audio events become binary audio frames streamed to Universal-3.6 Pro Realtime. Its server-side turn detection becomes AssemblyAI’s end-of-turn signal, which your orchestrator uses to call the LLM. Its output-audio events become the audio your TTS step returns. Tool definitions move to your LLM call largely unchanged, since they’re JSON Schema on both sides.
Keep the brain, swap the ears. The lowest-risk first step is to keep an OpenAI model for reasoning and replace only the listening layer. That isolates the accuracy gain so you can measure it.
Run in shadow. Send a slice of live calls through both stacks and compare transcripts, entity capture and tool-call success side by side before shifting traffic.
Use an orchestrator. LiveKit and Pipecat already handle streaming between stages, interruptions and barge-in. Start from the LiveKit guide for Universal-3.6 Pro rather than writing that plumbing yourself.
For a broader view of where else teams go, see our roundup of OpenAI Realtime API alternatives and the best speech-to-speech voice agent APIs.
When to choose each
Choose OpenAI’s Realtime API when: emotion detection from audio is core to your product; you want one vendor and one model for all three layers; the conversation is open-ended rather than transactional; and per-call cost isn’t your main constraint.
Choose a chained pipeline with AssemblyAI when: you need the highest accuracy on names, numbers and addresses; you want to forecast cost per hour line by line; you need to see what the agent heard on every turn; you want to pick, and later change, your LLM; or you’re building healthcare workflows that need Medical Mode and a BAA.
For most production voice agents, including customer support, phone agents, clinical workflows and coaching platforms, the chained pipeline wins on the things that break in production. Speech-to-speech wins on feel. Pick the failure mode you can live with.
Pair Universal-3.6 Pro Realtime with your LLM and AssemblyAI’s Text-to-Speech API for speech in and speech out from one vendor.
Frequently asked questions
OpenAI Realtime API vs. a chained pipeline: which is better?
For most production voice agents, a chained pipeline is the stronger choice. Built on Universal-3.6 Pro Realtime, it posts a 14.4% entity error rate on AssemblyAI’s voice-agent benchmark, gives you a transcript at every step, and lets you choose your LLM. OpenAI’s Realtime API is a good fit when emotion detection from audio or a single-vendor setup matters most.
Does AssemblyAI offer a speech-to-speech model like OpenAI Realtime?
No. AssemblyAI supplies the speech layers of a chained pipeline: Universal-3.6 Pro Realtime for streaming speech-to-text and the Text-to-Speech API for voice output. You bring the LLM, or use LLM Gateway, and connect the pieces with LiveKit, Pipecat, Vapi or your own code.
How does the cost per hour compare?
OpenAI’s Realtime API bills per audio token ($32/1M in, $64/1M out on gpt-realtime-2.1) and re-bills conversation context on every response, which works out to about $3–6/hr with good caching in our estimate, and more as conversations get longer. In a chained pipeline, Universal-3.6 Pro Realtime costs $0.45/hr billed per second, and you add your LLM’s text tokens and TTS usage. Each line is metered separately, so the total is easier to forecast.
Which is more accurate for names, numbers and entities?
Universal-3.6 Pro Realtime posts 5.19% WER and a 14.4% entity error rate on AssemblyAI’s English voice-agent benchmark, with phone numbers at 2.4%. OpenAI doesn’t publish comparable real-agent numbers for Realtime; its GPT-4o Transcribe scored 44.58% WER on async code-switching and a 23.3% missed entity rate in head-to-head English testing.
Can I keep using OpenAI models in a chained pipeline?
Yes. You can keep a GPT model as the reasoning layer, called directly or through LLM Gateway, and replace only the listening and speaking layers. That’s also the lowest-risk way to migrate, because it isolates the accuracy change.
Which has lower latency?
Both land around a second per turn when built well. In a chained pipeline, Universal-3.6 Pro Realtime delivers the final transcript a median 307 ms after the speaker stops on Pipecat’s open benchmark, and turn detection combines semantic context with voice activity, so the agent neither cuts callers off nor stalls in dead air.
