The rule of thumb in voice AI is that your speech-to-text layer should return a complete transcript in roughly the length of a natural conversational pause — about 300 milliseconds. That number is not arbitrary and it is not a marketing target; it is a fact about how people take turns. Go past it and the person on the other end starts to feel the machine. But 300 ms is a budget for one layer, not for a whole interaction, and conflating the two is the most common mistake in vendor comparisons. This post separates them: what each layer costs you in milliseconds, which AssemblyAI low-latency surface to reach for, and the published numbers for each.
The short version, with figures explained in full below: Sync API ~134 ms p50 for a short clip, Dictation API 0.36 s to finished, cleaned-up text, Universal-3.5 Pro Realtime streaming transcripts continuously as audio arrives, and a full agent loop — Universal-3.5 Pro Realtime plus an LLM plus AssemblyAI’s Text-to-Speech API — targeting about 1 second for a complete conversational turn.
Why 300 milliseconds is the number
Human conversation runs on a tighter clock than most people realize. The gap between one speaker finishing and the next starting sits, across languages and cultures, in the low hundreds of milliseconds. Listeners do not wait for a sentence to end before beginning to plan a reply — they predict the ending and start assembling their response while the other person is still talking. That is why a natural conversation can have turn gaps shorter than the time it takes to produce a single word.
The practical consequence is that latency in voice AI is not experienced linearly. Under about 300 ms, a delay reads as ordinary conversational rhythm and nobody notices it. Somewhere past 500 ms it reads as hesitation — the listener starts to wonder whether they were heard. Past a second, people repeat themselves, which produces overlapping speech, which produces exactly the kind of audio that degrades transcription accuracy and makes the problem worse. Latency failures compound.
So the design goal is not “as fast as possible.” It is: stay inside the window where the delay is invisible, and spend the accuracy budget everywhere else.
Where the milliseconds actually go
A voice interaction is a chain, and every link spends time:
- Capture and network transit. Getting audio off the device and to the API. Typically tens of milliseconds, more on a poor mobile connection, and largely out of your control — though picking a region close to your users helps more than most optimizations you can make in code.
- Endpointing. Deciding the speaker has finished. This is frequently the largest and least examined cost in the chain. A naive implementation waits out a fixed silence window — 700 ms, a full second — before it will commit to a turn ending, and that wait lands entirely in the user’s experience.
- Transcription. Turning audio into a finalized transcript. This is the layer the 300 ms rule is about.
- Reasoning. Whatever your application does with the text — a model call, a database lookup, a tool invocation.
- Synthesis. Turning the response back into audio, and the time to the first audible byte.
Most teams optimize transcription because it is the layer with a published number, then wonder why their agent still feels slow. The usual culprit is endpointing. Universal-3.5 Pro Realtime does not endpoint on silence alone: when a speaker pauses, the model evaluates what has been said so far and judges whether it reads as a finished thought or a mid-thought pause. That is a meaningfully different approach from a fixed timer, because a speaker who trails off mid-sentence and a speaker who has finished produce text that reads very differently even when the silence is identical.
Run your own audio through the current models and listen to where the milliseconds actually go. No setup, no integration work.
The low-latency surfaces, compared
AssemblyAI publishes three low-latency paths, plus the full agent loop you compose from them, and they are genuinely different shapes rather than tiers of the same one. Picking the wrong one is the most common source of “why is this slow” tickets.
| Surface | Published latency | Shape | Price | Reach for it when |
|---|---|---|---|---|
| Sync API | ~134 ms p50 on a 2-second clip, request to finished transcript (versus 5–6 s on async submit-and-poll) | Single HTTP request, single response | $0.45/hr | You have a short clip and you want the transcript itself, right now |
| Dictation API | 0.36 s to finished text; typical short clips come back in under one second | Single HTTP request; transcription plus an LLM cleanup pass | $0.62/hr | The text goes straight into whatever your user is writing |
| Universal-3.5 Pro Realtime | Transcripts emitted continuously as audio arrives, finalized at end of turn | Persistent WebSocket, continuous audio | $0.45/hr base | Audio is open-ended and you need transcripts as it arrives |
| Full agent loop (Realtime + LLM + Text-to-Speech API) | Target ~1 second end to end, for a complete conversational turn | Streaming WebSocket in, LLM call, text-to-speech out; LiveKit, Pipecat, or your own loop carries the turn | $0.45/hr base for speech-to-text, plus LLM and text-to-speech usage | You are building something that talks back |
The Sync API is the dictation path, and it is the one most teams should have started with
If your audio is short and bounded — a push-to-talk clip, a voice command, an IVR response, a voicemail — you do not need a WebSocket. You need one request and one response. That is the Sync API: POST https://sync.assemblyai.com/transcribe, clips from 80 ms to 2 minutes, up to 40 MB, WAV or raw PCM, ~134 ms p50 from request to finished transcript on a 2-second clip.
The comparison that matters is not against streaming, it is against the async submit-and-poll pattern that a lot of dictation features were built on by default: submit a job, poll for completion, render the result. That path takes 5 to 6 seconds for the same clip. For a dictation box or a push-to-talk button, that is the difference between a feature people use and a feature people work around.
Sync runs Universal-3.5 Pro with the required header X-AAI-Model: universal-3-5-pro, covers 19 languages, defaults to English, and includes Keyterms Prompting and Conversation Context at no extra charge. Authenticate with your key in the Authorization header, with no Bearer prefix. Be clear on what it does not do: no PII redaction, no speaker diarization, no Speech Understanding, and no Medical Mode. If you need any of those, the pre-recorded or streaming path is where they live. The getting-started guide has a working request, and the launch post has the full numbers.
The Dictation API buys you a hop you would otherwise pay for yourself
The Dictation API is a slightly different bet. It is transcription plus an LLM cleanup pass in a single low-latency call, and it clocks 0.36 s to finished text — the transcription plus the rewrite. Typical short clips come back in under one second.
Why that extra time is worth paying for: transcription models are verbatim by design, and what someone said is not what they want to send. Given “um so can we uh move the the meeting to thursday i think friday works better actually”, the Dictation API returns “Can we move the meeting to Friday? That works better.” Self-corrections resolved to what the speaker landed on, filler gone, tone intact.
The alternative is doing that cleanup yourself, which means a second network hop to an LLM after the transcript comes back — and that hop costs you more than the headroom you are trying to protect, plus a second line on your bill and a second failure mode to handle. Dictation is $0.62/hr flat; from our pricing page, verbatim: “Every feature is included in the $0.62/hr rate. One line on your bill covers the whole request — there is no second rate to model and no token math to do.”
The clean way to choose between them, from the launch post: “Reach for Sync when you want the transcript itself, and Dictation when the text is going straight into whatever your user is writing.” Both run on Universal-3.5 Pro; Sync covers 19 languages and Dictation accepts 32 language codes. The Dictation API launch post covers the rest, and the dictation documentation has the configuration fields.
Universal-3.5 Pro Realtime is the streaming path, and it is where the 300 ms rule lives
When audio is open-ended — a call in progress, a meeting, a live captioning session — you want a persistent connection and transcripts as the audio arrives. That is Universal-3.5 Pro Realtime, over wss://streaming.assemblyai.com/v3/ws, at $0.45/hr base.
The controls that matter for latency work on this API:
- Three modes: min_latency, balanced (the default), and max_accuracy. The mode sets the turn-detection defaults, so change it before you touch anything else. Most production systems should stay on balanced and only reach for min_latency after they have measured that the extra accuracy is not being used.
- Content-based end-of-turn detection. When a speaker pauses, the model evaluates what has been said so far rather than counting silence alone, which is what stops it splitting a phone number across two turns.
- Turn detection parameters. min_turn_silence and max_turn_silence have no fixed default — they follow the mode preset: 128 / 1280 ms on balanced, 128 / 640 on min_latency, 512 / 2560 on max_accuracy. vad_threshold defaults to 0.2. Enabling speaker_labels swaps in diarization-tuned values of 640 / 768 and disables continuous partials, so re-measure after you turn it on. Raise min_turn_silence mid-stream when you know an entity is coming, then re-send mode to restore the preset — the defaults are mode-derived, so hardcoding numbers on the way out leaves the session off-preset.
- agent_context to pass your agent’s most recent spoken reply, so short and mumbled replies resolve against what was asked. Context Carryover, a rolling memory of the conversation, is on by default.
- Voice Focus for microphone conditioning — near-field for headsets and phones, far-field for rooms, kiosks and drive-thrus, with voice_focus_threshold from 0.0 to 1.0 defaulting to 0.7 on this surface.
- language_codes as a list to bias toward specific languages, or omit it for automatic detection and code-switching across the 18 supported languages.
- Diarization with revision, up to 10 speakers, with corrections landing within about half a second of stream end. If you need speaker labels in a live transcript, streaming speaker diarization covers the tradeoffs.
Note the parameter name: streaming takes speech_model, singular. The pre-recorded API takes speech_models, plural, as a list. It is a small thing that produces a confusing error.
If you are still on Universal-Streaming, that was the prior default and it is no longer the recommendation for new builds. Migrate to Universal-3.5 Pro Realtime.
Where the full agent loop fits, and why its number is bigger
The 300 ms rule and the ~1 second figure for a full agent turn are not in tension, because they measure different spans. This is worth being precise about, because vendor comparisons in this category routinely put the two next to each other as if they were the same measurement.
The 300 ms budget is a speech-to-text measurement: from the moment the speaker stops to the moment a complete, finalized transcript exists. It is one layer.
About 1 second is an end-to-end conversational measurement: from the moment the caller stops speaking to the moment the agent starts speaking. That span contains the transcription, plus the reasoning step, plus speech synthesis, plus the network transit for all of it.
A vendor quoting 200 ms and a vendor quoting 1 second may be describing systems with identical felt latency, or wildly different ones. The only way to know is to ask which span they are measuring. When you benchmark, measure the end-to-end span yourself, because that is the only one your users experience.
A full agent loop is built around that end-to-end budget. Three models run inside it: Universal-3.5 Pro Realtime for recognition, an LLM through LLM Gateway for reasoning, and AssemblyAI’s Text-to-Speech API for the response, with LiveKit, Pipecat, or your own loop carrying the turn. About one second, end to end, is the budget to hold.
The architectural argument for consolidating vendors is a latency argument. When you assemble an agent from three vendors, you pay network transit between each pair, you pay serialization at each boundary, and you own the reconnection logic for three connections that can each fail independently. Taking both speech ends from one vendor does not remove the hops, but it puts them on one status page and one set of logs when the tail moves. Those costs do not show up in anyone’s published per-layer number. They show up in your p99.
Two things move end-to-end latency once the speech layer is tuned. First, stream every stage: hand the finalized turn to the LLM the moment it lands, and let text-to-speech start synthesizing the first sentence while the LLM is still writing the second — LiveKit and Pipecat both do this by default. Second, leave the turn-detection parameters on their mode preset unless you have measured a reason to move them. Hardcoded silence windows are how agents cut a caller off in the middle of a phone number, and agent_context plus better tool descriptions, not VAD knobs, are the way to improve turn-taking.
Pricing follows a simple shape too: Universal-3.5 Pro Realtime is $0.45/hr base, billed per second with no concurrency fees, and text-to-speech lands on the same invoice. No minimums, no commitment. If you are coming from a stack where you model three vendors’ bills to answer “what does a call cost,” the speech layer is now one line.
The seconds at the end of every call that you are probably paying for
One detail that belongs in a latency-and-cost post because almost nobody catches it on the first build: streaming bills on session duration, not on speech. A WebSocket left open after the caller hangs up — while your orchestrator tears down, while a reconnect timer runs — is billable silence.
Send the Terminate message (disconnect with terminate set in the SDK) the moment a call is intentionally over, and let your orchestrator’s reconnect logic handle unintentional drops. Terminate on intentional hangups; reconnect on everything else.
On accuracy, Universal-3.5 Pro Realtime posts a 6.99% word error rate and a 15.31% entity error rate on the Pipecat open STT benchmark, run against real agent conversations — against 15.58% / 50.50% for Deepgram Flux, 9.76% / 39.70% for ElevenLabs Scribe v2, and 9.04% / 21.51% for Google Chirp3. Methodology is at assemblyai.com/benchmarks, and the Universal-3.5 Pro Realtime launch post covers the design decisions.
The free tier is 185 hours of pre-recorded transcription plus 333 hours of streaming — enough to measure end-to-end latency on your own traffic before you commit to an architecture.
Where the LLM layer costs you time, and how to get it back
If you are doing anything with the transcript beyond displaying it — cleanup, summarization, formatting, routing — the model call is now part of your latency budget, and it is usually the largest single item in it.
The LLM Gateway gives you 37 models behind one endpoint, at llm-gateway.assemblyai.com/v1/chat/completions in the US and llm-gateway.eu.assemblyai.com/v1/chat/completions in the EU. For latency work the relevant pieces are automatic fallback to up to two backup models with a 500 ms retry default, per-fallback overrides for prompt, temperature and max tokens, prompt caching for Anthropic and OpenAI models, and multi-region failover across US and EU. A slow provider stops being your outage.
For the specific case of fast rewrite work — dictation cleanup, transcript rewriting, live formatting, turn summarization — Qwen3.5 4B Fast is self-hosted by AssemblyAI on our own GPUs with a latency-optimized 32k context window. It averages 612 ms on voice rewrite tasks, 1.9× faster than GPT-4.1 and 94% cheaper per hour of audio, at $0.10 per million prompt tokens and $0.50 per million completion tokens. The Qwen3.5 4B post has the benchmark detail, and reintroducing LLM Gateway covers the rest of the platform.
A related piece of architectural advice, and one that saves more latency than any model choice: do not make a voice agent call a tool to extract, summarize or classify what was just said. That adds a model round trip inside the conversational loop. Collect the transcript events and run a single LLM Gateway call against the finished transcript when you actually need the structured output. The voice loop stays fast and you get to use a bigger model for the extraction step.
If your application only needs cleanup on short dictated audio, skip the second hop entirely and use the Dictation API, which does the transcription and the rewrite inside one 0.36 s call.
A practical checklist for cutting latency
- Measure end to end, not per layer. Instrument from “user stopped speaking” to “system started responding.” Per-layer numbers hide the transit and serialization costs between layers, and those are frequently where the time is.
- Look at endpointing before you look at the model. If you are waiting out a fixed silence window, you are probably spending more there than in transcription. On Universal-3.5 Pro Realtime, the fastest fix is usually to stop hardcoding the silence windows and let the mode preset do its job.
- Match the API shape to the audio shape. Short bounded clips do not need a WebSocket. Open-ended audio does. Using streaming for push-to-talk adds connection setup to every interaction for nothing.
- Eliminate hops, not milliseconds. Removing one network round trip usually beats every micro-optimization you can make inside a layer.
- Deploy near your users. EU traffic should hit api.eu.assemblyai.com and streaming.eu.assemblyai.com — same price, and the data stays in the EU.
- Do not stack noise suppression. If you are capturing in a browser, leave noiseSuppression off and let server-side Voice Focus do the work. Two independent passes eat real speech and cost you accuracy, which costs you clarification turns, which are the slowest thing in any conversation.
- Flush your playback buffer on barge-in. Otherwise the user hears another second of stale speech after they interrupt, which reads as lag even though no API was slow.
- Optimize p99, not p50. Your median user has a fine experience on almost any architecture. The tail is where people hang up.
Self-hosted deployment, EU data residency, or a Business Associate Addendum for PHI — talk it through with someone who has shipped low-latency voice systems under those constraints.
Frequently asked questions
What is a good latency target for voice AI?
Roughly 300 milliseconds for the speech-to-text layer, and about one second end to end for a full conversational turn. The 300 ms figure approximates the natural gap between turns in human conversation, which is why delays under it go unnoticed. Past roughly 500 ms people begin to hear hesitation, and past a second they start repeating themselves.
What is the difference between the Sync API and the Dictation API?
The Sync API returns a verbatim transcript in a single request at ~134 ms p50; the Dictation API takes the same utterance and returns what the speaker meant to send, with self-corrections resolved and filler gone, in 0.36 s. Both run on Universal-3.5 Pro. Reach for Sync when you want the transcript itself, and Dictation when the text is going straight into whatever your user is writing.
How fast is the Sync API compared to submitting an async job?
About 134 ms at p50 for a 2-second clip, versus 5 to 6 seconds for the same clip through async submit-and-poll. That gap is the entire reason the Sync API exists. It handles clips from 80 ms to 2 minutes, up to 40 MB, in WAV or raw PCM, across 19 languages, at $0.45/hr.
Why is a full voice agent turn quoted at ~1 second when a transcription layer is measured in hundreds of milliseconds?
Because they measure different spans. A transcription figure covers the time from the speaker stopping to a finalized transcript existing — one layer. The ~1 second figure is end to end — caller stops speaking to agent starts speaking — and includes transcription, reasoning, speech synthesis and all the network transit between them. When comparing vendors, always ask which span the number describes.
Which streaming model should I use for a new build?
Universal-3.5 Pro Realtime, at $0.45/hr base over wss://streaming.assemblyai.com/v3/ws. It covers 18 languages with automatic code-switching, decides end-of-turn from what has been said rather than from silence alone, and supports contextual prompting and Voice Focus. Universal-Streaming was the prior default and existing integrations should migrate off it.
Does adding an LLM step break the latency budget?
It does not have to, but it is usually the largest single item in the budget once it is there. Through the LLM Gateway, Qwen3.5 4B averages 612 ms on voice rewrite tasks — 1.9× faster than GPT-4.1 and 94% cheaper per hour of audio — and is self-hosted on AssemblyAI GPUs with a latency-optimized 32k context. For short dictated audio, the Dictation API does transcription and cleanup in one 0.36 s call and removes the hop entirely.
