Your agent can take no for an answer

Short answers, heard right

The user utterance that decides a call is usually one word: “no”, “nah”, “yeah”. When the STT drops or flips it, the agent books the wrong appointment, and the failure shows up as a refund or an escalation rather than a transcript error anyone reviews. Most teams cover this with a confirmation loop, “just to confirm, did you say yes?”, which adds a turn to every decision.

Universal-3.6 Pro gets those turns wrong 1.45% of the time, down from 2.65% on 3.5 Pro. That is 45% fewer wrong confirmations overall, and 52% fewer in heavy background noise. The metric scores by meaning: “yeah” for “yes” is not an error; a different answer or a dropped answer is.

The same failure shows up on the values an agent acts on. Entity error counts how often a name, number or identifier comes back wrong, and it is where the distance between streaming models is widest.

Half as many wrong confirmations is the difference between a confirmation loop on every decision and a confirmation loop on the ones that carry money.

The gain tracks how hard the audio is. Against 3.5 Pro, the short-response error rate falls 38% on clean recordings, 42% on light background noise, and 52% on the heavy-noise third.

Background voices stay out of the transcript

Background speech leakage counts words transcribed from a television, a coworker or the next desk and handed to your agent as if the caller said them. It falls from 3.53 to 2.52 words per 1,000 overall, and from 6.82 to 4.69 on the heavy-noise third of the benchmark.

Noise is where streaming models separate. Word error rate across the three recorded noise environments:

Emitting words that were never spoken is the other half of this. On 2,950 clips of hold music, line noise and silence with no speech in them at all, Universal-3.6 Pro returns text 0.27% of the time. Deepgram Flux EN returns text 38.75% of the time.

Voice focus changed underneath. It is an optional stage that suppresses speakers farther from the microphone than the caller. Most teams already run a noise suppressor or speaker-isolation stage in front of the STT, tuned for what sounds clean to a person on a call. That is a different objective from what a speech model needs, and a filter that scrubs audio for human ears can take the cues the model was using with it.

Voice focus is not a stage in front of the model: the filter and the STT model ship together and are measured together, so what the filter removes shows up as word error rate rather than as a cleaner-sounding recording. It is off by default, and it is for other people’s voices. Where the challenge is background noise rather than other talkers, 3.6 Pro handles it on its own.

It stays in the caller’s language, across 32 of them

A caller drops a Spanish phrase into an English sentence. Most streaming models hear it as English-sounding nonsense, so the turn comes back with the wrong words in the right alphabet and nothing downstream catches it: intent detection, entity capture and the reply all fail on a transcript that looks well-formed. Teams work around it by pinning the language, which means a language menu before the conversation starts.

Universal-3.6 Pro Realtime has the lowest word error rate of any streaming system tested on speech that switches mid-sentence: 7.20% across twenty public sets covering five English-to-European language pairs, and it leads in all five individually.

To see our performance against all competitors, check out our benchmarks page.

Fourteen languages are new:

Korean·Catalan·Galician·Russian·Romanian·Persian·Estonian·Afrikaans·Marathi·Zulu·Urdu·Xhosa·Cantonese·Norwegian Nynorsk

All 32 run through the same endpoint with automatic language detection; pass language_codes if your app supports a fixed set.

Entity-aware endpointing

Every agent team owns a silence-window setting, and most have widened it once, after a caller got cut off mid-phone-number, and left it there. That is a latency cost on every turn to protect one turn in ten.

3.6 Pro’s adaptive endpointer holds a turn open only when the transcript looks like an entity in progress: a number, a code, an email, an address. End-of-turn precision rises from 74.1% to 77.9% at held recall (92.4% to 92.8%). Median endpoint latency is unchanged at 537 ms. Both models run the identical endpointer, so the difference is the recognition model producing fewer premature end-of-turn decisions, which means a silence window you widened on 3.5 Pro can move back toward the default.

Every turn also carries end_of_turn_confidence, a 0-to-1 signal. Set a threshold to start your LLM call early and discard the result if the caller keeps talking.

Where it stands in the field

Our own numbers are above. Pipecat and Coval independently benchmarked 3.6 Pro alongside other providers. Here is what they found.

Pipecat

Pipecat publishes an open benchmark of streaming speech-to-text services, measuring time to first speech against semantic word error rate and plotting the Pareto frontier: the set of models where nothing else is both faster and more accurate. Every model on the chart runs the same audio through the same harness, and the code and results are public.

On that run, Universal-3.6 Pro Realtime pushes the frontier: 0.96% pooled semantic word error rate at 307 ms median time to first speech, with 87.0% of transcripts exactly right. One model on the chart transcribes more accurately, Meta’s, and it sits further out on latency. That trade is the whole point of a frontier, and it is the one an agent has to make on every turn.

Coval

Coval runs an independent, continuously updated leaderboard of streaming speech models on real voice-agent audio, rerunning throughout the day rather than publishing a single dated result. Its 7-day window is the one to watch: long enough to smooth out a bad afternoon, short enough to reflect what is actually deployed right now.

The two boards disagree on the absolute number, and they are supposed to: they run different audio through different scoring. What travels between them is the placement, which is the part worth acting on.

The research post covers the training data, the evaluation sets and the methodology behind every number above. The benchmarks page updates as new runs land, so it is the place to check how 3.6 Pro stacks up on the audio profile your agent actually handles.

See the benchmarks Compare AssemblyAI models against other providers on the audio your agent actually handles.

Pricing

$0.45/hr of audio, unchanged from 3.5 Pro. Volume discounts apply at any tier.

Get started

Universal-3.6 Pro Realtime is available now with your existing API key. Swap the model name, run your own calls through it, and compare the short answers, the background, and the turn boundaries against what you see on 3.5 Pro today.

Frequently asked questions

API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
# A live AAC (ADTS) internet radio stream, so no microphone is needed.
STREAM_URL = "https://14123.live.streamtheworld.com/WBBRAMAAC.aac"
RUN_SECONDS = 25
# AAC is self-describing (ADTS headers carry the sample rate)
CONNECTION_PARAMS = {"speech_model": "universal-3-6-pro", "encoding": "aac"}
API_ENDPOINT = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(CONNECTION_PARAMS)}"