How real-time agent assist is changing conversation intelligence

Real-time agent assist uses live speech-to-text to transcribe a customer call as it happens, then surfaces answers, compliance alerts, and next-best actions to the human agent mid-conversation. It differs from post-call conversation intelligence in one respect that governs the entire architecture: the guidance has to arrive while the agent can still act on it, which puts the whole pipeline on a sub-second budget.

That single constraint sets everything below — latency, entity-level accuracy, and live speaker attribution.

What it changes in a contact center

Agent assist moves four things: how fast calls resolve, how often they resolve on the first attempt, how much recall pressure sits on the agent, and how consistent service is across a team where tenure ranges from three weeks to nine years.

But a badly implemented system makes all four worse, not better.

Every failure mode in the right column traces back to one of two root causes: the transcript was wrong, or the timing was wrong. That's the whole design problem.

It's also why the accuracy of the layer underneath matters more than the sophistication of the layer on top. Knowledge retrieval, sentiment scoring, compliance alerting, next-best-action — all of it is reasoning over a transcript. If the transcript says the wrong product name, the compliance rule fires on the wrong phrase and the agent gets coached toward the wrong answer, confidently and in real time. Agents route around tools they don't trust, and they do it permanently.

Calabrio builds workforce and conversation intelligence software for contact centers on AssemblyAI, and their product lead is direct about which properties matter at this layer:

"The transcription accuracy, reliability, and speed of AssemblyAI's API have greatly enhanced our operations, reinforcing our trust in their technology and solidifying our partnership."

Raj Shankar, SVP Product, Calabrio

Accuracy, reliability, and speed, in that order. Not features. The features are the easy part.

The three technical requirements

The transcript has to be right on the words that matter, it has to arrive inside the natural pause in the conversation, and the guidance has to fire on completed turns rather than partial ones. Everything else is product design. These three are infrastructure, and you can't fix them later.

1. Accuracy on entities, not headline WER

Word error rate is close to useless as a spec here. The words that break an assist workflow aren't distributed evenly through a sentence — they're concentrated in account numbers, product names, addresses, and proper nouns. A model can score well on aggregate WER and still fail every call where it matters.

‍

Entity error rate is the metric to write into your requirements. On the Pipecat open speech-to-text benchmark, built from real agent conversations rather than read speech (lower is better):

The two rows tell different stories, and the second one is the story that matters. 15.31% against 50.50% is the difference between an assist tool that retrieves the right account and one that doesn't.

Universal-3.5 Pro Realtime brings agent context, keyterm prompting, native code-switching, and diarization with revision to real-time use cases across 18 languages. For queues running the long tail beyond those 18, Universal-2 covers 99+ — worth raising early in procurement, because it changes which model handles which queue.

Two levers move accuracy more than anything you can do at the application layer:

• Agent context. Pass the agent's own last utterance to the model. Across a benchmark of 20,000 voice agent audio files, this cut word error rate by 10.2% — fabrications down 18.3%, hallucinations down 17.2%, name entities down 9.4%. Included in the base price.
• Mid-entity endpointing. On by default. The model holds the turn open through numbers and proper nouns, so a customer pausing halfway through an account number isn't split into two turns and re-read as two values.

Speaker attribution matters too, though not for the reason people expect. Most contact center calls are two-channel, so agent-versus-customer is easy. The hard cases are three-way calls, supervisor barge-ins, and conference bridges — and there, speaker diarization with revision labels speakers live, then re-clusters and sends one correction within about half a second of stream end, for up to 10 speakers. One visible revision beats labels flickering all call.

2. Guidance that lands inside the pause

Fast enough that the suggestion arrives before the agent has already answered. A late suggestion is worse than none, because now the agent has to decide whether to contradict themselves.

In practice that puts the transcript on a few-hundred-millisecond budget. Universal-3.5 Pro Realtime's end-of-turn detection resolves a turn in roughly 300ms, reading tonality, pacing, and rhythm rather than a silence threshold.

Budget the rest of the pipeline against what's left. Turn resolution is the first hop — if knowledge retrieval takes another 400ms on top, you're outside the pause on a fast conversation and inside it on a slow one. Measure your own retrieval step before blaming the speech layer.

3. Gate on completed turns, not partials

This sounds obvious and is the single most common implementation failure. Partial transcripts update several times per second. An assist engine wired to partials re-runs retrieval on every update, surfaces three different articles in two seconds, and flashes prompts while the customer is mid-sentence. The agent's experience is a screen that won't hold still.

The fix is to gate on turn detection that's actually reliable — VAD plus punctuation plus a dedicated turn-decision model, not a silence timer. A silence timer treats a customer thinking mid-sentence as a completed turn, which is precisely when your assist tool interrupts them.

What it costs

No concurrency limits and no rate limits on any of it, which matters more than the per-hour number if your volume is spiky. EU data residency is available at the same price via the EU endpoints.

For the reasoning layer on top — retrieval, sentiment, compliance classification, next-best-action — LLM Gateway gives you one API across OpenAI, Anthropic, and Google models instead of a second vendor relationship. A small fast model is free on the free tier, so the whole architecture is cost-free to prototype before anyone signs anything.

Build vs buy

Worth being clear about where AssemblyAI sits: underneath. Platforms like Genesys and Five9 own telephony and routing; AssemblyAI is the speech infrastructure that platform vendors and in-house teams build the assist experience on. The relevant question isn't AssemblyAI versus a suite — it's whether you want to own the transcript or rent it.

The honest version: if your assist logic is standard, buy. If what differentiates your contact center is what you surface to agents rather than that you surface anything at all, own the transcript. There's a hybrid dual-channel streaming reference architecture on GitHub under AssemblyAI-Solutions if you want to see the shape of the build first, and real-time conversation intelligence covers the broader post-call-to-live shift this sits inside.

Frequently asked questions

What is real-time agent assist?

Real-time agent assist uses live speech-to-text to transcribe a customer call as it happens, then surfaces answers, compliance alerts, and next-best actions to the human agent mid-conversation. Guidance has to reach the agent while they can still act on it, which puts the entire pipeline on a sub-second budget.

How fast does speech processing need to be for agent assist?

Guidance has to land inside the natural pause in a conversation, which means the transcript must be available in a few hundred milliseconds. Universal-3.5 Pro Realtime's end-of-turn detection reads tonality, pacing, and rhythm rather than raw silence and resolves a turn in roughly 300ms.

What happens when agent assist mishears customer speech?

A single misheard account number or product name propagates into every downstream step: the knowledge lookup retrieves the wrong article, the compliance check fires on the wrong phrase, and the agent is coached toward the wrong answer. This is why entity accuracy matters more than headline WER — Universal-3.5 Pro Realtime records a 15.31% entity error rate on the Pipecat benchmark against Deepgram Flux's 50.50%.

How much does real-time agent assist cost to build?

The speech layer is the predictable part: $0.45/hr base for Universal-3.5 Pro Realtime, billed per second, with diarization-with-revision at +$0.12/hr and voice isolation at +$0.10/hr. Agent context, rolling conversation memory, and keyterm prompting are included, with no concurrency or rate limits.

What's the difference between agent assist and a voice agent?

Agent assist keeps a human on the call and feeds them guidance; a voice agent replaces the human and speaks to the customer directly. They share a speech-to-text foundation but diverge on output — assist writes to a screen, an agent writes to a text-to-speech model. AssemblyAI's Voice Agent API bundles the full agent path into one WebSocket at a flat $4.50/hr.

How do you keep agent assist from interrupting the conversation?

Suppress guidance until the model signals a completed turn rather than firing on every partial transcript. Universal-3.5 Pro Realtime detects end-of-turn from tonality, pacing, and rhythm instead of silence thresholds, so it doesn't treat a mid-sentence pause as a handoff point.