How to build real-time agent assist on streaming speech-to-text

Real-time agent assist is one of those features that sounds like magic and turns out to be plumbing. The agent is on a call, and as the customer talks, the screen quietly surfaces the right answer, the next best action, a compliance reminder, a nudge on tone. Done well, it makes a new rep sound like a veteran. Done badly — a beat too slow, or coaching off a garbled transcript — it's noise the agent learns to ignore.

If you're evaluating agent assist, you'll find plenty of finished suites that do this. But if you're building your own — because you have a specific workflow, a Chrome extension over a dialer, or a CX product that needs assist as a feature — the interesting question isn't which suite to buy. It's how to build the real-time layer yourself, and get the latency and accuracy right so the coaching actually lands.

This guide walks through that build. (New to the category? Start with what real-time agent assist is. Comparing off-the-shelf tools instead of building? We keep a running roundup of the best agent assist software.)

What real-time agent assist actually is

Agent assist is the live layer that helps a human agent mid-conversation: real-time transcription, then analysis that fires while the call is still happening — surfacing knowledge-base answers, next-best-action prompts, objection handling, sentiment shifts, and compliance cues.

The key word is live. Post-call analytics is a solved, forgiving problem — you have all the time in the world after the call ends. Real-time assist is unforgiving: the transcript has to be accurate and fast, because a suggestion that arrives after the moment has passed is worse than useless. Everything in the build serves those two constraints.

Build vs. buy

Closed suites like Cresta, Level AI, and Genesys Cloud give you agent assist as a finished product. If you want a turnkey CCaaS experience and your workflow fits theirs, that's a reasonable path.

You build it yourself when you need control the suite won't give you: a specific UI (say, a browser extension that sits on top of an existing web dialer), a custom analysis model, tight integration with your own product, or simply better unit economics at scale. The tradeoff is that you own the real-time layer — but that layer is smaller than it looks. It's streaming transcription, speaker separation, and per-turn analysis. The rest is your product.

That's exactly how teams are building it today. One customer built an agent-assist Chrome extension that sits on top of web dialers, captures the conversation, and coaches agents live — and went from first test to roughly 2,500 hours a day of production traffic within a week. Another runs streaming transcription plus per-turn LLM calls for real-time call coaching. The build is very achievable.

Architecture for real-time assist

The trickiest architectural decision comes first: how do you capture and separate the two sides of the call?

A contact-center call has two audio sources — the agent's microphone and the customer coming through the dialer. You need to know who said what, in real time, to coach correctly. Two approaches work:

One streaming connection per channel. Open a separate streaming connection for the agent's audio and another for the customer's. Each connection handles one speaker, so speaker identity is deterministic — you always know which side you're transcribing. This is the cleanest option when you can access both audio channels separately (common in a Chrome extension capturing mic and dialer output).

A single mono stream with streaming diarization. If you only have one mixed audio stream, send it through a single connection and let streaming diarization label the speakers. Simpler to wire up; speaker labels are inferred rather than guaranteed by the channel.

One thing to know up front: multichannel audio in a single connection isn't supported — if you need deterministic per-speaker transcripts, use one connection per channel.

Step 1: Real-time transcription

Open a WebSocket to the streaming endpoint and stream audio in. Use Universal-3.5 Pro Realtime — the flagship streaming model — and pick a mode that fits contact-center audio.

import os
import requests

API_KEY = os.environ["ASSEMBLYAI_API_KEY"]

def on_turn(client, event):
    # Only act on finalized turns
    if not (event.transcript and event.end_of_turn):
        return

    transcript = event.transcript

    # Generative assist via the LLM Gateway (OpenAI-compatible REST endpoint)
    resp = requests.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        headers={"authorization": API_KEY, "content-type": "application/json"},
        json={
            "model": "claude-sonnet-4-6",
            "messages": [
                {
                    "role": "user",
                    "content": (
                        f"Customer just said: '{transcript}'. "
                        "Suggest the next best action for the agent in one sentence."
                    ),
                }
            ],
            "max_tokens": 200,
        },
    )
    resp.raise_for_status()
    suggestion = resp.json()["choices"][0]["message"]["content"]

    # Push the suggestion to your agent's screen (your own UI code)
    push_to_agent_ui(suggestion)

Two settings do the heavy lifting for call-center audio. Modes replace low-level flag tuning: min_latency for the fastest response, balanced as the default, and max_accuracy for noisy or far-field audio — pick based on how clean your audio is. voice_focus isolates the primary speaker and suppresses background noise (near_field for headsets and phones, far_field for rooms). End-of-turn detection reads tonality, pacing, and rhythm rather than just silence, so it knows the difference between a customer pausing and a customer finishing — which is what keeps your assist suggestions timed to the actual turn.

Step 2: Live speaker separation

If you went with the single-stream approach, turn on streaming diarization. It labels speakers live, then re-clusters and sends a single correction within about half a second of the stream ending, handling up to 10 speakers. For the two-connection approach, you already have deterministic speakers and can skip this.

Step 3: Real-time analysis and the assist itself

Now the transcript becomes coaching. As each turn completes, fire an analysis step and push the result to the agent's screen. Two tools do the work:

‍

Per-turn LLM calls through the LLM Gateway — one API to GPT, Claude, or Gemini — to generate the actual assist: the next-best-action, a suggested answer from your knowledge base, an objection-handling tip.

‍

Real-time Speech Understanding for structured signals you don't want an LLM to guess at — sentiment shifts, entity detection, key phrases. These are fast, deterministic inputs you can act on directly (escalate on a sharp negative sentiment turn, flag a competitor mention, catch a compliance keyword).

def on_turn(event):
    transcript = event.transcript
    # Fast structured signals
    signals = analyze_speech_understanding(transcript)  # sentiment, entities, key phrases
    # Generative assist
    suggestion = generate_via_llm_gateway(
        f"Customer just said: '{transcript}'. Suggest the next best action "
        f"for the agent in one sentence."
    )
    push_to_agent_ui(suggestion, signals)

The design goal is the same as any good copilot: surface the right thing at the right moment, and stay quiet otherwise.

Scaling and compliance

Two things tend to bite in production, so plan for them early.

Concurrency. Contact centers are spiky — Monday morning is not Sunday night. Universal-3.5 Pro Realtime runs with unlimited concurrency and no rate limits, so you're not architecting around a connection cap or throttling at peak. One production account has run tens of thousands of streaming hours over a couple of months without special provisioning.

Compliance. If your calls touch protected health information, AssemblyAI is a business associate under HIPAA and offers a standard Business Associate Addendum (BAA), available self-serve for customers processing PHI. If calls involve card payments, you'll want PII redaction in the stream and a PCI-aware design for the payment portion of the flow. Speech Understanding can redact PII across audio and transcripts.

When a full suite is the better fit

If you want agent assist as a bought outcome — pre-built playbooks, a managed UI, CCaaS integrations you don't have to maintain — a suite like Cresta or Level AI will get you there faster than building. Build the layer yourself when you need a custom experience, your own analysis logic, or the economics of owning the pipeline at scale. The competitive reality in real deals is that the transcription accuracy and latency are what separate a good assist from a frustrating one — so whichever path you take, that's the piece to get right.

Getting started

The real-time transcription layer — the foundation of the whole thing — is a short build. Grab a free API key, open a WebSocket to the streaming endpoint, and get diarized transcripts flowing. Then add your analysis step and start pushing suggestions to the agent. From there it's product work, which is where you want your time going anyway.

Frequently asked questions

What is real-time agent assist?

Real-time agent assist is a live layer that helps a human contact-center agent during a call — transcribing the conversation as it happens and surfacing knowledge-base answers, next-best-action prompts, objection handling, sentiment cues, and compliance reminders in the moment, rather than after the call ends.

How does real-time agent assist work?

It streams the call audio to a speech-to-text model for live transcription, separates the speakers, and runs each completed turn through an analysis layer — an LLM for generative suggestions and speech understanding for structured signals like sentiment and entities. The results are pushed to the agent's screen while the call is still in progress.

Which voice AI platform is best for real-time agent assist?

Closed CCaaS suites like Cresta, Level AI, and Genesys deliver agent assist as a finished product; if you're building your own layer, the platform choice comes down to the real-time speech engine underneath, because latency and transcription accuracy decide whether the coaching lands. AssemblyAI provides that layer as infrastructure — Universal-3.5 Pro Realtime streaming (6.99% pooled WER), live diarization, selectable latency modes, and unlimited concurrency — so you build the assist experience you want rather than buying a fixed one.

How do I test real-time agent assist before building the full thing?

Start by validating the foundation: stream a few sample call recordings to the speech-to-text WebSocket and check transcription accuracy and latency on your own audio (you can do this in the browser playground first). Once the live transcript looks right, add the analysis step on a single call, then a small pilot group, before wiring it into production dialers.

Should I build agent assist or buy a platform?

Buy a suite (Cresta, Level AI, Genesys) if you want a turnkey product and your workflow fits theirs. Build it yourself when you need a custom UI (like a browser extension over an existing dialer), your own analysis logic, tight product integration, or better economics at scale — the real-time layer is essentially streaming transcription, diarization, and per-turn analysis.

What latency do you need for live agent assist?

Fast enough that the suggestion arrives while it's still relevant to the current turn. Universal-3.5 Pro Realtime offers selectable modes — min_latency, balanced, and max_accuracy — and end-of-turn detection that reads tonality and pacing (not just silence), so assist suggestions can be timed to the actual conversational turn.

Can I add real-time agent assist to my existing dialer or contact center?

‍ Yes. A common pattern is a browser extension or overlay that captures the agent's microphone and the dialer's audio, streams both to a speech-to-text WebSocket, and displays suggestions on top of the existing tool — no rip-and-replace of the contact-center platform required.