Real-time speech-to-text: the complete developer guide

Real-time speech-to-text converts spoken words into written text within milliseconds as audio streams — not after a recording ends, but as you speak. It's the layer that makes live captions, meeting tools, and voice agents feel instant instead of laggy.

This guide is for developers building on that layer: how streaming recognition works, how to evaluate it, how to get started with real code, and — because it's the fastest-growing use case — which streaming model is best for voice agents and why. Let's get into it.

What is real-time speech-to-text?

Real-time speech-to-text (also called streaming transcription) processes audio in continuous small chunks — typically 20 to 100 milliseconds each — and emits text as the audio arrives. Instead of waiting for a complete file, the model works on a live stream and produces two kinds of output:

• Partial results: fast initial guesses that update as more context arrives.
• Final results: confirmed text once the model has enough context to lock it in.

The defining metric is latency — the delay between someone speaking and the text appearing. Everything about a good streaming system is a negotiation between latency and accuracy.

How streaming speech recognition works

Audio capture and streaming

A microphone converts sound into digital chunks, and a connection stays open for the whole session — unlike file transcription, where you record first and upload later. Audio quality drives accuracy: microphone type, background noise, and format (PCM or Opus) all matter. Most streaming APIs want 16 kHz mono PCM16.

Real-time processing and partial results

Two models work together. An acoustic model maps sound patterns to phonemes; a language model uses context to resolve ambiguity ("their" vs "there," "four" vs "for"). Stronger models handle accents, noise, and technical vocabulary without falling apart — and they revise partial results intelligently as context accumulates rather than committing to a bad early guess.

Speaker identification and timestamps

Speaker diarization labels each segment with a speaker identifier, and timestamps anchor every word in time. Multichannel audio (each speaker on their own channel) produces cleaner separation than single-channel, but modern models handle single-channel diarization well — Universal-3.5 Pro Realtime supports diarization for up to 10 speakers in a stream. For the details, see our deep dive on streaming speaker diarization.

Real-time vs. batch processing

The question that decides which one you need: does someone (or something) need to read or act on the text while the conversation is still happening? If yes, you need real-time. If you're analyzing a finished recording, batch is more accurate because it has the full context.

For a fuller treatment of the two modes, see our complete guide to speech-to-text.

What are the main use cases for real-time speech-to-text?

Live captions and accessibility

The lowest-latency requirement of the bunch, and the least forgiving — a transcription error blocks meaning for a deaf or hard-of-hearing viewer. Zoom and Google Meet now ship native live captioning built on streaming STT.

Meeting transcription and collaboration

Streaming creates searchable, timestamped records that integrate with Slack or Microsoft Teams. The value compounds — a library of searchable transcripts becomes more useful over time. Tools like Granola and Fireflies are built on exactly this.

Voice assistants and real-time commands

The foundation of voice-driven interfaces, and highly latency-sensitive. Enterprise versions show up in medical dictation, warehouse inventory lookups, and hands-free field-service guides.

Voice agents

The fastest-growing and most demanding use case. A voice agent has to transcribe perfectly enough to feed a live decision — mishear a command and the whole conversation derails. It needs accurate turn detection, clean interruption handling ("barge-in"), and rock-solid entity accuracy for emails, phone numbers, and dates. This is where the choice of streaming model matters most, so we'll come back to it.

How to evaluate real-time speech-to-text accuracy and performance

Two accuracy metrics matter, and they're not the same thing:

• Word Error Rate (WER): the percentage of words inserted, deleted, or substituted incorrectly. Lower is better. It’s the headline number — but it’s a blunt instrument. See why in word error rate is broken.
• Entity error rate: how well the model captures the business-critical tokens — names, account numbers, emails, domain terms. For voice agents this matters more than overall WER, because a wrong confirmation number breaks the task even if the sentence around it is perfect.

On the latency side, three measurements matter:

And two things that aren’t traditional benchmarks but decide the experience for voice agents: turn-detection quality and interruption handling. Benchmarks only tell part of the story — testing with your own audio before you commit gives a far more reliable picture. Our guide on how to evaluate speech recognition models walks through a proper test.

‍

Benchmark callout — voice-agent streaming WER (Pipecat open STT benchmark, real agent conversations):

• Universal-3.5 Pro Realtime: 6.99% WER
• Google Chirp3: 9.04%
• ElevenLabs Scribe v2: 9.76%
• Deepgram Flux: 15.58%

Entity error rate: Universal-3.5 Pro Realtime 15.31% vs Deepgram 50.5%, ElevenLabs 39.7%, Google 21.51%. Full data at assemblyai.com/benchmarks.

Why turn detection is the real differentiator

Here's the thing most streaming comparisons miss: for voice agents, accuracy gets you in the door, but turn detection decides whether the conversation feels human.

Traditional voice activity detection waits for silence to decide you're done talking. But people pause. They say "my number is... hang on... 8 6 7..." and a silence-based system talks right over them. Universal-3.5 Pro Realtime uses neural end-of-turn detection that reads tonality and pacing — not just silence — and lands an end-of-turn decision in roughly 300ms. That's the difference between an agent that waits its turn and one that interrupts.

It's also why bring-your-own-stack setups that bolt a generic VAD onto a general-purpose STT model struggle: they're inferring turns from silence when the model itself should be reading the human signals. This is the differentiator that carries the rest.

How to get started with real-time speech-to-text

Three paths, depending on how much you want to build.

Path 1 (recommended for voice agents): the Voice Agent API

The Voice Agent API combines speech-to-text, LLM, and text-to-speech into a single WebSocket connection, with turn detection, VAD, and interruption handling built in. It’s a flat $4.50/hr, billed per second. Most developers get a working voice agent running the same afternoon they start — no orchestration to wire up, no three-vendor latency budget to manage. It frames AssemblyAI the way we think about ourselves: invisible infrastructure — we’re not the agent, we’re the platform you build it on.

Path 2: cloud STT API for custom applications (bring your own stack)

If you want to own the pipeline — your LLM, your TTS, your orchestration on LiveKit or Pipecat — use Universal-3.5 Pro Realtime ($0.45/hr) as your streaming speech-to-text layer. The current streaming endpoint is the v3 WebSocket:

wss://streaming.assemblyai.com/v3/ws?sample_rate=16000&speech_model=universal-3-5-pro

Note the singular speech_model param for streaming (async transcription uses a plural speech_models fallback list). The old v2 URL wss://api.assemblyai.com/v2/realtime/ws is inactive and returns 410 — if you're on it, migrate.

Here's a compact, current v3 streaming example using the Python SDK. It connects, prints partial and final turns, and closes cleanly:

# pip install "assemblyai>=1.0.0"
import os
from assemblyai.streaming.v3 import (
    StreamingClient, StreamingClientOptions, StreamingEvents,
    StreamingParameters, TurnEvent,
)

def on_turn(_, event: TurnEvent):
    tag = "FINAL" if event.end_of_turn else "partial"
    print(f"{tag}: {event.transcript}")

client = StreamingClient(StreamingClientOptions(api_key=os.environ["ASSEMBLYAI_API_KEY"]))
client.on(StreamingEvents.Turn, on_turn)
client.connect(StreamingParameters(sample_rate=16000, speech_model="universal-3-5-pro"))
# Feed 16 kHz mono PCM16 chunks (50-1000ms each) via client.stream(chunk)
client.disconnect(terminate=True)  # sends Terminate and closes cleanly

Universal-3.5 Pro Realtime ships three latency modes — min_latency, balanced, and max_accuracy — plus voice_focus for near/far-field audio, diarization for up to 10 speakers, and agent_context, which feeds prior conversation turns into the model and cuts WER by 10.2%. For a longer walkthrough, see real-time transcription in Python.

Path 3: ready-to-use applications

Apps like Otter.ai or Google Live Transcribe need no code and are handy for quality-testing a provider before you build.

Why speech-to-text accuracy is the foundation of voice agent quality

For a voice agent, STT isn't one component among many — it's the foundation. If the transcription is wrong, the LLM responds to the wrong thing, confidently. There's no recovering downstream from a misheard account number.

Purpose-built streaming models handle conversational audio — overlapping speech, disfluencies, phone-line compression — far better than general-purpose transcription models retrofitted for streaming. Universal-3.5 Pro Realtime is built for exactly this: 6.99% WER on the Pipecat voice-agent benchmark, a 15.31% entity error rate against Deepgram's 50.5%, and neural turn detection that reads the conversation instead of counting silence. It's also the streaming foundation underneath the Voice Agent API.

If you’re weighing the managed path against building it yourself, our post on where voice agent stacks start showing their limits is the honest version of that trade-off — and our cornerstone guide to AI voice agents covers the architecture end to end.

Final words

Real-time speech-to-text is streaming audio in small chunks, generating partial results that refine into final transcripts with timestamps and speaker labels attached. For most applications, that's enough. For voice agents, the stakes are higher: you need a streaming model designed for conversational audio, with strong entity accuracy and turn detection that reads humans instead of silence.

The teams shipping the best voice experiences in 2026 aren't the ones who found a clever prompt. They're the ones who got the foundation right — and then stopped thinking about it, because good infrastructure is the kind you forget is there.

Try it yourself

Talk to our live Voice Agent API demo to hear real-time transcription and neural turn detection in an actual conversation.

Building your own? Get your free API key — free API credit to start, no credit card required — and try streaming transcription on your own audio. Prefer to browse first? Open the playground.

Frequently asked questions about real-time speech-to-text

What is real-time speech-to-text, and how is it different from regular transcription?

Real-time speech-to-text transcribes audio as it streams in, emitting text within milliseconds instead of waiting for a finished recording. Regular batch transcription processes a complete file afterward, which is slightly more accurate but useless for live captions, voice commands, or voice agents.

Which streaming speech-to-text model is best for voice agents in 2026?

You want low latency, strong neural turn detection, and high entity accuracy. Universal-3.5 Pro Realtime posts 6.99% WER on the Pipecat voice-agent benchmark and a 15.31% entity error rate versus Deepgram's 50.5%, with sub-second end-to-end responsiveness — which is why it's the foundation of the Voice Agent API.

How does the Voice Agent API differ from wiring up separate STT, LLM, and TTS providers?

It combines all three into a single WebSocket connection at a flat $4.50/hr, billed per second, with built-in turn detection and tool calling. You collapse three vendors, three invoices, and three debugging surfaces into one — and skip the latency budget juggling.

What is the current WebSocket endpoint for AssemblyAI streaming, and did the old one change?

The current streaming endpoint is wss://streaming.assemblyai.com/v3/ws with a singular speech_model parameter. The old v2 URL wss://api.assemblyai.com/v2/realtime/ws is inactive and returns 410, so any integration still on it needs to migrate.

What latency should a real-time voice agent target end to end?

Aim for a full response under one second (Time to Complete Turn) for a natural feel. The STT component should deliver its first partial result within about 300ms (Time to First Token), leaving room for the LLM and text-to-speech to finish the cycle.

How accurate is real-time speech-to-text in noisy, real-world conditions?

Accuracy dips in noise, but models trained on diverse and telephony audio hold up far better than general-purpose ones. Features like voice_focus for near/far-field audio and agent_context — which cuts WER 10.2% using prior turns — help preserve accuracy on messy live calls.