Time to first token: the latency metric that decides voice agents

Ask an STT vendor how fast they are and you'll get a word error rate and an average latency number. Both are real. Neither tells you whether a conversation with your voice agent will feel alive or feel broken.

Word error rate tells you how accurate the transcript is. Average latency tells you how fast the pipeline runs on average. But the thing a user actually experiences — the silence between "…and that's my question" and the agent's first sound back — isn't captured well by either. That gap is where voice agents win or lose, and the metric that governs it is time to first token.

If you're evaluating speech-to-text for a voice agent, this is the number to build your test around. Here's what it is, why the usual metrics miss it, and how to measure it on your own audio.

What time to first token actually means

Time to first token (TTFT) is the elapsed time from the moment the user stops speaking to the moment your system produces the first usable piece of output it can act on.

The phrase comes from the LLM world, where it means the delay before a model streams its first token. In a voice pipeline it's more useful to think of it end to end: the clock starts when the user finishes their turn, and stops when the first token the agent can use appears — the first chunk of finalized transcript that the LLM can start reasoning on.

That framing matters because a voice agent is a chain — speech-to-text, then an LLM, then text-to-speech — and each link has its own "first token" moment. But they're not independent. The STT stage gates everything downstream: the LLM can't produce a first token until it has text to read, and the TTS can't produce a first sound until the LLM has generated something to say. So the responsiveness the user feels is anchored on how quickly transcription commits to a usable result after the turn ends.

This is why TTFT is the metric that decides voice agents. It's the first domino.

Why WER and average latency don't tell you this

Two numbers dominate STT marketing, and both are genuinely useful for the questions they answer. They just don't answer this one.

Word error rate measures the wrong axis. WER tells you how many words the model got wrong across a whole transcript. It says nothing about when those words arrived. A model can post a beautiful WER and still feel sluggish in conversation because it waits for extra audio context before committing to a result. Accuracy and responsiveness are different axes, and a voice agent needs both. For the accuracy side of this story — and why WER itself is a slippery benchmark — see why your WER benchmark might be lying to you.

Average latency hides the moments that hurt. An average folds together the fast responses and the slow ones. But conversation is unforgiving about the slow ones — a single two-second stall in an otherwise snappy exchange is the thing the user remembers. If a vendor quotes you one latency number, ask which percentile it is. A p50 (median) and a p90 (the slow tail) can be worlds apart, and your users live in the tail as much as the middle.

End-of-turn detection is the hidden clock. Here's the subtle one. Before a streaming model can hand you a final transcript, it has to decide the user is done talking. Wait too long and you've added dead air before the LLM ever sees a word — latency the user blames on your agent. Decide too early and you cut them off. So turn detection isn't separate from latency; it's part of TTFT. A model that reads tonality, pacing, and rhythm to land an end-of-turn decision quickly is buying you responsiveness that no average-latency number will show. We go deep on this in our breakdown of turn detection vs forced endpoints.

The numbers, by transcription path

TTFT isn't a single value — it depends on how you're getting text back. The three transcription paths have very different profiles, and picking the right one for the interaction is half the battle.

• Streaming (real-time). The model transcribes as audio arrives and commits final text shortly after the turn ends. With Universal-3.5 Pro Realtime, end-of-turn detection lands around 300ms by reading tonality, pacing, and rhythm rather than waiting on a fixed silence timer — so the first usable token reaches your LLM fast. This is the path built for live, back-and-forth conversation.
• Sync (short-clip HTTP). For a short utterance sent as a single HTTP request, the Sync API returns the full transcript in one response at roughly 134ms p50. There's no turn-detection step to wait on because you decide when the clip ends. That makes TTFT and time-to-complete-transcript essentially the same moment — which is exactly why it's a strong fit for structured, turn-by-turn agents that own their own voice-activity detection.
• Async (batch). Submit a recording, get it back in seconds to minutes. TTFT as a concept barely applies, because async isn't trying to be interactive — it optimizes for accuracy and full-file speech understanding, not first-token speed. If you're reaching for async in a live agent, you're using the wrong tool.

For the full tradeoff across these three paths — pricing, accuracy, and when to use each — see our guide to real-time vs batch transcription.

Zoom out to the whole agent and the STT number rolls up into an end-to-end budget. A well-tuned cascaded pipeline — STT, LLM, TTS — targets roughly one second from end of user speech to first agent audio, and AssemblyAI's Voice Agent API is built to that ~1s end-to-end target. Every one of those milliseconds starts with how fast transcription hands off the first token.

How to measure it yourself

Don't take anyone's quoted number — including ours — at face value. TTFT is easy to measure on your own audio, and your audio is the only benchmark that matters. A simple protocol:

1. Use realistic clips. Record utterances that sound like your actual users — the accents, the background noise, the domain terms, the short "yes"/"next" answers as well as the long ones. Clean read-aloud audio will flatter every vendor equally and tell you nothing.
2. Timestamp the two events. Mark the sample where the user stops speaking, and the timestamp when your system receives the first finalized token it can act on. The difference is your TTFT for that utterance.
3. Report percentiles, not an average. Collect a few hundred utterances and look at p50 and p90. The median tells you the typical experience; the tail tells you how often the agent feels stuck.
4. Separate the stages. Log first-token time for STT, LLM, and TTS independently. When something feels slow, you want to know which link to fix — and more often than not, the fix isn't where teams assume it is.

Do this once and the conversation with any vendor changes. You stop comparing marketing numbers and start comparing the only thing your users will ever feel.

Why this is the metric that matters

Accuracy is table stakes now — streaming models have closed most of the gap that used to make real-time feel like a compromise. What separates a voice agent that people happily talk to from one they hang up on is responsiveness, and responsiveness is governed by how fast the first token arrives after someone stops speaking.

So when you're evaluating speech-to-text for anything conversational, put TTFT at the center of the test. Ask for percentiles. Measure end-of-turn behavior, not just processing speed. And run it on audio that sounds like your users, because the felt experience of a conversation is decided in a window too small for an average to see.

Get the first token fast and everything downstream has room to work. Get it wrong and no amount of accuracy will make the agent feel alive.

Frequently asked questions

What is time to first token (TTFT) in a voice agent?

Time to first token is the elapsed time from the moment a user stops speaking to the moment the system produces the first usable piece of output it can act on. In a voice pipeline, that first token is the initial chunk of finalized transcript the LLM can start reasoning on — the domino that gates the LLM and text-to-speech stages downstream. It's the metric that most directly governs whether a conversation feels responsive.

Why isn't word error rate or average latency enough to evaluate voice-agent speed?

Word error rate measures how accurate a transcript is, not when the words arrived, so a model can post a great WER and still feel sluggish because it waits for extra context before committing. Average latency folds fast and slow responses together and hides the tail — and a single two-second stall is what a user remembers. To evaluate voice-agent responsiveness you need time to first token, reported as percentiles (p50 and p90), plus end-of-turn behavior.

What is a good time to first token, and what latency should a speech-to-text API hit for voice agents?

For live conversation, you want the first usable token within a few hundred milliseconds of the user finishing their turn. With streaming Universal-3.5 Pro Realtime, end-of-turn detection lands around 300 ms; a short clip sent to the Sync API returns a full transcript at roughly 134 ms p50. At the whole-agent level, a well-tuned STT-LLM-TTS pipeline targets about one second end-to-end from end of speech to first agent audio.

How does time to first token differ across streaming, sync, and async transcription?

Streaming transcribes as audio arrives and commits final text shortly after the turn ends (~300 ms end-of-turn with Universal-3.5 Pro Realtime), making it the path built for live conversation. Sync returns a full short-clip transcript in a single HTTP response at ~134 ms p50, with no turn-detection wait because you decide when the clip ends. Async optimizes for accuracy and full-file speech understanding over seconds to minutes, so TTFT barely applies — it's the wrong tool for a live agent.

How do I measure time to first token on my own audio?

Record clips that sound like your real users — accents, background noise, domain terms, and both short and long answers — then timestamp two events per utterance: when the user stops speaking and when your system receives the first finalized token. The difference is your TTFT. Collect a few hundred utterances, report p50 and p90 rather than an average, and log first-token time for STT, LLM, and TTS separately so you know which stage to fix.

How is turn detection related to time to first token?

Turn detection is part of TTFT, not separate from it. Before a streaming model can hand you a final transcript, it has to decide the user is done talking — wait too long and you add dead air the user blames on your agent; decide too early and you cut them off. A model that reads tonality, pacing, and rhythm to land an end-of-turn decision quickly (rather than waiting on a fixed silence timer) directly improves responsiveness that no average-latency number will reveal.

‍