TTS latency explained: time to first audio vs. time to first byte

A caller asks your voice agent for their balance. They stop talking. Then nothing happens. Half a second goes by, then a full second, and they say "Hello?" right as the agent starts to answer. Now both of them are talking at once.

TTS latency is the part of that silence spent turning the agent's reply into sound. Text-to-speech (TTS) is the last step before the caller hears anything, so it gets a lot of the blame. The trouble is that the number most TTS specs report, time to first byte, isn't what the caller hears.

This guide separates time to first byte (TTFB) from time to first audio (TTFA), shows where TTS sits in the latency budget, and explains how to measure TTFA yourself instead of trusting a spec sheet.

What is TTS latency?

TTS latency is the delay between sending text to a text-to-speech system and hearing the first part of the spoken result. In a voice agent, it's the last step of the response loop, after speech-to-text has transcribed the caller and a language model has started writing a reply.

But "latency" covers several different spans: only the model, only the server, the server plus the network, or the whole path to the caller's ear. Two systems can quote the same number and feel very different on a call. So the first thing to ask about any TTS latency figure is where the clock starts and where it stops.

Time to first byte vs. time to first audio

TTFB tells you how fast the server responds. TTFA tells you how fast the caller hears something.

Time to first byte (TTFB)

Time to first byte comes from web performance, where it measures how long a browser waits for the first byte of a server's response. In TTS, TTFB is the time from sending text to receiving the first audio bytes. Unlike web TTFB, those bytes may be a header or silence.

TTFB isolates server-side work: queueing, inference for the first audio frames, and encoding. It's the easiest number to benchmark, which is why it's the most common figure in TTS specs. But a received byte isn't a heard sound.

Time to first audio (TTFA)

Time to first audio is the time from requesting speech to the moment the first audible sample plays for the listener. It includes everything TTFB measures, plus what happens after the bytes leave the server:

• Network transit to your application server and on to the caller, often through a telephony provider
• Decoding, resampling, or format conversion
• The playback buffer that keeps audio from stuttering
• Device and audio-driver output latency

In a voice agent, TTFA is the TTS share of the silence your caller sits through. It's the number to optimize.

Why TTFB and TTFA drift apart

Start with the first message itself. A streaming TTS response can open with a container header, an empty frame, or a few milliseconds of leading silence. Each of those counts as a "first byte," and none of them is something the caller can hear. TTFB can look fast while the line is still quiet.

After that, the gap depends on your setup, not the model. Resampling for a phone line, a cross-region hop, and a conservative buffer can add more delay after the first byte than the model spent producing it. A server colocated with your telephony provider, using the call's codec, keeps TTFA close to TTFB.

Where TTS sits in the voice agent latency budget

Callers don't experience TTS latency by itself. They experience the whole loop: end-of-turn detection, a final transcript, the LLM's first token, then TTS, with network hops in between. TTS is last, so it inherits every delay before it. A fast TTS behind a slow first token still leaves dead air.

How much time do you have? People typically leave a gap of about 200 milliseconds between turns (Stivers et al., PNAS, 2009). A voice agent won't match human timing end to end, but every leg spends from the same small window. For the full breakdown, see why latency makes or breaks voice AI, and for the LLM leg, our guide to time to first token.

What is streaming TTS?

Streaming TTS is text-to-speech that starts sending audio before it has finished generating the whole utterance, and often before it has received all of the text. Playback starts while the rest is still being generated.

The term covers two separate capabilities, and realtime TTS for voice agents usually needs both.

Streaming output vs. batch synthesis

Batch synthesis generates all of the audio for a block of text and returns it as one response, so the listener waits for the whole thing. Streaming output sends audio in small chunks as soon as each one is ready. TTFA then depends on the first chunk, not on how long the whole sentence takes to render. For real-time text to speech in a conversation, streaming output is required.

Streaming input: feeding text as the LLM writes it

Streaming input means the TTS system accepts text a piece at a time over an open connection while the LLM is still generating. Systems that support both directions are often called dual-streaming or bidirectional. That's what lets an agent speak the first clause of a reply while the LLM is still writing the second sentence. Our voice agent architecture guide shows how the stages overlap in a streaming pipeline.

Chunking: how much text to send at once

Even with streaming input, you decide when to flush text to the TTS:

• Word by word: lowest theoretical TTFA, but little context, so prosody suffers.
• Clause or punctuation boundaries: a common middle ground. Flush on commas, periods, and question marks.
• Full sentences: the most natural intonation, but a long opening sentence delays the first audio.

The highest-impact fix is often editorial, not technical: prompt your LLM to open with a short first sentence. "Sure, I can help with that." reaches the caller's ear long before a 40-word opener would.

WebSockets vs. HTTP for real-time text to speech

Chunked HTTP responses work for one-off requests. Voice agents favor a persistent WebSocket for three reasons:

• Connection setup happens once. A new HTTPS request may pay for DNS, TCP, and TLS handshakes. A WebSocket opened when the call starts, and reused for every turn, moves that cost out of the response window.
• Text can stream in. Streaming input needs a channel you can keep writing to while audio comes back.
• Interruptions are cheap. When the caller barges in, stopping playback and canceling synthesis is a single control message.

Pre-warm everything: open the TTS, speech-to-text, and LLM connections when the call connects, not when the first turn ends.

Where TTS latency hides

When TTFA is higher than the TTFB you measured, the extra time is usually in one of these places:

• A long first chunk. If your chunker waits for a 30-word sentence, first audio waits too.
• Format conversion. Phone audio typically runs at 8 kHz mu-law. If your TTS returns 24 kHz PCM, every chunk gets resampled. Request the format your telephony layer expects.
• Playback buffering. Every millisecond buffered is a millisecond of silence. Tune the buffer to your network.
• Region mismatch. A TTS endpoint on another continent from your telephony provider adds a round trip to every chunk.
• Load. One test stream isn't peak traffic. Measure P95 at production concurrency.

Text normalization (expanding "$1,249.50" into words) costs a little time too, but it's mostly a correctness problem, so test it separately.

How to measure time to first audio

Published figures are measured under someone else's conditions. To know your own TTFA, take timestamps on your side:

• t_send: when your code sends the first text chunk to TTS.
• t_first_byte: when the first non-empty audio message arrives. The difference from t_send is your observed TTFB.
• t_playable: when your playback buffer is full and some audio is non-silent. The difference from t_send is TTFA at your server.
• Device output: add the caller's output path, including your telephony provider's media stream.

Here's a vendor-neutral sketch in Python 3.11+. The endpoint and message format are placeholders, so swap in your provider's streaming protocol. It skips empty frames, counts TTFA only once the buffer is full and a non-silent sample has arrived, and reports P50 and P95 across repeated runs.

import asyncio
import json
import statistics
import time
from array import array

import websockets  # pip install websockets

TTS_URL = "wss://tts.example.com/v1/stream"  # placeholder endpoint
SAMPLE_RATE = 24000        # Hz, match the format you request
BYTES_PER_SAMPLE = 2       # 16-bit PCM
PLAYOUT_BUFFER_MS = 60     # audio your player holds before it starts
SILENCE_THRESHOLD = 500    # 16-bit amplitude; below this counts as silence
TIMEOUT_S = 10


def ms(start, end):
    return None if end is None else round((end - start) * 1000, 1)


def is_audible(chunk: bytes) -> bool:
    samples = array("h", chunk[: len(chunk) - len(chunk) % 2])
    return any(abs(s) > SILENCE_THRESHOLD for s in samples)


async def measure_ttfa(text: str) -> dict:
    min_bytes = SAMPLE_RATE * BYTES_PER_SAMPLE * PLAYOUT_BUFFER_MS // 1000
    t_connect = time.perf_counter()
    async with websockets.connect(TTS_URL) as ws:
        t_open = time.perf_counter()
        t_send = time.perf_counter()
        await ws.send(json.dumps({"text": text, "format": "pcm_s16le", "sample_rate": SAMPLE_RATE}))
        buffered, t_first_byte, t_playable, t_audible = 0, None, None, None
        try:
            async with asyncio.timeout(TIMEOUT_S):  # Python 3.11+
                async for message in ws:
                    if not isinstance(message, (bytes, bytearray)) or not message:
                        continue  # skip JSON status messages and empty frames
                    now = time.perf_counter()
                    t_first_byte = t_first_byte or now
                    buffered += len(message)
                    if t_playable is None and buffered >= min_bytes:
                        t_playable = now  # the player can start here
                    if t_audible is None and is_audible(message):
                        t_audible = now   # first chunk with non-silent samples
                    if t_playable and t_audible:
                        break
        except TimeoutError:
            pass
    return {
        "connect_ms": ms(t_connect, t_open),
        "ttfb_ms": ms(t_send, t_first_byte),
        "ttfa_ms": ms(t_send, max(t_playable, t_audible)) if t_playable and t_audible else None,
    }


async def run_many(text: str, n: int = 200) -> None:
    results = [await measure_ttfa(text) for _ in range(n)]
    for key in ("ttfb_ms", "ttfa_ms"):
        values = [r[key] for r in results if r[key] is not None]
        if len(values) < 2:
            print(f"{key}: not enough samples ({len(values)}/{n})")
            continue
        q = statistics.quantiles(values, n=100, method="inclusive")
        print(f"{key}: P50 {q[49]:.1f} ms  P95 {q[94]:.1f} ms  ({len(values)}/{n} runs)")


if __name__ == "__main__":
    asyncio.run(run_many("Your order number is W 3 1 7 8 4 2.", n=200))

Three habits make the numbers worth trusting:

• Report percentiles. The tail decides whether your agent talks over callers.
• Use your real text. Account numbers, addresses, prices. Clean demo phrases flatter every system.
• Measure from where you'll run. Same region, concurrency, and audio format as production, then measure the full voice-to-voice loop. Our Vapi latency guide walks through how to measure each leg in a real stack.

What a fast TTS can't fix

Here's what latency benchmarks miss: the most expensive delay in a voice agent conversation isn't a slow first audio frame. It's a whole extra turn.

If speech-to-text mishears an account number, the agent reads back the wrong digits, the caller corrects it, and the agent asks again. That's two more trips through the loop, each paying the entire latency budget. No TTFA improvement makes that conversation feel fast.

Turn detection works the same way. A detector that fires on every pause cuts callers off mid-number, and a long silence timer adds dead air before the LLM sees a word. Content-aware turn detection, which looks at what the caller said and not just how long they've been quiet, protects the budget at its first step. Universal-3.6 Pro Realtime, AssemblyAI's streaming speech-to-text model, is built for this first leg of the loop. Our guide to choosing a speech-to-text API for voice agents covers what to test.

So treat TTS latency as one line in a budget, not a leaderboard rank. Measure TTFA, not just TTFB. Stream everything. Keep the first sentence short. And make sure the agent heard the caller correctly, because the fastest turn is the one you don't have to repeat. For the rest of the stack, see our voice agent solutions and our comparison of text-to-speech APIs.

Frequently asked questions

What is TTS latency?

TTS latency is the delay between sending text to a text-to-speech system and hearing the first part of the spoken result. In voice agents, it's the final leg of the loop after speech-to-text and the LLM. The most useful measure is time to first audio, because it tracks what the listener hears.

What is the difference between time to first byte and time to first audio?

Time to first byte (TTFB) measures how long it takes to receive the first audio data from the TTS server. Time to first audio (TTFA) measures how long it takes until the first audible sample plays for the listener. TTFA adds network, decoding, format conversion, and buffering, so it's never lower than TTFB.

What is a good TTS latency for voice agents?

There's no single target, because TTS shares one budget with turn detection, speech-to-text, the LLM, and the network. People typically leave about 200 milliseconds between turns, so every leg has to be small. Set a voice-to-voice target, subtract your measured STT and LLM time, and treat the rest as your TTFA budget at P95.

What is streaming TTS?

Streaming TTS is text-to-speech that sends audio in chunks as they're generated instead of returning one finished file. Many streaming systems also accept text incrementally, so playback can start before the agent's full reply exists.

Why is my TTS time to first byte low but callers still wait?

A low TTFB only proves the server answered quickly; the first bytes can be a header, an empty frame, or silence. The rest of the wait usually comes from resampling to phone audio, a cross-region hop, or a large playback buffer. Measure time to first audio on your side and compare it to TTFB to see which one it is.

How do I measure time to first audio?

Timestamp when you send the first text chunk, when the first non-empty audio bytes arrive, and when enough audible audio has arrived to start playback. That last interval is your TTFA at the server. Run hundreds of requests with your real text and report P50 and P95.