> ## Documentation Index
> Fetch the complete documentation index at: https://assemblyai.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Optimizing Accuracy and Latency

> Get the first audio of each reply sooner, and have the text read the way you mean it.

A voice agent is judged on how long the user waits for a reply to start, and on whether the reply says what you meant. Synthesis itself runs on our servers. Most of what you can change sits around it, in how you connect, how you send the text and how you write it. This page covers the choices that matter, with the saving we measured for each one, run from a client in the same AWS region as the API.

## Measure the time to first audio

Before tuning anything, find out where the wait is.

* **HTTP API.** Every `200` carries a `Server-Timing` header whose `ttfa` is our server's own time from receiving the request to sending the first audio byte. Your client's time to the first byte, minus `ttfa`, is the network, the connection setup and your client. When that difference is large, the connection and data zone habits below are the ones that help. See [Response headers](https://assemblyai.com/docs/tts/api-reference/http-api/overview.md#response-headers).
* **WebSocket API.** Time a turn from the `Generate` you send to the first `Audio` frame, and time opening a session, from connecting to `Begin`, separately. The first is what a turn waits for on an open session. The second is what a turn adds when it opens its own.

The SDKs do not expose `Server-Timing`, so measure the HTTP split with a plain client. `httpx` ships with the Python SDK:

```python theme={null}
import os
import time

import httpx

client = httpx.Client(headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]})
body = {"text": "Hi, thanks for calling. How can I help you today?", "voice_id": "aai-audrey"}

for _ in range(5):
    start = time.perf_counter()
    with client.stream("POST", "https://tts.us.assemblyai.com/v1/speech", json=body) as response:
        next(response.iter_bytes())  # the first audio bytes
        client_ms = (time.perf_counter() - start) * 1000
        server_timing = response.headers["server-timing"]  # "open;dur=2.7, model;dur=40.4, ttfa;dur=43.1"
    print(f"client {client_ms:.0f} ms  server {server_timing}")
```

The first run also opens the connection; the rest run on the kept connection. The difference between the client's time and `ttfa` is what the habits below can remove.

## Use one WebSocket session per conversation

A conversation belongs on the [WebSocket API](https://assemblyai.com/docs/tts/connect-parameters.md), with one session held open for the whole call and each reply a request on it. Text can be sent a token at a time as an LLM writes it, the server starts speaking once it holds whole sentences, and `Cancel` stops the request in flight when the user interrupts. The rest of this page assumes that shape. [Choose an interface](https://assemblyai.com/docs/tts/getting-started/choose-an-interface.md) covers the decision itself, and when whole texts over HTTP or the OpenAI-compatible route fit better.

## Open the session before the first reply

Opening a session costs a TCP and a TLS handshake, the WebSocket upgrade, then our server's checks before `Begin`, which are the credential, the rate limit and the voice it loads when the session opens. A session opened per sentence pays all of it on every turn, and spends the [sessions-per-minute limit](https://assemblyai.com/docs/tts/limits.md#session-lifetime) on every turn as well. A turn on a session opened earlier pays none of it, and the voice is ready before your text arrives. In the API's region that is about 25 ms per turn (24 to 27 ms at the median, 20 to 22 ms at p95), and about 75 ms on the first turn with a voice the server has not used recently (105 ms at p95). From farther away, add the three round trips of a new WebSocket.

Open the session when the call starts or the page loads, not when the first reply is ready, and keep it open between turns. The server closes a session after two minutes without a frame in either direction, or after your shorter [`inactivity_timeout`](https://assemblyai.com/docs/tts/connect-parameters#inactivity-timeout), so send `KeepAlive` at least once per idle timeout while the caller is silent. It only restarts the idle timer, so sending it more often makes nothing faster, and a tight loop closes the session with `3010`. A session ends at `Begin.expires_at`, at most an hour after it opened. Open its replacement in a pause before then, send the following turns on it once its `Begin` arrives, and `Terminate` the old one, so no turn waits for a reconnect. One session per conversation is enough: an idle session still holds resources on our servers, so do not keep spares open in reserve.

**Python SDK**

```python theme={null}
# Fragment. You supply call_is_open(), next_reply() and play().
import os

import assemblyai as aai

config = aai.TTSConfig(voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", language="english")  # Audrey

# When the call starts, not when the first reply is ready.
session = aai.TTSSession(config, api_key=os.environ["ASSEMBLYAI_API_KEY"])
session.connect()  # waits for Begin; the voice is loaded before any text arrives

while call_is_open():
    reply = next_reply(timeout=60)  # your agent; None after 60 s of silence
    if reply is None:
        session.keep_alive()  # restarts the idle timer, nothing else
        continue
    for chunk in session.speak(reply):
        play(chunk)

session.terminate()
```

**JavaScript SDK**

```javascript theme={null}
// Fragment. You supply callIsOpen(), nextReply() and play().
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

// When the call starts, not when the first reply is ready.
const session = client.tts.session({ voiceId: "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", language: "english" }); // Audrey
await session.connect(); // waits for Begin; the voice is loaded before any text arrives

while (callIsOpen()) {
  const reply = await nextReply(60_000); // your agent; null after 60 s of silence
  if (reply === null) {
    session.keepAlive(); // restarts the idle timer, nothing else
    continue;
  }
  for await (const chunk of session.speak(reply)) play(chunk);
}

await session.close();
```

The integrations do this for you. [LiveKit](https://assemblyai.com/docs/tts/livekit.md#latency) opens the session when you call `tts.prewarm()` at the start of the call, and [Pipecat](https://assemblyai.com/docs/tts/pipecat.md#latency) connects when the pipeline starts.

### Reuse one TLS context in Python

With the raw `websockets` package, `websockets.connect` builds a new TLS context for every connection unless you give it one. On Python linked against OpenSSL 3.0, such as the system Python of Ubuntu 24.04, building one took about 14 ms of CPU time, and 22 to 52 ms on a busy machine; with OpenSSL 3.5 it took 4 ms or less. Build one when your process starts and pass it to every connection. The SDKs and HTTP clients such as httpx build their context once, when the client is created, which is one more reason to keep one client per process.

```python theme={null}
import ssl

import websockets

# Once per process. One context can be shared by any number of connections.
SSL_CONTEXT = ssl.create_default_context()


async def open_session(url, headers):
    """Open a session; the caller reads Begin and closes the connection."""
    return await websockets.connect(url, additional_headers=headers, ssl=SSL_CONTEXT)
```

In the API's region, a turn that opened its own session reached first audio 25 ms sooner with a shared context than with a new context per connection (38 ms sooner at p95).

## Keep HTTP connections open

If you send whole replies over HTTP, the same idea applies to the connection. Build one HTTP client, or one OpenAI client, per process, and send every request through it. A new client starts with no open connection, and in Python it also builds a new TLS context.

Our load balancer keeps an idle connection open for an hour, but common clients close it much sooner: Python's httpx and httpx2, which the OpenAI Python SDK is built on, after 5 seconds, and Node's built-in `fetch` after 4. A voice agent whose replies are further apart than that opens a new connection, a TCP and a TLS handshake, before nearly every reply. Raise the client's idle limit above your longest pause between replies; 2 minutes covers most conversations.

**Python**

```python theme={null}
import httpx

# One client per process: httpx's own pool sizes, with idle connections kept
# for 2 minutes instead of 5 seconds.
client = httpx.Client(
    limits=httpx.Limits(max_connections=100, max_keepalive_connections=20, keepalive_expiry=120),
    timeout=httpx.Timeout(5.0, read=30.0),
)
```

**Node**

```javascript theme={null}
import { Agent, fetch } from "undici";

// One agent per process: idle connections kept for 2 minutes instead of 4 seconds.
// Pass undici's own fetch with its Agent; npm install undici.
const agent = new Agent({ keepAliveTimeout: 120_000 });

const response = await fetch("https://tts.us.assemblyai.com/v1/speech", {
  method: "POST",
  dispatcher: agent,
  headers: { Authorization: process.env.ASSEMBLYAI_API_KEY, "Content-Type": "application/json" },
  body: JSON.stringify({ text: "Hello from AssemblyAI." }),
});
```

For the OpenAI SDKs, Pipecat and LiveKit, see [OpenAI compatibility](https://assemblyai.com/docs/tts/openai-compatibility.md#keep-connections-between-replies).

The first request through a client still opens a connection. Open it while nobody is waiting, for example when a call starts, with any cheap request through the same client. `HEAD /v1/openai/models` takes no credential and synthesizes nothing:

```python theme={null}
client.head("https://tts.us.assemblyai.com/v1/openai/models")
```

The connection then stays open for the client's idle limit, so open it shortly before the first reply rather than at application startup, unless you raised the limit as above.

In the API's region, a reply sent on a kept connection after a pause of 6 or 65 seconds reached its first byte 5 to 7 ms sooner than one that reconnected, with httpx and with the OpenAI Python SDK. With Node's `fetch` the difference was too small to measure in the region. From farther away, every reconnect costs two round trips, with any client.

## Send text as it is written, and flush each reply

The server starts speaking as soon as it holds enough of the first sentence, so when text streams from an LLM the first audio cannot come before that sentence has arrived. Much of the slowest first audio we see on the WebSocket API comes from text that is held back on the client and sent a few characters at a time, not from synthesis. Send each piece of text with `Generate` as soon as you have it, and keep the space at the edge of each piece, since pieces are joined exactly as sent.

Send `Flush` as soon as a reply is complete. Without it, the end of the reply is held until more text arrives, and a short reply is never spoken at all. Flush once per reply, or at sentence boundaries within a long one, and never per token. A request that ends mid-sentence is synthesized on its own, which produces choppy prosody, and a per-token loop exceeds the control-frame limit and closes the session. See [Generate](https://assemblyai.com/docs/tts/message-frames.md#generate) and [Limits](https://assemblyai.com/docs/tts/limits.md).

Keep the first sentence of a reply short. The agent speaks after the first sentence, so a short opening starts the audio sooner than a long one with the same content.

## Stream raw PCM in the format your player needs

A streamed response sends its first byte with the first audio. A buffered one, `"stream": false` on the HTTP API, sends nothing until the whole clip is ready, which is right for a file and wrong for a conversation. MP3 adds about 46 ms of encoder silence to the start of every streamed response at 24 kHz, and more at lower rates. For a voice agent, stream `pcm_s16le`, which is `pcm` on the OpenAI-compatible route.

The service's native output is 24 kHz 16-bit PCM. Every other rate or codec is converted on the server before it reaches you, and the conversion adds no measurable latency, so set the format your playback path needs explicitly rather than converting it yourself. 16 kHz `pcm_s16le` is the common choice when the audio feeds a pipeline that already runs at 16 kHz, and 8 kHz `pcm_mulaw` or `pcm_alaw` matches a carrier media stream directly. Read from the socket continuously and buffer the audio locally. Your player, not the socket, should be what waits. See [Audio formats](https://assemblyai.com/docs/tts/connect-parameters.md#audio-formats) on WebSocket API basics, and the [HTTP API's](https://assemblyai.com/docs/tts/api-reference/http-api/overview.md#audio-formats) WAV and MP3 containers.

## Use the nearest data zone

`tts.us.assemblyai.com` runs in us-east-1 (N. Virginia) and `tts.eu.assemblyai.com` in eu-north-1 (Stockholm). Choose the zone your data residency requires first, and the zone that holds your [cloned voices](https://assemblyai.com/docs/tts/cloned-voices.md#data-zones), since a voice is only usable in the zone it was created in. With no such requirement, choose the zone nearer to the machine that calls the API. See [Cloud endpoints and data residency](https://assemblyai.com/docs/tts/endpoints-and-data-zones.md).

Every request costs at least one network round trip to the zone, and a new connection costs more before the request can leave: two round trips for HTTPS (TCP, then TLS 1.3) and three for a WebSocket (TCP, TLS and the upgrade). From AWS us-west-2 (Oregon) we measured a round trip of about 60 ms to us-east-1 and about 155 ms to eu-north-1. A request from Oregon to `tts.us` on an open connection therefore spends about 60 ms on the network, a new HTTPS connection adds about 120 ms to it, and a new WebSocket about 180 ms. The farther your client is from its zone, the more the connection habits above are worth.

## In a browser

A page opens each session with a [temporary token](https://assemblyai.com/docs/tts/authenticate-with-a-temporary-token.md), so a connection first waits for the page to ask your server and for your server to mint the token. In our tests from the token host's region a mint took about 34 ms (57 ms at p95), and a session opened with a token minted ahead reached first audio about 32 ms sooner (40 ms at p95), before counting the round trip from the page to your server.

* **Keep a spare token.** Mint it on an early hint that a session will be needed, such as the pointer entering the text box, and only when the page will have to open a session, such as after the idle session closed. While a session is open, a spare goes unused and only sits in the page as a live credential. Drop it well before its `expires_in_seconds` runs out, and give every connection a token of its own.
* **Reopen on intent.** A page that lets an idle session close and reopens it when the user presses Speak makes that click wait for the token, the handshakes and `Begin`. Reopen when the user focuses the text box instead, while they type, and the click finds the session open. A click that comes before the reopened session's `Begin` should keep its text and speak it when `Begin` arrives.

## Write the text the way it should be read

The server sends your text to the model as written. Numbers, currency, dates, URLs, Markdown, emoji and bracketed text such as `[pause]` all reach the model unchanged, and the model reads them on its own. For most sentences that is what you want. Write currencies, phone numbers, addresses and abbreviations the way they are normally written, and the model reads them the way they are normally pronounced. Where the reading matters, make it explicit.

* **Use full sentences with punctuation.** The model takes pace and tone from it. Text with no punctuation may not be read in the sentences you expect, and a question mark, an exclamation mark or a comma each change the delivery.
* **Spell out a reading the model could get wrong.** Put dashes between digits to read a number digit by digit, as in a card number, and write it with commas to read it as a whole number. Put dashes between the letters of an uncommon acronym. Write out a date or an amount in words when the written form is ambiguous.
* **Remove anything that should not be spoken.** Markup, citations and stage directions are read aloud. `[pause]` is text like any other, not a pause instruction.
* **Send complete sentences per request.** A request that ends mid-sentence is spoken on its own, without the context of the sentence it belongs to, which is the same reason not to flush per token.

See the [Prompting guide](https://assemblyai.com/docs/tts/prompting-guide.md) for the full normalization table and the punctuation the model responds to.

## Set the language of the text

`language` defaults to `english` and decides which model the session runs on. Set it to the language of the text you send, and choose a voice recorded in that language: each voice speaks the language of its recording. Text sent under the wrong language setting, or with a voice of another language, comes out mispronounced or garbled. Read `configuration.voice_id` and `configuration.language` back from `Begin` to confirm what the session resolved, defaults included. See [Voices](https://assemblyai.com/docs/tts/voices.md).

## Related

* [Message frames](https://assemblyai.com/docs/tts/message-frames.md): when the server starts speaking a `Generate`, and what `Flush`, `Cancel` and `KeepAlive` do.
* [Limits](https://assemblyai.com/docs/tts/limits.md): text, session and rate limits, and how each one reports a breach.
* [Prompting guide](https://assemblyai.com/docs/tts/prompting-guide.md): pacing, tone and normalization.
* [WebSocket API basics](https://assemblyai.com/docs/tts/connect-parameters.md#audio-formats): the encodings and sample rates, and playing raw PCM from `Begin`'s sample rate.
* [OpenAI compatibility](https://assemblyai.com/docs/tts/openai-compatibility.md#keep-connections-between-replies): keeping connections with the OpenAI SDKs, Pipecat and LiveKit.
* [HTTP API](https://assemblyai.com/docs/tts/api-reference/http-api/overview.md): streamed and buffered responses, audio formats and `Server-Timing`.