> ## Documentation Index
> Fetch the complete documentation index at: https://assemblyai.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Word boundaries

> Per-word timings for synthesized speech, for highlighting text in sync with playback.

Word boundaries tell you when each word is spoken in the audio being streamed to you. The usual reason to want them is karaoke-style highlighting: following along in the source text while the speech plays.

They are on by default on the wire. Pass `word_boundaries=false` at connect time to turn them off. The SDKs turn them off unless you ask for them; see [In the SDKs](#in-the-sdks). On the HTTP API they come from a route of their own; see [Over the HTTP API](#over-the-http-api).

## The frame

You receive at most one `WordBoundaries` frame per request (each `Flush` concludes one request), and it covers the spoken words in that request's text, however long the request is. This one is for a request whose text began `Hello there, friend,`:

```json theme={null}
{
  "type": "WordBoundaries",
  "flush_id": 0,
  "audio_start_ms": 0,
  "words": [
    { "word": "Hello", "start": 265, "end": 530, "confidence": 0.4791773557662964 },
    { "word": "there,", "start": 546, "end": 712, "confidence": 0.9253882169723511 },
    { "word": "friend,", "start": 845, "end": 1126, "confidence": 0.8478114008903503 }
  ]
}
```

* `flush_id` is the request the words belong to. It is the same id that the request's `FlushDone` carries.
* `audio_start_ms` is where this request's audio begins on the session's audio timeline (see below). It is `0` for the first request.
* `words` has one entry per whitespace-delimited token of the request's text, in the order you sent them. Each entry has four keys:
  * `word`: the word as it appeared in your text.
  * `start` and `end`: when the word is spoken, in milliseconds.
  * `confidence`: a number from `0` to `1` for that word's timing.

All times are integer milliseconds and do not depend on the session's `sample_rate` or `encoding`.

### What `word` contains

`word` is a whitespace-delimited token from your text, exactly as you sent it, with its punctuation still attached: `there,`, `today?`, `dog.`. The entries are one to one with the tokens you get by splitting the request's text on whitespace, in the same order, so the `n`th entry is the `n`th token. Non-ASCII text is returned as is.

A token with no sound of its own, such as the dash in `Hello - world` or the `?` in the French `ça va ?`, still gets an entry. It has `start` equal to `end` and a `confidence` of `0`, and sits at the edge of the word next to it: an opening mark such as `¿` or `(` at the start of the word after it, any other token at the end of the word before it. `Bonjour, ça va ?` gives four entries, the last one `?` at the end of `va`. A token in square brackets, such as `[laughs]` or `[pause]`, also gets a zero-length entry, placed the same way. Such a tag reaches the model as written: `[pause]` is not a pause instruction (see [How text is read](https://assemblyai.com/docs/tts/prompting-guide.md#how-text-is-read)). Skip zero-length entries when you highlight.

A token the model reads as several words, such as `$1,234,567`, gets one entry that spans all of them, several seconds long in that case. Markup stays in `word` as you sent it: `**Hello**` is one entry, timed over what the model reads for it, markup characters included, because the text is read as written. An emoji standing alone between spaces gets a zero-length entry. Timing follows your text, not the spoken form, so numbers, symbols, URLs and tags can get stretched or zero-length spans with a low `confidence`. See [How text is read](https://assemblyai.com/docs/tts/prompting-guide.md#how-text-is-read).

A frame never lists words the audio does not contain. When the audio of a request leaves part of its text unspoken, the request gets no frame at all.

## Times are on the session timeline

`start` and `end` are positions on the session's continuous output-audio timeline: the audio delivered since the session began, across every request. They are not relative to the request's own audio. If the first request produced 2800 ms of audio, the second request's frame has `audio_start_ms: 2800` and its first word starts at 2800 or later.

Two consequences:

* **Do not add `audio_start_ms` to `start`.** `start` is already a position in the concatenated session audio. Adding the offset misplaces every highlight after the first request.
* **Subtract it to get a position inside the request's own audio.** `start - audio_start_ms` is where the word falls in that request's `Audio` frames.

The timeline never rewinds. Every byte of audio delivered to you holds its place on it, including the audio of a request you then cancelled and the short tail that arrives between your `Cancel` and the server's `Cancelled`, so the next request's `audio_start_ms` counts audio you may have dropped and never played. That is why the mapping to your player's clock below anchors on each request rather than on the session.

## When it arrives

Each `WordBoundaries` frame is sent after the `FlushDone` for the same `flush_id`, usually within tens of milliseconds. When you have several requests in flight it often lands after the next request's first `Audio` frames have already arrived. Match frames to requests by `flush_id`, never by arrival order. A real sequence with three pipelined requests:

```text theme={null}
Begin | Audio(0) x43 | FlushDone(0) | Audio(1) x2 | WordBoundaries(0) | Audio(1) x37 | FlushDone(1)
| Audio(2) x3 | WordBoundaries(1) | Audio(2) x42 | FlushDone(2) | WordBoundaries(2) | Termination | close 1000
```

Nothing arrives after `Termination`. `Terminate` waits for pending word boundaries before it ends the session, so you can send it right after your last `Flush` and still receive that request's frame. Text you never flushed is flushed for you by `Terminate` and gets a frame of its own. The same drain runs when the session reaches the `expires_at` time announced in `Begin`.

`FlushDone` means the request's audio has been delivered to the socket, not that it has finished playing. Synthesis runs several times faster than real time, so when the `WordBoundaries` frame arrives the whole request is usually buffered on your side and its first words may already be playing. Schedule highlights against your audio clock instead of reacting to each frame as it arrives, and apply a highlight immediately when its time has already passed.

You can print the frames yourself. The program below sends two requests and prints each `FlushDone` and `WordBoundaries` as it arrives, with every word's position on the session timeline and inside its own request. Set `ASSEMBLYAI_API_KEY` and run it with the `websockets` package (14 or newer):

```python theme={null}
import asyncio
import json
import os

import websockets  # pip install -U websockets

URL = "wss://tts.us.assemblyai.com/v1/ws?voice_id=49c443a4-8a1d-5f7e-b70b-98bdbd9823c3&language=english"
API_KEY = os.environ["ASSEMBLYAI_API_KEY"]

REQUESTS = ["Hello there, friend.", "Pack my box with five dozen liquor jugs."]


async def main():
    async with websockets.connect(URL, additional_headers={"Authorization": API_KEY}) as ws:
        first = json.loads(await ws.recv())
        if first["type"] == "Error":
            raise SystemExit(f"{first['error_code']}: {first['error']}")
        if first["configuration"]["word_boundaries"] is not True:
            raise SystemExit("this session will not send WordBoundaries")

        for text in REQUESTS:
            await ws.send(json.dumps({"type": "Generate", "text": text}))
            await ws.send(json.dumps({"type": "Flush"}))
        await ws.send(json.dumps({"type": "Terminate"}))

        async for raw in ws:
            message = json.loads(raw)
            if message["type"] == "FlushDone":
                print(f"FlushDone      flush_id={message['flush_id']} audio_duration_ms={message['audio_duration_ms']}")
            elif message["type"] == "WordBoundaries":
                print(f"WordBoundaries flush_id={message['flush_id']} audio_start_ms={message['audio_start_ms']}")
                for word in message["words"]:
                    in_request = word["start"] - message["audio_start_ms"]
                    print(f"  {word['word']:<10} session {word['start']:>5} to {word['end']:>5} ms   request +{in_request} ms")
            elif message["type"] == "Termination":
                break
            elif message["type"] == "Error":
                raise SystemExit(f"{message['error_code']}: {message['error']}")


asyncio.run(main())
```

## Mapping timings to playback

A word time is a position in the audio, and your player has its own clock. To turn one into the other, remember the player time at which a request's first audio buffer was scheduled and offset from there. In a WebAudio client that is the `AudioContext` time you passed to `source.start()` for the request's first `Audio` frame:

```javascript theme={null}
// Fragment: the scheduling side of a WebAudio client.
const ctx = new AudioContext();  // plays the session's audio
const anchors = new Map();       // flush_id -> AudioContext time at which that request's
                                 // first Audio buffer was scheduled; set it where you enqueue audio
const highlightWord = (word) => console.log("speaking:", word); // your UI hook

function highlight(message) {
  const anchor = anchors.get(message.flush_id);
  if (anchor === undefined) return; // no audio was scheduled for this request
  for (const word of message.words) {
    if (word.start === word.end) continue; // nothing is spoken for this token
    // start is on the session timeline; the anchor is at audio_start_ms on that timeline.
    const at = anchor + (word.start - message.audio_start_ms) / 1000;
    const delay = Math.max(0, (at - ctx.currentTime) * 1000);
    setTimeout(() => highlightWord(word.word), delay);
  }
}
```

Anchoring on each request is what keeps highlights correct after a `Cancel`, when the audio you stopped playing still counts on the server's timeline. If you play everything the server sends as one uninterrupted stream and never cancel, a single anchor at the session's first sample is enough: highlight at `sessionStart + word.start / 1000`.

Clear pending highlight timers together with your scheduled audio when you cancel or start a new request.

## Treat them as best-effort

> [!WARNING]
>
> `WordBoundaries` frames are best-effort. A request can complete, with its
> audio delivered in full, and never get one. Nothing is sent in the frame's
> place, and the audio stream itself is unaffected either way.

You will not get a frame for:

* a request you cancelled with `Cancel`;
* a request containing a stretch the server speaks in one piece whose audio runs past about 32 seconds. The server splits text into such pieces at sentence ends, and inside a sentence that runs long (see [How text is read](https://assemblyai.com/docs/tts/prompting-guide.md#how-text-is-read)), so this is rare. Other requests in the same session still get theirs;
* a `Flush` with nothing to speak, which gets a `FlushDone` with `audio_duration_ms: 0` and no `WordBoundaries`;
* a request whose audio does not cover all of its text;
* a request for which the server could not compute timings, for example after a transient problem during that request; the audio itself is unaffected.

A long request needs no splitting for highlighting: the server times it in parts of whole sentences and sends one frame that lists every token of the request, after its `FlushDone`.

Two consequences worth designing around:

* **Check the `Begin` frame, not your request.** The `configuration` echoed back on `Begin` is the session you actually got. If the service cannot supply word timings for a session, `Begin` echoes `word_boundaries: false` even though you asked for `true`, and `word_boundaries_requested: true` tells you it was the service's choice. Read the echo before enabling highlighting.
* **Don't build a feature that requires them.** Highlighting should degrade to no highlighting, not to a broken player. If your UI cannot work without word timings, this is not the right foundation for it.

```javascript theme={null}
// Fragment. message is the parsed frame in your onmessage handler;
// disableHighlighting is your UI hook.
const disableHighlighting = () => {};

if (message.type === "Begin") {
  const available = message.configuration.word_boundaries === true;
  if (!available) disableHighlighting();
}
```

## In the SDKs

The SDKs open a session with word boundaries off. Turn them on with `word_boundaries=True` on a Python `TTSSession`, or `wordBoundaries: true` on a JavaScript session, and subscribe to the event. Each event carries the frame's fields: `flush_id`, `audio_start_ms` and `words`, each word with `word`, `start`, `end` and `confidence`. It arrives after the request's audio, so keep the audio you played and match the timings to it as above. Neither SDK exposes word boundaries over HTTP; use `/v1/speech/events` directly for that.

**Python SDK**

```python theme={null}
# Fragment. You supply play().
import assemblyai as aai

config = aai.TTSConfig(voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3")  # Audrey


def on_words(event):
    for w in event.words:
        print(f"request {event.flush_id}: {w.word!r} at {w.start - event.audio_start_ms} ms")


with aai.TTSSession(config, word_boundaries=True) as session:
    session.on(aai.TTSEvents.WordBoundaries, on_words)
    for chunk in session.speak("Hi, thanks for calling. How can I help you today?"):
        play(chunk)
```

**JavaScript SDK**

```javascript theme={null}
// Fragment. You supply play().
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const session = client.tts.session({ voiceId: "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", wordBoundaries: true }); // Audrey
session.on("wordBoundaries", (event) => {
  for (const w of event.words) console.log(`request ${event.flush_id}: ${w.word} at ${w.start - event.audio_start_ms} ms`);
});
await session.connect();
for await (const chunk of session.speak("Hi, thanks for calling. How can I help you today?")) play(chunk);
await session.close();
```

## Over the HTTP API

On the HTTP API, [`POST /v1/speech/events`](https://assemblyai.com/docs/tts/api-reference/http-api/stream-speech-events) takes the body of [Create speech](https://assemblyai.com/docs/tts/api-reference/http-api/create-speech.md), without the file formats, and answers [server-sent events](https://html.spec.whatwg.org/multipage/server-sent-events.html) instead of raw audio. They are the frames of the WebSocket API, with the audio in base64 and, after it, the time at which each word is spoken. A stream looks like this, with `Begin.configuration` and `Termination` abridged to the fields this section describes:

```text theme={null}
event: Begin
data: {"type":"Begin","id":"5e8970c3-26f7-4b8e-9c55-6c1c7f0b6d2e","expires_at":1791480600,"configuration":{"voice_id":"49c443a4-8a1d-5f7e-b70b-98bdbd9823c3","language":"english","sample_rate":24000,"encoding":"pcm_s16le","word_boundaries":true,"model":"assemblyai-tts-1","container":"raw","bit_rate":null,"transport":"http"}}

event: Audio
data: {"type":"Audio","audio":"AAAAAP7/AQD+/wIA","flush_id":0}

event: FlushDone
data: {"type":"FlushDone","flush_id":0,"audio_duration_ms":1440}

event: WordBoundaries
data: {"type":"WordBoundaries","flush_id":0,"audio_start_ms":0,"words":[{"word":"Hello","start":265,"end":530,"confidence":0.48},{"word":"there,","start":546,"end":712,"confidence":0.93},{"word":"friend.","start":845,"end":1126,"confidence":0.85}]}

event: Termination
data: {"type":"Termination","session_duration_seconds":1,"total_input_char_length":20,"audio_duration_seconds":1.44,"billed_characters":20,"word_boundaries_status":"complete"}
```

* A successful stream is `Begin`, one or more `Audio`, `FlushDone`, usually one `WordBoundaries`, then `Termination`. As on `/v1/speech`, the status waits for the first audio, and `Begin` arrives with it.
* `Audio` carries base64 of raw `pcm_s16le`, `pcm_mulaw` or `pcm_alaw` audio, about 80 ms per event. MP3, WAV, `bit_rate` and `"stream": false` are refused on this route, because the word times are positions in that raw audio.
* `WordBoundaries` has one entry per whitespace-delimited token of your text, in order, with its punctuation, and `start` and `end` in milliseconds from the first sample. It follows the rules above: best effort, with zero-length entries for tokens that make no sound. `Termination.word_boundaries_status` says whether it was sent: `complete`, `none` when the timings could not be computed, or `disabled` when you sent `"word_boundaries": false`. `Termination.billed_characters` is the request's final billed count.
* A failure after the first audio sends one `Error` event, with the fields of the WebSocket `Error` frame plus `code`, `message`, `request_id` and `status`, and the stream ends. A stream that ends with neither `Termination` nor `Error` was cut, by the network or by a server failure: send the request again.
* While nothing is sent for 15 seconds, a `: keepalive` comment arrives. Skip lines that start with `:`, and ignore event types and fields you do not know: `Begin.configuration` and `Termination` carry more fields than this section describes.

```python theme={null}
import base64
import json
import os

import httpx

URL = "https://tts.us.assemblyai.com/v1/speech/events"
HEADERS = {"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}
TEXT = "Hello there, friend. The weather is lovely today."

audio = bytearray()
words = []
ended = False
with httpx.stream(
    "POST", URL, headers=HEADERS, json={"text": TEXT}, timeout=httpx.Timeout(5.0, read=30.0)
) as response:
    if response.status_code != 200:
        response.read()
        raise SystemExit(f"{response.status_code}: {response.text[:300]}")
    event = None
    for line in response.iter_lines():
        if line.startswith(":"):
            continue  # keepalive
        if line.startswith("event: "):
            event = line[len("event: "):]
        elif line.startswith("data: "):
            data = json.loads(line[len("data: "):])
            if event == "Audio":
                audio += base64.b64decode(data["audio"])
            elif event == "WordBoundaries":
                words = data["words"]
            elif event == "Termination":
                ended = True
            elif event == "Error":
                raise SystemExit(f"{data['code']}: {data['message']} ({data['request_id']})")

if not ended:
    raise SystemExit("the stream was cut before its last event")
with open("out.pcm", "wb") as f:
    f.write(audio)
for word in words:
    print(f"{word['start']:>6} ms  {word['word']}")
```

## Related

* [WebSocket API basics](https://assemblyai.com/docs/tts/connect-parameters.md): reading `word_boundaries` back from the `Begin` echo.
* [Message frames](https://assemblyai.com/docs/tts/message-frames.md): every frame, and where `WordBoundaries` sits in the order.