TTS word timestamps: build read-along highlighting with word boundaries

Karaoke-style highlighting looks simple: the agent speaks, and the word it's saying lights up. Then you build it. The first sentence lines up. The second drifts. After the user interrupts, every highlight lands on the wrong word.

The cause is almost always the same mistake, and it's a timing-model mistake, not a rendering one. This guide walks through how AssemblyAI's Text-to-Speech API reports word timings, the one rule that keeps highlights in sync across requests and interruptions, and complete code for read-along highlighting and live captions in the browser.

Word boundaries are part of AssemblyAI-TTS-1, the text-to-speech model built for voice agents. If you haven't opened a session yet, start with how to stream text to speech in Python over a WebSocket, then come back.

What word boundaries are, and the frame they arrive in

Word boundaries tell you when each word is spoken in the audio being streamed to you. You get them in a WordBoundaries frame, at most one per request, where a request is everything you sent before a Flush:

{
  "type": "WordBoundaries",
  "flush_id": 0,
  "audio_start_ms": 0,
  "words": [
    { "word": "Hello", "start": 265, "end": 530, "confidence": 0.4791773557662964 },
    { "word": "there,", "start": 546, "end": 712, "confidence": 0.9253882169723511 },
    { "word": "friend,", "start": 845, "end": 1126, "confidence": 0.8478114008903503 }
  ]
}

Three properties make the frame easy to map back to your text:

• One entry per token, in order. Split the request's text on whitespace and the nth entry is the nth token. That's why highlighting by index works.
• Silent tokens still get an entry. A dash, a lone ?, a bracketed tag like [laughs] or a lone emoji gets a zero-length entry (start equals end, confidence 0). Skip those when you highlight.
• Times don't depend on audio format. They're the same whether the session streams 24 kHz PCM or 8 kHz μ-law.

A token the model reads as several words, like $1,234,567, gets one long entry. Markup stays in word as you sent it, so **Hello** is one entry timed over the spoken "Hello".

Turn them on

Word boundaries are on by default on the raw WebSocket and off by default in both SDKs.

Raw WebSocket

Connect to wss://tts.us.assemblyai.com/v1/ws and you'll get frames unless you pass word_boundaries=false. This program from the docs sends two requests and prints each word's position on the session timeline and inside its own request:

import asyncio
import json
import os

import websockets  # pip install -U websockets

URL = "wss://tts.us.assemblyai.com/v1/ws?voice_id=49c443a4-8a1d-5f7e-b70b-98bdbd9823c3&language=english"
API_KEY = os.environ["ASSEMBLYAI_API_KEY"]

REQUESTS = ["Hello there, friend.", "Pack my box with five dozen liquor jugs."]


async def main():
    async with websockets.connect(URL, additional_headers={"Authorization": API_KEY}) as ws:
        first = json.loads(await ws.recv())
        if first["type"] == "Error":
            raise SystemExit(f"{first['error_code']}: {first['error']}")
        if first["configuration"]["word_boundaries"] is not True:
            raise SystemExit("this session will not send WordBoundaries")

        for text in REQUESTS:
            await ws.send(json.dumps({"type": "Generate", "text": text}))
            await ws.send(json.dumps({"type": "Flush"}))
        await ws.send(json.dumps({"type": "Terminate"}))

        async for raw in ws:
            message = json.loads(raw)
            if message["type"] == "FlushDone":
                print(f"FlushDone      flush_id={message['flush_id']} audio_duration_ms={message['audio_duration_ms']}")
            elif message["type"] == "WordBoundaries":
                print(f"WordBoundaries flush_id={message['flush_id']} audio_start_ms={message['audio_start_ms']}")
                for word in message["words"]:
                    in_request = word["start"] - message["audio_start_ms"]
                    print(f"  {word['word']:<10} session {word['start']:>5} to {word['end']:>5} ms   request +{in_request} ms")
            elif message["type"] == "Termination":
                break
            elif message["type"] == "Error":
                raise SystemExit(f"{message['error_code']}: {message['error']}")


asyncio.run(main())

Run it once. The second request's audio_start_ms equals the first request's audio length, and its words start there. That's the timing model in one screen of output.

Python SDK

Pass word_boundaries=True to TTSSession and register a handler for TTSEvents.WordBoundaries:

import os

import assemblyai as aai

config = aai.TTSConfig(voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", language="english")  # Audrey


def on_words(event):
    for w in event.words:
        print(f"request {event.flush_id}: {w.word!r} at {w.start - event.audio_start_ms} ms")


with aai.TTSSession(config, api_key=os.environ["ASSEMBLYAI_API_KEY"], word_boundaries=True) as session:
    if session.begin.configuration.get("word_boundaries") is not True:
        print("word timings unavailable for this session; play audio without highlighting")
    session.on(aai.TTSEvents.WordBoundaries, on_words)
    with open("reply.pcm", "wb") as out:
        for chunk in session.speak("Hi, thanks for calling. How can I help you today?"):
            out.write(chunk)  # raw 24 kHz 16-bit mono PCM

JavaScript SDK

Pass wordBoundaries: true to the session and listen for wordBoundaries:

import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const session = client.tts.session({
  voiceId: "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", // Audrey
  wordBoundaries: true,
});
session.on("wordBoundaries", (event) => {
  for (const w of event.words) {
    console.log(`request ${event.flush_id}: ${w.word} at ${w.start - event.audio_start_ms} ms`);
  }
});

const begin = await session.connect();
console.log("word timings:", begin.configuration.word_boundaries === true);
for await (const chunk of session.speak("Hi, thanks for calling. How can I help you today?")) {
  // chunk is raw 24 kHz 16-bit mono PCM; hand it to your player
}
await session.close();

Session-timeline times: the one mistake to avoid

Here's the mistake from the intro. start and end are positions on the session's continuous audio timeline: all the audio delivered since the session began, across every request. They aren't relative to the request's own audio.

So if the first request produced 2,800 ms of audio, the second request's frame has audio_start_ms: 2800, and its first word starts at 2,800 ms or later. Two rules follow:

1. Never add audio_start_ms to start. start is already on the session timeline. Adding the offset double-counts it, and every highlight after the first request lands late.
2. Subtract it to get a position inside the request. start - audio_start_ms is where the word falls in that request's own audio.

There's a second trap behind the first. The timeline never rewinds. Audio from a request you cancelled still holds its place on it, including the short tail that arrives between your Cancel and the server's Cancelled. So after a barge-in, the session timeline counts audio your user never heard.

That's why the reliable approach anchors on each request:

• When you schedule a request's first audio buffer, record the player time it will start at, keyed by flush_id.
• When that request's WordBoundaries frame arrives, place each word at anchor + (start - audio_start_ms).

Anchored this way, cancelled audio, gaps between replies and pauses in playback all stop mattering.

Timing also matters for when the frame arrives. Each WordBoundaries frame comes after the FlushDone for the same flush_id, usually within tens of milliseconds. With several requests in flight it often lands after the next request's first audio. Match frames to requests by flush_id, never by arrival order.

Build read-along highlighting in the browser

Now the full build: a page that renders the text as spans, plays the speech through WebAudio, and highlights each word as it's spoken. It has three parts:

1. A small server route that mints a temporary token, so your API key never reaches the page.
2. A session opened in the browser with that token.
3. A scheduler that anchors each request and times highlights against the AudioContext clock.

Mint a temporary token on your server

The browser opens the session with a short-lived token that your server mints. This Node server uses the SDK's createTemporaryToken():

import http from "node:http";
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

http
  .createServer(async (req, res) => {
    if (req.url !== "/tts-token") {
      res.writeHead(404).end();
      return;
    }
    // In production, check that the caller is a signed-in user before minting.
    const token = await client.tts.createTemporaryToken({ expires_in_seconds: 60 });
    res.writeHead(200, { "Content-Type": "application/json" });
    res.end(JSON.stringify({ token }));
  })
  .listen(3000);

Mint a new token for every connection, reconnects included, and keep expires_in_seconds short. See authenticate with a temporary token for the details.

The read-along page

This module runs in the page, bundled with a tool such as Vite. It imports TextToSpeechSession directly, so there's no client and no API key in the browser:

import { TextToSpeechSession } from "assemblyai";

const SAMPLE_RATE = 24000;
const ctx = new AudioContext({ sampleRate: SAMPLE_RATE });

const anchors = new Map(); // flush_id -> AudioContext time of the request's first sample
const spansById = new Map(); // flush_id -> the <span> of each token in that request
const early = new Map(); // WordBoundaries frames that arrived before their anchor
const timers = new Set();
const sources = new Set();
let playhead = 0;
let leftover = null; // an odd trailing byte carried to the next chunk
let highlighting = true;

// Play one chunk of raw 16-bit little-endian PCM and return its start time.
function enqueue(chunk) {
  let bytes = chunk;
  if (leftover) {
    bytes = new Uint8Array(chunk.length + 1);
    bytes.set(leftover);
    bytes.set(chunk, 1);
    leftover = null;
  }
  if (bytes.length % 2 === 1) {
    leftover = bytes.slice(-1);
    bytes = bytes.subarray(0, bytes.length - 1);
  }
  const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
  const buffer = ctx.createBuffer(1, bytes.length / 2, SAMPLE_RATE);
  const channel = buffer.getChannelData(0);
  for (let i = 0; i < channel.length; i++) channel[i] = view.getInt16(i * 2, true) / 32768;

  const source = ctx.createBufferSource();
  source.buffer = buffer;
  source.connect(ctx.destination);
  const at = Math.max(playhead, ctx.currentTime);
  source.start(at);
  sources.add(source);
  source.onended = () => sources.delete(source);
  playhead = at + buffer.duration;
  return at;
}

function scheduleHighlights(event) {
  const anchor = anchors.get(event.flush_id);
  const spans = spansById.get(event.flush_id);
  if (anchor === undefined || !spans) {
    early.set(event.flush_id, event); // its first audio isn't scheduled yet
    return;
  }
  event.words.forEach((word, i) => {
    if (word.start === word.end) return; // nothing is spoken for this token
    const at = anchor + (word.start - event.audio_start_ms) / 1000;
    const delay = Math.max(0, (at - ctx.currentTime) * 1000);
    const timer = setTimeout(() => {
      timers.delete(timer);
      for (const span of spans) span.classList.remove("speaking");
      spans[i].classList.add("speaking");
    }, delay);
    timers.add(timer);
  });
}

export async function openSession() {
  const { token } = await (await fetch("/tts-token")).json();
  const session = new TextToSpeechSession({
    token,
    voiceId: "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", // Audrey
    language: "english",
    wordBoundaries: true,
  });
  session.on("wordBoundaries", (event) => {
    if (highlighting) scheduleHighlights(event);
  });
  const begin = await session.connect();
  // Best effort: the service can decline word timings for a session.
  highlighting = begin.configuration.word_boundaries === true;
  return session;
}

export async function readAloud(session, container, text) {
  await ctx.resume();
  // Split on spaces and send the tokens joined by single spaces,
  // so the nth word entry is always the nth span.
  const tokens = text.split(" ").filter((token) => token.length > 0);
  const spans = tokens.map((token) => {
    const span = document.createElement("span");
    span.textContent = token + " ";
    return span;
  });
  container.replaceChildren(...spans);

  session.generate(tokens.join(" "));
  const flushId = session.flush();
  spansById.set(flushId, spans);

  let first = true;
  for await (const chunk of session.audio(flushId)) {
    const at = enqueue(chunk);
    if (first) {
      first = false;
      anchors.set(flushId, at);
      if (early.has(flushId)) {
        scheduleHighlights(early.get(flushId));
        early.delete(flushId);
      }
    }
  }
}

export function stopReading(session) {
  session.cancel();
  for (const timer of timers) clearTimeout(timer);
  timers.clear();
  for (const source of sources) source.stop();
  sources.clear();
  playhead = ctx.currentTime;
  leftover = null;
}

Add one CSS rule, such as .speaking { background: #fde68a; }, and wire the functions to your UI: open the session when the page loads, call readAloud() for each passage, and call stopReading() when the user interrupts.

A few design choices are worth calling out:

• Highlighting by index. Because the page sends exactly the tokens it rendered, joined by single spaces, entry i in words is always span i. No string matching, no fuzzy alignment. The server splits on any whitespace, so replace line breaks and tabs with spaces before you call readAloud().
• Scheduling against the audio clock. Synthesis runs several times faster than real time, so by the time a WordBoundaries frame arrives, the whole request is usually buffered and its first words may already be playing. The scheduler computes each word's player time and fires immediately for any word whose time has passed.
• The early-frame guard. A frame almost always arrives after its audio is scheduled. The early map covers the rare case where it doesn't.
• Barge-in. stopReading() cancels the request, stops queued audio and clears pending highlight timers together, so the next request starts on a fresh anchor.

If you want highlighting to start from the first sentence of a long passage, send each sentence as its own request (generate() plus flush() per sentence). A request's timings arrive right after its audio, so one long request only gets its timings once all of its audio has been sent. Each request is spoken on its own, which adds a short pause between sentences, so it's a trade-off to make on purpose.

Sync live captions from the same timings

The same frames can drive a caption line for an agent's speech: a short line of text under a video call tile, or an accessibility caption in a kiosk. Instead of highlighting single words, group them into short cues and show each cue for its span:

// Group one WordBoundaries frame into caption cues of a few words each.
// Times stay on the session timeline; anchor them the same way as highlights.
export function toCues(event, { maxWords = 7, maxGapMs = 350 } = {}) {
  const cues = [];
  let current = null;
  for (const word of event.words) {
    if (word.start === word.end) {
      if (current) current.text += " " + word.word; // keep punctuation tokens visible
      continue;
    }
    const gap = current ? word.start - current.end : 0;
    const full = current && current.count >= maxWords;
    if (!current || full || gap > maxGapMs) {
      current = { text: word.word, start: word.start, end: word.end, count: 1 };
      cues.push(current);
    } else {
      current.text += " " + word.word;
      current.end = word.end;
      current.count += 1;
    }
    if (/[.!?]$/.test(word.word)) current.count = maxWords; // end the cue at a sentence end
  }
  return cues;
}

Schedule each cue exactly like a word: show it at anchor + (cue.start - event.audio_start_ms) and clear it at the matching end. Because captions come from the request's own timings, they stay in step after interruptions for the same reason the highlights do.

If what you need is captions for recorded audio, that's a speech-to-text job. See how to transcribe audio with timestamps and our guide to subtitle file formats.

Word timings over HTTP

Don't need a WebSocket? POST https://tts.us.assemblyai.com/v1/speech/events takes the regular speech endpoint's request body, minus the file-format options, and answers with server-sent events: Begin, one or more Audio events (base64, about 80 ms each), FlushDone, usually one WordBoundaries, then Termination.

import base64
import json
import os

import httpx

URL = "https://tts.us.assemblyai.com/v1/speech/events"
HEADERS = {"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}
TEXT = "Hello there, friend. The weather is lovely today."

audio = bytearray()
words = []
ended = False
with httpx.stream(
    "POST", URL, headers=HEADERS, json={"text": TEXT}, timeout=httpx.Timeout(5.0, read=30.0)
) as response:
    if response.status_code != 200:
        response.read()
        raise SystemExit(f"{response.status_code}: {response.text[:300]}")
    event = None
    for line in response.iter_lines():
        if line.startswith(":"):
            continue  # keepalive
        if line.startswith("event: "):
            event = line[len("event: "):]
        elif line.startswith("data: "):
            data = json.loads(line[len("data: "):])
            if event == "Audio":
                audio += base64.b64decode(data["audio"])
            elif event == "WordBoundaries":
                words = data["words"]
            elif event == "Termination":
                ended = True
            elif event == "Error":
                raise SystemExit(f"{data['code']}: {data['message']} ({data['request_id']})")

if not ended:
    raise SystemExit("the stream was cut before its last event")
with open("out.pcm", "wb") as f:
    f.write(audio)
for word in words:
    print(f"{word['start']:>6} ms  {word['word']}")

The HTTP route has its own rules, listed in the stream speech events reference:

• Raw audio only. pcm_s16le, pcm_mulaw or pcm_alaw. MP3, WAV containers, bit_rate and "stream": false are refused, because the word times are positions in that raw audio.
• One request per call, up to 5,000 characters of text. Times count from the request's first sample.
• Check the status. Termination.word_boundaries_status is complete, none (timings couldn't be computed) or disabled (you sent "word_boundaries": false).
• Skip keepalives. A : keepalive comment arrives after 15 quiet seconds.
• No SDK wrapper. Neither SDK exposes HTTP word boundaries, so call the route directly. The OpenAI-compatible route doesn't return word timings at all.

Use HTTP when you're generating a passage once, for example a read-along for a help article, and the WebSocket when the text arrives turn by turn from an agent.

Design for best-effort delivery

Word boundaries are best-effort, and the docs say so plainly. A request can finish, with its audio delivered in full, and never get a frame. Nothing is sent in its place, and the audio is unaffected. You won't get a frame for:

• a request you cancelled;
• a request with a stretch the server speaks in one piece whose audio runs past about 32 seconds (rare, since the server splits long text at sentence ends);
• a Flush with nothing to speak;
• a request whose audio doesn't cover all of its text;
• a request where the server couldn't compute timings, for example after a transient problem.

Design around that from the start:

• Read the Begin echo, not your request. If the service can't supply timings for a session, Begin echoes word_boundaries: false even though you asked for true. The read-along page above turns highlighting off in that case.
• Degrade to plain playback. If a frame never arrives for a request, the passage plays without highlights and the next request highlights normally. That's the right failure mode.
• Don't build a feature that requires them. A reading app can treat highlighting as an enhancement. A compliance flow that must prove which words played needs a different foundation.
• Treat low confidence as a hint. Numbers, symbols, URLs and tags can get stretched spans with low confidence, because timing follows your text, not the spoken form. A UI that dims or skips low-confidence words looks smoother than one that jumps.

Turning word boundaries on doesn't slow anything down: a request's first audio arrives as soon with them as without. For more on what affects the wait before the first audio, read TTS latency explained.

Where word timings fit

Word boundaries turn TTS output from an opaque audio stream into something your UI can follow. Read-along text, live captions, a transcript that knows exactly what the agent said before it was interrupted: they're all the same frame, scheduled the same way. Get the anchor right once, per request and keyed by flush_id, and every one of those features stays in sync.

If you're new to TTS, what is text-to-speech covers the basics, and the word boundaries docs have the full frame reference.

Frequently Asked Questions (FAQs)

Does AssemblyAI text-to-speech return word timestamps?

Yes. AssemblyAI TTS returns per-word timings, called word boundaries, in a WordBoundaries frame for each request: every token of your text with its start and end in milliseconds and a timing confidence. They're on by default on the raw WebSocket, off by default in the Python and JavaScript SDKs, and available over HTTP through POST /v1/speech/events.

How do I highlight words in sync with text-to-speech audio?

Record the player time at which each request's first audio buffer starts, keyed by flush_id. When that request's WordBoundaries frame arrives, schedule each word at the anchor plus start - audio_start_ms, skip zero-length entries, and clear pending timers when you cancel. Highlighting by index works because the nth entry is the nth whitespace-split token of your text.

Why do my TTS highlights drift after the first sentence?

You're most likely adding audio_start_ms to start. Word times are already on the session's continuous audio timeline, so adding the offset counts it twice. Subtract audio_start_ms to get a position inside the request, and anchor each request to your player's clock.

Are TTS word boundaries guaranteed for every request?

No. They're best-effort: there's no frame for a cancelled request, an empty flush, or a request whose timings couldn't be computed, and the audio is unaffected either way. Check word_boundaries in the Begin frame's configuration before turning highlighting on, and let playback continue without highlights when a frame is missing.

Can I get word timings from the OpenAI-compatible TTS route?

No. The OpenAI-compatible route doesn't return word timings. Use the WebSocket API with word boundaries enabled, or POST /v1/speech/events over HTTP, which streams base64 audio plus a WordBoundaries event for raw PCM, μ-law or A-law output.

Do word boundaries add latency to text-to-speech?

No. A request's first audio arrives as soon with word boundaries as without them. The frame itself comes after the request's audio, so to highlight a long passage from its first sentence, send each sentence as its own request.