Per-word timings for synthesized speech, for highlighting text in sync with playback.
Word boundaries tell you when each word is spoken in the audio being streamed to you. The usual reason to want them is karaoke-style highlighting: following along in the source text while the speech plays.They are on by default on the wire. Pass word_boundaries=false at connect time to turn them off. The SDKs turn them off unless you ask for them; see In the SDKs. On the HTTP API they come from a route of their own; see Over the HTTP API.
You receive at most one WordBoundaries frame per request (each Flush concludes one request), and it covers the spoken words in that request’s text, however long the request is. This one is for a request whose text began Hello there, friend,:
word is a whitespace-delimited token from your text, exactly as you sent it, with its punctuation still attached: there,, today?, dog.. The entries are one to one with the tokens you get by splitting the request’s text on whitespace, in the same order, so the nth entry is the nth token. Non-ASCII text is returned as is.A token with no sound of its own, such as the dash in Hello - world or the ? in the French ça va ?, still gets an entry. It has start equal to end and a confidence of 0, and sits at the edge of the word next to it: an opening mark such as ¿ or ( at the start of the word after it, any other token at the end of the word before it. Bonjour, ça va ? gives four entries, the last one ? at the end of va. A token in square brackets, such as [laughs] or [pause], also gets a zero-length entry, placed the same way. Such a tag reaches the model as written: [pause] is not a pause instruction (see How text is read). Skip zero-length entries when you highlight.A token the model reads as several words, such as $1,234,567, gets one entry that spans all of them, several seconds long in that case. Markup stays in word as you sent it: **Hello** is one entry, timed over what the model reads for it, markup characters included, because the text is read as written. An emoji standing alone between spaces gets a zero-length entry. Timing follows your text, not the spoken form, so numbers, symbols, URLs and tags can get stretched or zero-length spans with a low confidence. See How text is read.A frame never lists words the audio does not contain. When the audio of a request leaves part of its text unspoken, the request gets no frame at all.
start and end are positions on the session’s continuous output-audio timeline: the audio delivered since the session began, across every request. They are not relative to the request’s own audio. If the first request produced 2800 ms of audio, the second request’s frame has audio_start_ms: 2800 and its first word starts at 2800 or later.Two consequences:
Do not add audio_start_ms to start.start is already a position in the concatenated session audio. Adding the offset misplaces every highlight after the first request.
Subtract it to get a position inside the request’s own audio.start - audio_start_ms is where the word falls in that request’s Audio frames.
The timeline never rewinds. Every byte of audio delivered to you holds its place on it, including the audio of a request you then cancelled and the short tail that arrives between your Cancel and the server’s Cancelled, so the next request’s audio_start_ms counts audio you may have dropped and never played. That is why the mapping to your player’s clock below anchors on each request rather than on the session.
Each WordBoundaries frame is sent after the FlushDone for the same flush_id, usually within tens of milliseconds. When you have several requests in flight it often lands after the next request’s first Audio frames have already arrived. Match frames to requests by flush_id, never by arrival order. A real sequence with three pipelined requests:
Nothing arrives after Termination. Terminate waits for pending word boundaries before it ends the session, so you can send it right after your last Flush and still receive that request’s frame. Text you never flushed is flushed for you by Terminate and gets a frame of its own. The same drain runs when the session reaches the expires_at time announced in Begin.FlushDone means the request’s audio has been delivered to the socket, not that it has finished playing. Synthesis runs several times faster than real time, so when the WordBoundaries frame arrives the whole request is usually buffered on your side and its first words may already be playing. Schedule highlights against your audio clock instead of reacting to each frame as it arrives, and apply a highlight immediately when its time has already passed.You can print the frames yourself. The program below sends two requests and prints each FlushDone and WordBoundaries as it arrives, with every word’s position on the session timeline and inside its own request. Set ASSEMBLYAI_API_KEY and run it with the websockets package (14 or newer):
import asyncioimport jsonimport osimport websockets # pip install -U websocketsURL = "wss://tts.us.assemblyai.com/v1/ws?voice_id=49c443a4-8a1d-5f7e-b70b-98bdbd9823c3&language=english"API_KEY = os.environ["ASSEMBLYAI_API_KEY"]REQUESTS = ["Hello there, friend.", "Pack my box with five dozen liquor jugs."]async def main(): async with websockets.connect(URL, additional_headers={"Authorization": API_KEY}) as ws: first = json.loads(await ws.recv()) if first["type"] == "Error": raise SystemExit(f"{first['error_code']}: {first['error']}") if first["configuration"]["word_boundaries"] is not True: raise SystemExit("this session will not send WordBoundaries") for text in REQUESTS: await ws.send(json.dumps({"type": "Generate", "text": text})) await ws.send(json.dumps({"type": "Flush"})) await ws.send(json.dumps({"type": "Terminate"})) async for raw in ws: message = json.loads(raw) if message["type"] == "FlushDone": print(f"FlushDone flush_id={message['flush_id']} audio_duration_ms={message['audio_duration_ms']}") elif message["type"] == "WordBoundaries": print(f"WordBoundaries flush_id={message['flush_id']} audio_start_ms={message['audio_start_ms']}") for word in message["words"]: in_request = word["start"] - message["audio_start_ms"] print(f" {word['word']:<10} session {word['start']:>5} to {word['end']:>5} ms request +{in_request} ms") elif message["type"] == "Termination": break elif message["type"] == "Error": raise SystemExit(f"{message['error_code']}: {message['error']}")asyncio.run(main())
A word time is a position in the audio, and your player has its own clock. To turn one into the other, remember the player time at which a request’s first audio buffer was scheduled and offset from there. In a WebAudio client that is the AudioContext time you passed to source.start() for the request’s first Audio frame:
// Fragment: the scheduling side of a WebAudio client.const ctx = new AudioContext(); // plays the session's audioconst anchors = new Map(); // flush_id -> AudioContext time at which that request's // first Audio buffer was scheduled; set it where you enqueue audioconst highlightWord = (word) => console.log("speaking:", word); // your UI hookfunction highlight(message) { const anchor = anchors.get(message.flush_id); if (anchor === undefined) return; // no audio was scheduled for this request for (const word of message.words) { if (word.start === word.end) continue; // nothing is spoken for this token // start is on the session timeline; the anchor is at audio_start_ms on that timeline. const at = anchor + (word.start - message.audio_start_ms) / 1000; const delay = Math.max(0, (at - ctx.currentTime) * 1000); setTimeout(() => highlightWord(word.word), delay); }}
Anchoring on each request is what keeps highlights correct after a Cancel, when the audio you stopped playing still counts on the server’s timeline. If you play everything the server sends as one uninterrupted stream and never cancel, a single anchor at the session’s first sample is enough: highlight at sessionStart + word.start / 1000.Clear pending highlight timers together with your scheduled audio when you cancel or start a new request.
WordBoundaries frames are best-effort. A request can complete, with its
audio delivered in full, and never get one. Nothing is sent in the frame’s
place, and the audio stream itself is unaffected either way.
You will not get a frame for:
a request you cancelled with Cancel;
a request containing a stretch the server speaks in one piece whose audio runs past about 32 seconds. The server splits text into such pieces at sentence ends, and inside a sentence that runs long (see How text is read), so this is rare. Other requests in the same session still get theirs;
a Flush with nothing to speak, which gets a FlushDone with audio_duration_ms: 0 and no WordBoundaries;
a request whose audio does not cover all of its text;
a request for which the server could not compute timings, for example after a transient problem during that request; the audio itself is unaffected.
A long request needs no splitting for highlighting: the server times it in parts of whole sentences and sends one frame that lists every token of the request, after its FlushDone.Two consequences worth designing around:
Check the Begin frame, not your request. The configuration echoed back on Begin is the session you actually got. If the service cannot supply word timings for a session, Begin echoes word_boundaries: false even though you asked for true, and word_boundaries_requested: true tells you it was the service’s choice. Read the echo before enabling highlighting.
Don’t build a feature that requires them. Highlighting should degrade to no highlighting, not to a broken player. If your UI cannot work without word timings, this is not the right foundation for it.
// Fragment. message is the parsed frame in your onmessage handler;// disableHighlighting is your UI hook.const disableHighlighting = () => {};if (message.type === "Begin") { const available = message.configuration.word_boundaries === true; if (!available) disableHighlighting();}
The SDKs open a session with word boundaries off. Turn them on with word_boundaries=True on a Python TTSSession, or wordBoundaries: true on a JavaScript session, and subscribe to the event. Each event carries the frame’s fields: flush_id, audio_start_ms and words, each word with word, start, end and confidence. It arrives after the request’s audio, so keep the audio you played and match the timings to it as above. Neither SDK exposes word boundaries over HTTP; use /v1/speech/events directly for that.
Python SDK
JavaScript SDK
# Fragment. You supply play().import assemblyai as aaiconfig = aai.TTSConfig(voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3") # Audreydef on_words(event): for w in event.words: print(f"request {event.flush_id}: {w.word!r} at {w.start - event.audio_start_ms} ms")with aai.TTSSession(config, word_boundaries=True) as session: session.on(aai.TTSEvents.WordBoundaries, on_words) for chunk in session.speak("Hi, thanks for calling. How can I help you today?"): play(chunk)
// Fragment. You supply play().import { AssemblyAI } from "assemblyai";const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });const session = client.tts.session({ voiceId: "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", wordBoundaries: true }); // Audreysession.on("wordBoundaries", (event) => { for (const w of event.words) console.log(`request ${event.flush_id}: ${w.word} at ${w.start - event.audio_start_ms} ms`);});await session.connect();for await (const chunk of session.speak("Hi, thanks for calling. How can I help you today?")) play(chunk);await session.close();
On the HTTP API, POST /v1/speech/events takes the body of Create speech, without the file formats, and answers server-sent events instead of raw audio. They are the frames of the WebSocket API, with the audio in base64 and, after it, the time at which each word is spoken. A stream looks like this, with Begin.configuration and Termination abridged to the fields this section describes:
A successful stream is Begin, one or more Audio, FlushDone, usually one WordBoundaries, then Termination. As on /v1/speech, the status waits for the first audio, and Begin arrives with it.
Audio carries base64 of raw pcm_s16le, pcm_mulaw or pcm_alaw audio, about 80 ms per event. MP3, WAV, bit_rate and "stream": false are refused on this route, because the word times are positions in that raw audio.
WordBoundaries has one entry per whitespace-delimited token of your text, in order, with its punctuation, and start and end in milliseconds from the first sample. It follows the rules above: best effort, with zero-length entries for tokens that make no sound. Termination.word_boundaries_status says whether it was sent: complete, none when the timings could not be computed, or disabled when you sent "word_boundaries": false. Termination.billed_characters is the request’s final billed count.
A failure after the first audio sends one Error event, with the fields of the WebSocket Error frame plus code, message, request_id and status, and the stream ends. A stream that ends with neither Termination nor Error was cut, by the network or by a server failure: send the request again.
While nothing is sent for 15 seconds, a : keepalive comment arrives. Skip lines that start with :, and ignore event types and fields you do not know: Begin.configuration and Termination carry more fields than this section describes.
import base64import jsonimport osimport httpxURL = "https://tts.us.assemblyai.com/v1/speech/events"HEADERS = {"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}TEXT = "Hello there, friend. The weather is lovely today."audio = bytearray()words = []ended = Falsewith httpx.stream( "POST", URL, headers=HEADERS, json={"text": TEXT}, timeout=httpx.Timeout(5.0, read=30.0)) as response: if response.status_code != 200: response.read() raise SystemExit(f"{response.status_code}: {response.text[:300]}") event = None for line in response.iter_lines(): if line.startswith(":"): continue # keepalive if line.startswith("event: "): event = line[len("event: "):] elif line.startswith("data: "): data = json.loads(line[len("data: "):]) if event == "Audio": audio += base64.b64decode(data["audio"]) elif event == "WordBoundaries": words = data["words"] elif event == "Termination": ended = True elif event == "Error": raise SystemExit(f"{data['code']}: {data['message']} ({data['request_id']})")if not ended: raise SystemExit("the stream was cut before its last event")with open("out.pcm", "wb") as f: f.write(audio)for word in words: print(f"{word['start']:>6} ms {word['word']}")
