Quickstart

​

Before you begin

• An API key. Copy it from API Keys and set it once:
  export ASSEMBLYAI_API_KEY=<your-key>
  
• The SDK. Skip this if you only want the curl command. Text to speech is in the Python SDK from release 1.8.0 and in the JavaScript SDK from 4.42.0.
  • Python SDK
  • JavaScript SDK

  pip install -U assemblyai
  

  npm install assemblyai
  

​

Generate your first audio

import os

import assemblyai as aai

tts = aai.TextToSpeech(api_key=os.environ["ASSEMBLYAI_API_KEY"])

response = tts.synthesize(
    "Hi, thanks for calling. How can I help you today?",
    aai.TTSConfig(voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", container=aai.TTSContainer.wav),  # Audrey
)

with open("hello.wav", "wb") as f:
    f.write(response.audio)

print(f"voice {response.voice_id}, model {response.model}, "
      f"{response.sample_rate} Hz {response.encoding}")
print(f"wrote hello.wav: {response.audio_duration_ms} ms of audio, "
      f"{response.billed_characters} characters billed")

import { writeFileSync } from "node:fs";
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const response = await client.tts.synthesize({
  text: "Hi, thanks for calling. How can I help you today?",
  voice_id: "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", // Audrey
  container: "wav",
});

writeFileSync("hello.wav", response.audio);

console.log(`voice ${response.voiceId}, model ${response.model}, ${response.sampleRate} Hz ${response.encoding}`);
console.log(`wrote hello.wav: ${response.audioDurationMs} ms of audio, ${response.billedCharacters} characters billed`);

curl -sS --fail-with-body -D headers.txt https://tts.us.assemblyai.com/v1/speech \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text": "Hi, thanks for calling. How can I help you today?", "voice_id": "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", "container": "wav", "stream": false}' \
  -o hello.wav

HTTP/2 200
content-type: audio/wav
x-request-id: 83c9fae1-e0b0-433a-b97e-26f372778f2a
x-tts-model: assemblyai-tts-1
x-tts-voice-id: 49c443a4-8a1d-5f7e-b70b-98bdbd9823c3
x-tts-language: english
x-tts-encoding: pcm_s16le
x-tts-sample-rate: 24000
x-tts-container: wav
x-tts-billed-characters: 49
x-tts-audio-duration-ms: 3200

voice 49c443a4-8a1d-5f7e-b70b-98bdbd9823c3, model assemblyai-tts-1, 24000 Hz pcm_s16le
wrote hello.wav: 3200 ms of audio, 49 characters billed

• Only the text is required. voice_id names Audrey here by her ID. Leave it out and you get aai-jane, the default voice, in English at 24 kHz 16-bit PCM. Each voice speaks the language it was recorded in; Voices explains how to name one, find its ID and set the language, and the same voice_id takes a voice you cloned in the dashboard.
• The response says what it ran with. The voice, model, language, encoding and sample rate come back as response headers, which the SDKs expose on the result. Read the sample rate from there rather than from your own request when you write a header or set up a player.
• The duration varies between runs. Synthesis is not deterministic, so the same sentence can come back a few hundred milliseconds longer or shorter. Billing counts characters, not seconds.
• The JavaScript SDK uses two spellings. synthesize() and stream() take the API’s own field names, voice_id and sample_rate, while session() takes camelCase options, voiceId and sampleRate. Results are camelCase, and the Begin and Termination frames keep their wire names.

​

Speak a conversation over one session

import os

import assemblyai as aai

config = aai.TTSConfig(voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", language="english")  # Audrey
reply = [
    "Hi, thanks for calling.",
    "You have reached the accounts team.",
    "There is one open invoice from last month.",
    "Would you like to pay it today?",
]

with aai.TTSSession(config, api_key=os.environ["ASSEMBLYAI_API_KEY"]) as session:
    print(f"session {session.begin.id}")
    with open("reply.pcm", "wb") as out:
        for sentence in reply:
            n = 0
            for chunk in session.speak(sentence):  # raw 24 kHz 16-bit mono PCM
                out.write(chunk)
                n += len(chunk)
            print(f"{sentence!r}: {n} bytes")

summary = session.termination  # None if the summary did not arrive in time
if summary:
    print(f"{summary.audio_duration_seconds} s of audio "
          f"for {summary.total_input_char_length} characters")

import { createWriteStream } from "node:fs";
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });
const reply = [
  "Hi, thanks for calling.",
  "You have reached the accounts team.",
  "There is one open invoice from last month.",
  "Would you like to pay it today?",
];

const session = client.tts.session({ voiceId: "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", language: "english" }); // Audrey
const begin = await session.connect();
console.log(`session ${begin.id}`);

const out = createWriteStream("reply.pcm");
for (const sentence of reply) {
  let bytes = 0;
  for await (const chunk of session.speak(sentence)) { // raw 24 kHz 16-bit mono PCM
    out.write(chunk);
    bytes += chunk.byteLength;
  }
  console.log(`'${sentence}': ${bytes} bytes`);
}
out.end();

const summary = await session.close(); // undefined if the summary did not arrive in time
if (summary) {
  console.log(`${summary.audio_duration_seconds} s of audio for ${summary.total_input_char_length} characters`);
}

play -t raw -r 24000 -e signed -b 16 -c 1 reply.pcm           # SoX
ffplay -f s16le -ar 24000 -ac 1 reply.pcm                      # ffmpeg
sox -t raw -r 24000 -e signed -b 16 -c 1 reply.pcm reply.wav   # SoX, to a WAV file

session 285ab483-997d-439d-a207-411b373a4ae1
'Hi, thanks for calling.': 88320 bytes
'You have reached the accounts team.': 99840 bytes
'There is one open invoice from last month.': 134400 bytes
'Would you like to pay it today?': 84480 bytes
8.48 s of audio for 131 characters

• The server speaks first. Connecting waits for the Begin frame, which echoes the configuration the session actually got, with the session id to quote in support requests. A rejected key or parameter, an unknown voice_id included, arrives as an Error frame in its place: the Python with and the JavaScript connect() raise it. See Error handling.
• Each sentence is one request. speak() sends the text in Generate frames, concludes the request with a Flush and yields the audio frames until the request’s FlushDone. The iterator ends with the request, which is what lets the loop count bytes per sentence. See Message frames.
• The audio has no header. A session streams raw codec bytes, 16-bit PCM here, so the file will not play if you hand it to an <audio> element or a Blob. Write a WAV header yourself, or decode the samples and play them as they arrive. Chunk sizes vary and are not aligned to samples, so carry an odd trailing byte over to the next chunk. See Audio formats on WebSocket API basics.
• Leaving the session ends it cleanly. Exiting the Python block and awaiting close() in JavaScript send Terminate, which speaks any text not yet flushed, and wait for the Termination summary: the seconds of audio and the characters of text that reached synthesis.

​

Stream text in as an LLM writes it

# Fragment: continues reply.py. You supply llm_tokens(), user_started_talking() and play().
with aai.TTSSession(config, api_key=os.environ["ASSEMBLYAI_API_KEY"]) as session:
    for token in llm_tokens():
        session.generate(token)  # joined exactly as sent: include the spaces
    flush_id = session.flush()

    for chunk in session.audio(flush_id):
        if user_started_talking():
            session.cancel()  # ends this iteration at once; the server drops the rest
            break
        play(chunk)

// Fragment: continues reply.mjs. You supply llmTokens(), userStartedTalking() and play().
const session = client.tts.session({ voiceId: "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3" });
await session.connect();

for await (const token of llmTokens()) session.generate(token); // joined as sent: include the spaces
const flushId = session.flush();

for await (const chunk of session.audio(flushId)) {
  if (userStartedTalking()) {
    session.cancel(); // ends this iteration at once; the server drops the rest
    break;
  }
  play(chunk);
}
await session.close();

​

Using the WebSocket API directly

import asyncio
import base64
import json
import os
import struct

import websockets

URL = (
    "wss://tts.us.assemblyai.com/v1/ws"
    "?voice_id=49c443a4-8a1d-5f7e-b70b-98bdbd9823c3&language=english"
    "&sample_rate=24000&encoding=pcm_s16le"
)


def wav_header(n_bytes, rate, channels=1, bits=16):
    byte_rate = rate * channels * bits // 8
    return (
        b"RIFF" + struct.pack("<I", 36 + n_bytes) + b"WAVEfmt "
        + struct.pack("<IHHIIHH", 16, 1, channels, rate, byte_rate,
                      channels * bits // 8, bits)
        + b"data" + struct.pack("<I", n_bytes)
    )


async def main():
    # The raw key works; a "Bearer " prefix is accepted too.
    headers = {"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}

    async with websockets.connect(URL, additional_headers=headers) as ws:
        # The server sends the first frame. Begin carries the session
        # configuration; a rejected key or parameter arrives as Error instead.
        first = json.loads(await ws.recv())
        if first["type"] == "Error":
            raise SystemExit(f"{first['error_code']}: {first['error']}")
        config = first["configuration"]
        print(f"voice {config['voice_id']}, model {config['model']}, "
              f"{config['sample_rate']} Hz {config['encoding']}")

        await ws.send(json.dumps({
            "type": "Generate",
            "text": "Hi, thanks for calling. How can I help you today?",
        }))
        await ws.send(json.dumps({"type": "Flush"}))

        pcm = bytearray()
        async for raw in ws:
            message = json.loads(raw)
            kind = message.get("type")

            if kind == "Audio":
                pcm += base64.b64decode(message["audio"])
            elif kind == "FlushDone":
                print(f"flush {message['flush_id']}: "
                      f"{message['audio_duration_ms']} ms of audio")
                await ws.send(json.dumps({"type": "Terminate"}))
            elif kind == "Termination":
                break
            elif kind == "Error":
                raise SystemExit(f"{message['error_code']}: {message['error']}")
            # Other frame types, such as WordBoundaries, are ignored here.

    with open("out.wav", "wb") as f:
        f.write(wav_header(len(pcm), config["sample_rate"]))
        f.write(pcm)

    print(f"wrote out.wav ({len(pcm)} bytes of PCM)")


asyncio.run(main())

• Audio.audio is base64. Each Audio frame is JSON with the codec bytes base64-encoded, under the flush_id of the request it belongs to. flush_id starts at 0, each Flush takes the next id, and ids skip one after a Cancel, so read them from the frames rather than counting.
• WordBoundaries is on by default on the wire. A frame with one entry per word, its start and end in milliseconds and a confidence, follows each FlushDone. Ignore any frame type you do not handle; the protocol may gain frames. The SDKs turn word boundaries off unless you ask for them. See Word boundaries.

​

Where to next

• Optimizing accuracy and latency: the session, text and format habits that get the first audio of each reply out sooner and read as you meant it.
• Choose an interface: the SDKs, the HTTP API, the WebSocket API and the OpenAI-compatible route, and which fits how your text arrives.
• Authenticate with a temporary token: open a session from a browser or an app without shipping your API key.
• Voices: AssemblyAI’s voices and the languages they speak.
• HTTP API basics: streamed and buffered responses, MP3 and telephony formats, errors and limits.
• LiveKit: wire Streaming TTS into a LiveKit voice agent.
• WebSocket API basics and Message frames: authentication, the connect sequence, and every frame in the protocol.
• Limits: text, session and rate limits, and how each one reports a breach.

​

Need help?