OpenAI compatibility

Streaming TTS answers the request shape of the OpenAI speech API, POST /audio/speech, under the base URL https://tts.us.assemblyai.com/v1/openai. Point an OpenAI SDK, or a framework that is built on one, at that base URL with your AssemblyAI API key as its API key, and it speaks with AssemblyAI voices without other changes. This page is for code that already speaks the OpenAI speech API. For a new integration, the Python and JavaScript SDKs and the native LiveKit and Pipecat plugins reach the whole API, word timings and barge-in on one session included. The route is part of the HTTP API and behaves like Create speech: the audio streams as it is synthesized, the status waits for the first audio so every earlier failure is a real HTTP status, a failure after that point cuts the connection, and the same limits and billing apply. Only the request fields and the error body follow OpenAI; Create speech (OpenAI compatible) is the route reference.

Data zone	Base URL
US	`https://tts.us.assemblyai.com/v1/openai`
EU	`https://tts.eu.assemblyai.com/v1/openai`

OpenAI SDKs

Python
JavaScript

import os

import httpx2
from openai import DefaultHttpx2Client, OpenAI

client = OpenAI(
    base_url="https://tts.us.assemblyai.com/v1/openai",
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
    # The SDK's own pool sizes, with idle connections kept for 2 minutes
    # instead of 5 seconds; see "Keep connections between replies" below.
    http_client=DefaultHttpx2Client(
        limits=httpx2.Limits(
            max_connections=1000, max_keepalive_connections=100, keepalive_expiry=120
        )
    ),
)

with client.audio.speech.with_streaming_response.create(
    model="assemblyai-tts-en",
    voice="alloy",
    input="Hello from AssemblyAI.",
    response_format="pcm",
) as response:
    print("request id:", response.headers.get("x-request-id"))
    response.stream_to_file("hello.pcm")

import fs from "node:fs/promises";
import OpenAI from "openai";
import { Agent, fetch } from "undici";

const client = new OpenAI({
  baseURL: "https://tts.us.assemblyai.com/v1/openai",
  apiKey: process.env.ASSEMBLYAI_API_KEY,
  // Idle connections kept for 2 minutes instead of Node's 4 seconds; see
  // "Keep connections between replies" below.
  fetch,
  fetchOptions: { dispatcher: new Agent({ keepAliveTimeout: 120_000 }) },
});

const response = await client.audio.speech.create({
  model: "assemblyai-tts-en",
  voice: "alloy",
  input: "Hello from AssemblyAI.",
  response_format: "wav",
});
await fs.writeFile("hello.wav", Buffer.from(await response.arrayBuffer()));

The Python example is for openai 3, which is built on httpx2; with openai 2 or earlier, write httpx for httpx2 and DefaultHttpxClient for DefaultHttpx2Client. The JavaScript example needs npm install undici, because the SDK takes a dispatcher only together with undici’s own fetch; leave both out in a browser. The SDKs send the key as Authorization: Bearer <key>, which the API accepts. hello.pcm holds raw 16-bit mono PCM at 24 kHz. In a browser, give the JavaScript SDK a temporary token minted with product=tts as its apiKey, never your API key, and set its dangerouslyAllowBrowser option to true: without it the SDK refuses to run in a browser.

Keep connections between replies

Build the client once per process and reuse it. Our load balancer keeps an idle connection open for an hour, but the OpenAI SDKs close it much sooner: the Python SDK after 5 seconds, and the JavaScript SDK on Node after 4, the default of Node’s fetch. A voice agent whose replies are further apart than that opens a new connection, a TCP and a TLS handshake, before nearly every reply. The limits and dispatcher settings in the examples above keep idle connections for 2 minutes. In the API’s region, a reply on a kept connection reached its first byte about 6 ms sooner with the Python SDK, after pauses of 6 and 65 seconds (40 replies per way), and too little sooner to measure with Node. From farther away, every reconnect costs two network round trips: about 120 ms between the US west coast and tts.us. To open the connection before the first reply as well, call client.models.list() when a call starts; it reads GET /v1/openai/models, which synthesizes nothing and needs no credential. See Optimizing accuracy and latency for the other habits that shorten the wait.

Voice agent frameworks

LiveKit and Pipecat both have native AssemblyAI plugins, documented on LiveKit and Pipecat, which run on the WebSocket API with one session per call, barge-in without reconnecting and word-timed transcripts. Use those for a new agent. The OpenAI plugins below are for a pipeline already built on them, and give up the word timings. Voice agents should ask for pcm: raw PCM starts playing on the first bytes and needs no decoding, while MP3 adds about 46 ms of encoder silence (at 24 kHz) to the start of every sentence.

LiveKit
Pipecat

import os

from livekit.plugins import openai

tts = openai.TTS(
    base_url="https://tts.us.assemblyai.com/v1/openai",
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
    model="assemblyai-tts-en",
    voice="alloy",
    response_format="pcm",
)

The plugin already keeps idle connections for 2 minutes, and LiveKit opens one when the agent session starts, so it needs no change to keep connections between replies.

import os

import httpx2
from openai import DefaultAsyncHttpx2Client
from pipecat.services.openai.tts import OpenAITTSService

tts = OpenAITTSService(
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
    base_url="https://tts.us.assemblyai.com/v1/openai",
    # Idle connections kept for 2 minutes instead of 5 seconds.
    http_client=DefaultAsyncHttpx2Client(
        limits=httpx2.Limits(
            max_connections=1000, max_keepalive_connections=100, keepalive_expiry=120
        )
    ),
    settings=OpenAITTSService.Settings(model="assemblyai-tts-en", voice="alloy"),
)

http_client keeps connections between replies, as in OpenAI SDKs. Pipecat takes http_client from version 1.7, and openai 3 from 1.10; on 1.7 to 1.9, or with openai 2 or earlier, build it with httpx and DefaultAsyncHttpxClient instead.Pipecat’s service always asks for pcm at 24 kHz, and accepts only OpenAI voice names, which the route maps to AssemblyAI voices (see Voices).

Request fields

model and input are required. voice names the voice, response_format the audio (mp3 by default, pcm for a voice agent, wav, mulaw or alaw), and stream_format: "sse" asks for speech events instead of bytes. Create speech (OpenAI compatible) lists every field, its values and its refusals; the sections below cover the ones that are not plain OpenAI.

Models

`model`	Speaks with
`assemblyai-tts-en`, `assemblyai-tts-es`, `assemblyai-tts-de`, `assemblyai-tts-it`, `assemblyai-tts-pt`, `assemblyai-tts-fr`	The default model of that language. The code also sets the language, so `assemblyai-tts-es` asks for Spanish
A model’s name, such as `assemblyai-tts-1`	That model, as `model_id` selects it on the WebSocket API. The name sets no language; see Extensions for how the language is chosen

Names are case-insensitive. OpenAI’s model names, such as tts-1, are refused with 400 and the code model_not_found; List models (OpenAI compatible) answers the language model names, and the X-TTS-Model response header reports the model that spoke.

Voices

voice keeps its OpenAI name and takes one of these:

An OpenAI voice name, in any letter case. Each one speaks with an AssemblyAI voice in its place, as the table below shows.
The library slug of an AssemblyAI voice, such as aai-jane, in lowercase: the voice_id that names the voice on every other route.
A voice’s ID, a UUID. A cloned voice can also go in as an object.

A bare AssemblyAI voice name such as jane is not one of them. It is refused with 422 (invalid_request_error, code invalid_parameter, param voice), and the message names the slug to send, aai-jane.

OpenAI voice	AssemblyAI voice
`alloy`	`aai-jane`
`ash`	`aai-michael`
`ballad`	`aai-charles`
`cedar`	`aai-george`
`coral`	`aai-anna`
`echo`	`aai-paul`
`fable`	`aai-charles`
`marin`	`aai-mary`
`nova`	`aai-eve`
`onyx`	`aai-george`
`sage`	`aai-vera`
`shimmer`	`aai-alba`
`verse`	`aai-michael`

An OpenAI voice name speaks with the AssemblyAI voice in this table, which speaks English. Jane has no OpenAI name; ask for her, like any AssemblyAI voice, by her slug, aai-jane. The response header X-TTS-Voice-Id gives the voice_id the request spoke with, and when you sent an OpenAI voice name, X-TTS-Voice-Alias holds it. Other values are not supported: a misspelt name is refused with 422 like a bare name, and a slug or UUID that names no voice with 400, code invalid_request, both on voice.

Cloned voices

A cloned voice you created in the AssemblyAI dashboard goes in voice as an object holding its ID or slug, {"id": "<voice ID>"}, which the OpenAI Python SDK passes through as it is, or as a string holding its UUID. The slug of one of your voices in a plain string is refused: send it as {"id": "<slug>"}. Name the model of the voice’s own language, such as assemblyai-tts-en for an English voice, or a model’s name, such as assemblyai-tts-1; another language’s model is refused with 400 on language.

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://tts.us.assemblyai.com/v1/openai",
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
)

with client.audio.speech.with_streaming_response.create(
    model="assemblyai-tts-en",
    voice={"id": os.environ["ASSEMBLYAI_VOICE_ID"]},
    input="Thanks for calling.",
    response_format="pcm",
) as response:
    print("voice_id:", response.headers.get("x-tts-voice-id"))
    response.stream_to_file("thanks.pcm")

The voice’s refusals answer 400 invalid_request_error with the code voice_not_found, voice_expired, voice_blocked, voice_not_paired, model_retired, model_not_found or not_owner, and biometric_terms_not_accepted answers 403; see Errors.

Extensions

These fields are not part of the OpenAI request; most OpenAI clients pass them through an extra-body option (extra_body in the Python SDK):

Field	Values
`language`	A language name (`english`) or an ISO 639-1 code (`en`), in any letter case. Refused when it differs from the language `model` implies
`sample_rate`	`8000`, `16000`, `22050`, `24000`, `44100`, `48000`; default `24000`
`voice_id`	The ID or slug of a cloned voice, for clients that cannot send `voice` as an object. Leave `voice` out: next to any other `voice`, such as `alloy` or `aai-jane`, it is refused (`voice_id: conflicts with voice; give one of them`)
`extend_biometric_consent`	`true` renews the biometric consent of a cloned voice named in `voice` as `{"id": ...}` or by its UUID, or in `voice_id`, and attests renewed consent on behalf of the account that owns it. Only the voice’s owner may set it; see Renew consent

The language is the one language sets, else the one the model name implies, else a cloned voice’s own language, else English. Clients that cannot add fields pick a language with the model name.

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://tts.us.assemblyai.com/v1/openai",
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
)

with client.audio.speech.with_streaming_response.create(
    model="assemblyai-tts-en",
    voice="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3",
    input="Thanks for calling.",
    response_format="pcm",
    extra_body={"language": "en", "sample_rate": 16000},
) as response:
    response.stream_to_file("thanks.pcm")

Word timings are not available on this route: word_boundaries must be left out or false. Use Stream speech with word timings for them.

Speech events

With "stream_format": "sse" the response is text/event-stream in OpenAI’s form: one data: line per event and no event: lines. While nothing is sent for 15 seconds, a : keepalive comment line arrives; skip lines that start with :.

data: {"type":"speech.audio.delta","audio":"AAAAAP7/AQD+/wIA"}

data: {"type":"speech.audio.delta","audio":"/f8DAP7/AQAAAP//"}

data: {"type":"speech.audio.done","usage":{"input_tokens":0,"output_tokens":0,"total_tokens":0,"input_characters":22,"billed_characters":22,"audio_duration_seconds":1.62}}

Each speech.audio.delta carries base64 of the next bytes of the response_format: with wav, the WAV header comes first; with mp3, whole MP3 frames.
speech.audio.done ends the stream. Its usage counts characters, not tokens: the token counts are always 0, input_characters is the length of input, and billed_characters what the request is billed. There is no [DONE] line.
A failure after the first audio sends {"type": "error", "error": {...}}, with the fields of the error body below, and then cuts the connection. With httpx, read with iter_bytes() or iter_lines() and no chunk_size, or the error event can be lost with the cut; see Streamed and buffered responses.

Errors

Errors use OpenAI’s body, so OpenAI SDKs raise their usual exceptions (openai.BadRequestError, openai.RateLimitError, …). The code is the one the HTTP API uses:

{
  "error": {
    "message": "model: 'tts-1' is not an AssemblyAI model; use one of GET /v1/openai/models, such as assemblyai-tts-en",
    "type": "invalid_request_error",
    "param": "model",
    "code": "model_not_found",
    "request_id": "93309b7e-9131-474f-99e4-6e0623463374",
    "retryable": false,
    "retry_after_seconds": null
  }
}

Every status and code of the route is listed on Create speech (OpenAI compatible). In short: 400 for a bad field, model or voice, 401 for the credential, 402 for funds, 422 for a voice string that is not an OpenAI name, a slug or an ID, 429 for both the requests-per-minute limit (rate_limited) and no room just now (at_capacity), and 503 when the service is unavailable. OpenAI SDKs retry a 429 on their own, twice by default. A request can be refused at_capacity after it was counted against your requests per minute, so each of those retries counts as a request too. To keep retries within your limit, set max_retries on the client and back off for Retry-After yourself. Every response carries the request’s id in X-Request-Id, and a 200 also in X-Generation-Id; quote it in support requests. Send X-Client-Request-Id (1 to 512 printable ASCII characters) to have your own id echoed back.

Create speech (OpenAI compatible): the route reference, with every field, status and code.
List models (OpenAI compatible) and List models (OpenRouter): the public models documents.
HTTP API basics: hosts, headers, limits and billing shared with the native routes; Word boundaries for word timings over HTTP.
LiveKit and Pipecat: the native plugins.
Optimizing accuracy and latency: client habits that shorten the time to first audio.

Getting started

Features

Integrations

API reference

Advanced

OpenAI compatibility

OpenAI SDKs

Keep connections between replies

Voice agent frameworks

Request fields

Models

Voices

Cloned voices

Extensions

Speech events

Errors

​OpenAI SDKs

​Keep connections between replies

​Voice agent frameworks

​Request fields

​Models

​Voices

​Cloned voices

​Extensions

​Speech events

​Errors

​Related

OpenAI SDKs

Keep connections between replies

Voice agent frameworks

Request fields

Models

Voices

Cloned voices

Extensions

Speech events

Errors

Related