> ## Documentation Index > Fetch the complete documentation index at: https://assemblyai.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Streaming TTS on LiveKit > Give a LiveKit voice agent an AssemblyAI voice with the assemblyai.TTS plugin, with word-aligned transcripts and barge-in on one pooled session. ## Overview [LiveKit Agents](https://docs.livekit.io/agents/) is an open-source framework for building real-time voice agents. Your agent joins a LiveKit room as a participant and exchanges audio with users over WebRTC or SIP telephony. An `AgentSession` wires together speech-to-text, LLM, and text-to-speech plugins and handles the orchestration, including turn detection and interruptions. The AssemblyAI plugin, `livekit-plugins-assemblyai`, provides `assemblyai.TTS`, which speaks your agent's replies with the [Streaming TTS API](https://assemblyai.com/docs/tts/getting-started/quickstart.md). Use it with the plugin's `assemblyai.STT` and the [LLM Gateway](https://assemblyai.com/docs/llm-gateway/quickstart.md), and every model in the agent runs on one AssemblyAI API key. > [!NOTE] > > `assemblyai.TTS` is available in **`livekit-plugins-assemblyai` 1.8.9+**. What the plugin handles for you: * **Streaming by sentence.** Each sentence of the LLM's reply is its own request, so the agent starts speaking after the first sentence rather than after the whole reply. * **One session for many replies.** The plugin keeps its WebSocket session open and reuses it for every reply, instead of opening a new one per reply. * **Barge-in without reconnecting.** When the user interrupts, the plugin cancels the audio still in flight and keeps the session. * **Word-aligned transcripts.** Per-word timings let LiveKit record an interrupted reply up to the word the user heard. ## Quickstart **1. Install the packages** Install LiveKit Agents with the AssemblyAI, OpenAI, and Silero plugins. The OpenAI plugin connects the agent to the LLM Gateway, which speaks the OpenAI Chat Completions API. ```bash theme={null} pip install "livekit-agents[openai,silero]~=1.8" \ "livekit-plugins-assemblyai>=1.8.9" \ python-dotenv ``` **2. Set your API keys** Set your API keys in a `.env` file: ```env theme={null} ASSEMBLYAI_API_KEY=your_assemblyai_key # Not needed for console mode, only to connect to LiveKit LIVEKIT_URL=wss://your-project.livekit.cloud LIVEKIT_API_KEY=your_livekit_api_key LIVEKIT_API_SECRET=your_livekit_api_secret ``` > [!TIP] > > You can get an AssemblyAI API key by [signing up for a free > account](https://www.assemblyai.com/dashboard/signup) and going to the [API > Keys tab](https://www.assemblyai.com/dashboard/home) of the dashboard. **3. Build the agent** Save this as `agent.py`. It uses AssemblyAI for speech-to-text, the LLM, and text-to-speech: ```python expandable theme={null} import os from dotenv import load_dotenv from livekit.agents import ( Agent, AgentServer, AgentSession, JobContext, JobProcess, TurnHandlingOptions, cli, ) from livekit.plugins import assemblyai, openai, silero load_dotenv() # Silero (used for barge-in) should agree with AssemblyAI's own VAD on what # counts as speech, so both use the STT's default threshold. VAD_THRESHOLD = 0.2 class Assistant(Agent): def __init__(self) -> None: super().__init__( instructions=( "You are a friendly voice assistant. Keep replies to one or two " "short sentences, with no lists, markdown, or emojis." ), ) async def on_enter(self) -> None: # A fixed greeting is spoken right away, without waiting on the LLM self.session.say("Hi, thanks for calling. How can I help?") server = AgentServer() def prewarm(proc: JobProcess) -> None: proc.userdata["vad"] = silero.VAD.load(activation_threshold=VAD_THRESHOLD) server.setup_fnc = prewarm @server.rtc_session() async def entrypoint(ctx: JobContext) -> None: tts = assemblyai.TTS( model="assemblyai-tts-1", voice="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", # Audrey ) # Open the TTS session now, so the first reply doesn't wait on the handshake tts.prewarm() session = AgentSession( stt=assemblyai.STT( model="universal-3-6-pro", mode="balanced", vad_threshold=VAD_THRESHOLD, ), # The LLM Gateway, through its OpenAI-compatible API llm=openai.LLM( model="gemma-4-31b", base_url="https://llm-gateway.assemblyai.com/v1", api_key=os.environ["ASSEMBLYAI_API_KEY"], ), tts=tts, vad=ctx.proc.userdata["vad"], turn_handling=TurnHandlingOptions( # AssemblyAI decides when the user has finished speaking... turn_detection="stt", # ...so LiveKit shouldn't add its own delay on top endpointing={"min_delay": 0}, ), tts_text_transforms=["filter_emoji", "filter_markdown"], # When the user interrupts, keep only the words they heard in the chat history use_tts_aligned_transcript=True, ) await session.start(agent=Assistant(), room=ctx.room) if __name__ == "__main__": cli.run_app(server) ``` **4. Talk to it** Run the agent in console mode, which uses your computer's microphone and speakers and needs only your AssemblyAI API key: ```bash theme={null} python agent.py console ``` Recent `livekit-agents` releases print a deprecation notice for `console` and point at `lk agent console`, which runs the same mode through the LiveKit CLI. On a machine without a microphone, `python agent.py console --text` takes typed turns instead. Use headphones, so the agent doesn't hear its own voice. To connect the agent to LiveKit for a web or phone frontend, run `python agent.py dev` with your LiveKit credentials set. For the speech-to-text settings in this example, such as `mode` and `vad_threshold`, see [Universal-3.6 Pro on LiveKit](https://assemblyai.com/docs/voice-agents/livekit-universal-3-6-pro.md). ## Reference agent > **[AssemblyAI reference voice agent for LiveKit](https://github.com/AssemblyAI-Solutions/assemblyai-livekit-reference-voice-agent)** > > A complete starter agent that runs every model on AssemblyAI: a medical > receptionist that takes a new patient's intake details or looks up a > returning patient, then books an appointment. It uses Universal-3.6 Pro with > Medical Mode, the LLM Gateway, and Streaming TTS, and comes with tests and a > Dockerfile for deploying to LiveKit Cloud. Fork it and make it yours. ## Parameters reference | Parameter | Default | Description | | - | - | - | | `voice` | `987db62e-a2c2-569a-9c4d-82460cdea1bd` (Keegan) | The voice's ID or slug. See [Choose a voice](#choose-a-voice). | | `language` | Unset: English, or a library or cloned voice's own language | A language code such as `"es"` or an English name such as `"spanish"`. See [Set the language](#set-the-language). | | `model` | `assemblyai-tts-1` | The TTS model. | | `sample_rate` | `24000` | Output sample rate in Hz: `8000`, `16000`, `22050`, `24000`, `44100`, or `48000`. `24000` is the model's native rate. | | `word_timestamps` | `True` | Request per-word timings and publish them as an aligned transcript. | | `api_key` | `ASSEMBLYAI_API_KEY` environment variable | Your AssemblyAI API key. | | `base_url` | `wss://tts.us.assemblyai.com` | The Streaming TTS host. Use `wss://tts.eu.assemblyai.com` for the EU data zone. | | `tokenizer` | LiveKit's Blingfire sentence tokenizer | Splits the LLM's reply into the requests sent to the server, one per sentence. | | `text_pacing` | `False` | A LiveKit `SentenceStreamPacer`, or `True` to use the default pacer. | | `http_session` | LiveKit's shared session | An existing `aiohttp.ClientSession` to use. | ## Choose a voice Pass a voice to `voice` by its ID, a UUID. The default is Keegan, `987db62e-a2c2-569a-9c4d-82460cdea1bd`, which differs from the API's own default, `aai-jane`. `voice` takes any voice your account can use: * One of the [AssemblyAI voices](https://assemblyai.com/docs/tts/voices.md#assemblyai-voices), each of which speaks the language it was recorded in. * A voice from the AssemblyAI voice library. These speak one language, the language of their recording. * A voice you [cloned](https://assemblyai.com/docs/tts/cloned-voices.md) in the AssemblyAI dashboard. The [voice catalog](https://www.assemblyai.com/dashboard/text-to-speech/voices) in the AssemblyAI dashboard has a **Copy voice ID** action for each voice, and [Voices](https://assemblyai.com/docs/tts/voices.md) lists good voices to start with. A slug such as `aai-anna` works too, but a voice's ID never changes. A cloned voice works only on the host of the [data zone](#keep-data-in-the-eu) it was created in. ## Set the language Set `language` to the language of the text your LLM writes: ```python theme={null} tts = assemblyai.TTS(voice="aai-anna", language="es") ``` `language` takes a language code (`"es"`, `"es-MX"`) or the language's English name (`"spanish"`). The plugin sends the server the English name the API expects, so you can use the same codes you pass to `assemblyai.STT`. See [Languages](https://assemblyai.com/docs/tts/voices.md#languages) for the six supported languages. Choose a voice that speaks the language. Every voice, [AssemblyAI](https://assemblyai.com/docs/tts/voices.md#assemblyai-voices), library or cloned, speaks only the language it was recorded in: leave `language` unset and the session speaks the voice's language. The API doesn't detect language, so an AssemblyAI voice speaks English unless you set `language`. ## Keep data in the EU `base_url` sets the data zone. It defaults to the US, `wss://tts.us.assemblyai.com`. To keep text and audio in the EU, point the TTS at the EU host, and the STT and LLM Gateway at theirs: ```python theme={null} assemblyai.STT(..., base_url="wss://streaming.eu.assemblyai.com") assemblyai.TTS(..., base_url="wss://tts.eu.assemblyai.com") openai.LLM(..., base_url="https://llm-gateway.eu.assemblyai.com/v1") ``` The EU LLM Gateway serves a different set of models; see [Cloud endpoints and data residency](https://assemblyai.com/docs/llm-gateway/cloud-endpoints-and-data-residency.md). For the TTS endpoints, see [Endpoints and data zones](https://assemblyai.com/docs/tts/endpoints-and-data-zones.md). ## Interruptions and transcripts When the user interrupts, the plugin sends `Cancel` on the open session and drops the audio of the interrupted reply that is still in flight. The session stays open, so the next reply doesn't wait for a new handshake. The plugin requests [word boundaries](https://assemblyai.com/docs/tts/word-boundaries.md) by default (`word_timestamps=True`) and publishes them to LiveKit as an aligned transcript. Set `use_tts_aligned_transcript=True` on the `AgentSession` to use them: * The transcript your frontend shows follows the agent's speech word by word. * When the user cuts the agent off, the chat history keeps the reply only up to the word the user heard, so the LLM doesn't think it said things it didn't. ## Latency * **Call `tts.prewarm()`** when the call starts, as in the [quickstart](#quickstart). It opens the session before the first reply, so the greeting doesn't wait on the handshake. * **Greet with `session.say()`.** A fixed greeting in `on_enter` is spoken as soon as the user joins, without waiting for the LLM. * **Keep replies short.** The agent speaks after the first sentence, so a short first sentence starts the audio sooner. ## Sessions and rate limits AssemblyAI counts TTS sessions per account by [new sessions per minute](https://assemblyai.com/docs/tts/limits.md#session-lifetime), not by sessions open at once. Because the plugin reuses one session for many replies, a call usually opens one session. A session lasts at most an hour. The plugin doesn't start a reply on a session that is about to expire: it opens a new one instead. The plugin retries a failed request when the server marks the error as retryable, and waits as long as the server's `retry_after_seconds` asks before opening another session. If that wait is longer than the connection timeout, the plugin stops retrying, so a LiveKit `tts.FallbackAdapter` can move on to another TTS. See [Error handling](https://assemblyai.com/docs/tts/error-handling.md) for the error codes. ## Change the voice during a call `update_options` changes the voice, language, or model: ```python theme={null} tts.update_options(voice="aai-michael", language="fr") ``` The API fixes a session's voice, language, and model when it opens, so the next reply opens a new session with the new options. A reply already playing keeps its voice until it finishes. Each new session counts toward your new sessions per minute. ## Related * [Universal-3.6 Pro on LiveKit](https://assemblyai.com/docs/voice-agents/livekit-universal-3-6-pro.md): speech-to-text, turn detection, and interruption settings for LiveKit agents. * [Voices](https://assemblyai.com/docs/tts/voices.md): the AssemblyAI voices, languages, and how to find a voice's ID. * [Voice cloning](https://assemblyai.com/docs/tts/cloned-voices.md): clone a voice from a short recording. * [Word boundaries](https://assemblyai.com/docs/tts/word-boundaries.md): the per-word timings behind aligned transcripts. * [LiveKit Agents documentation](https://docs.livekit.io/agents/): sessions, tools, workflows, and telephony.