> ## Documentation Index
> Fetch the complete documentation index at: https://assemblyai.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Streaming TTS on Pipecat

> Give a Pipecat voice agent an AssemblyAI voice with AssemblyAITTSService, with word-timed context and barge-in on one session per call.

## Overview

[Pipecat](https://docs.pipecat.ai/) is an open-source Python framework for building real-time voice agents. You compose your agent as a pipeline of services (speech-to-text, LLM, text-to-speech) that stream audio and data frames to each other, and Pipecat handles the orchestration, including turn-taking, interruptions, and transports such as WebRTC, Daily, and telephony.

`AssemblyAITTSService` speaks your agent's replies with the [Streaming TTS API](https://assemblyai.com/docs/tts/getting-started/quickstart.md). Use it with `AssemblyAISTTService` and the [LLM Gateway](https://assemblyai.com/docs/llm-gateway/quickstart.md), and every model in the agent runs on one AssemblyAI API key.

> [!NOTE]
>
> `AssemblyAITTSService` is available in **`pipecat-ai` 1.14.0+**, with the `assemblyai` extra.

What the service handles for you:

* **Streaming by sentence.** Each sentence of the LLM's reply is its own request, so the agent starts speaking after the first sentence rather than after the whole reply.
* **One session per call.** The service opens its WebSocket session when the pipeline starts and keeps it for every reply, sending a keepalive while the agent is quiet.
* **Barge-in without reconnecting.** When the user interrupts, the service cancels the audio still in flight and keeps the session.
* **Word-timed context.** Per-word timings drive Pipecat's spoken-text frames, so an interrupted reply is saved to the LLM context only up to the word the user heard.

## Quickstart

**1. Install the packages**

Install Pipecat with the AssemblyAI, OpenAI, and Silero extras, and the runner and WebRTC extras to talk to the agent in your browser. The OpenAI service connects the agent to the LLM Gateway, which speaks the OpenAI Chat Completions API.

```bash theme={null}
pip install "pipecat-ai[assemblyai,openai,silero,runner,webrtc]>=1.14.0" python-dotenv
```

**2. Set your API key**

Set your API key in a `.env` file:

```env theme={null}
ASSEMBLYAI_API_KEY=your_assemblyai_key
```

> [!TIP]
>
> You can get an AssemblyAI API key by [signing up for a free
> account](https://www.assemblyai.com/dashboard/signup) and going to the [API
> Keys tab](https://www.assemblyai.com/dashboard/home) of the dashboard.

**3. Build the agent**

Save this as `bot.py`. It uses AssemblyAI for speech-to-text, the LLM, and text-to-speech:

```python expandable theme={null}
import os

from dotenv import load_dotenv
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.frames.frames import TTSSpeakFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.worker import PipelineParams, PipelineWorker
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.runner.types import RunnerArguments
from pipecat.runner.utils import create_transport
from pipecat.services.assemblyai.stt import AssemblyAISTTService
from pipecat.services.assemblyai.tts import AssemblyAITTSService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.base_transport import BaseTransport, TransportParams
from pipecat.workers.runner import WorkerRunner

load_dotenv()

# Silero (used for barge-in) should agree with AssemblyAI's own VAD on what
# counts as speech, so both use the STT's default threshold.
VAD_THRESHOLD = 0.2

transport_params = {
    "webrtc": lambda: TransportParams(audio_in_enabled=True, audio_out_enabled=True),
}


async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
    api_key = os.environ["ASSEMBLYAI_API_KEY"]

    stt = AssemblyAISTTService(
        api_key=api_key,
        # AssemblyAI decides when the user has finished speaking
        vad_force_turn_endpoint=False,
        settings=AssemblyAISTTService.Settings(
            model="universal-3-6-pro",
            mode="balanced",
            vad_threshold=VAD_THRESHOLD,
        ),
    )
    # The LLM Gateway, through its OpenAI-compatible API
    llm = OpenAILLMService(
        api_key=api_key,
        base_url="https://llm-gateway.assemblyai.com/v1",
        settings=OpenAILLMService.Settings(
            model="gemma-4-31b",
            system_instruction=(
                "You are a friendly voice assistant. Keep replies to one or two "
                "short sentences, with no lists, markdown, or emojis."
            ),
        ),
    )
    tts = AssemblyAITTSService(
        api_key=api_key,
        settings=AssemblyAITTSService.Settings(
            model="assemblyai-tts-1",
            voice="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3",  # Audrey
        ),
    )

    context = LLMContext()
    user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
        context,
        user_params=LLMUserAggregatorParams(
            vad_analyzer=SileroVADAnalyzer(params=VADParams(confidence=VAD_THRESHOLD))
        ),
    )

    pipeline = Pipeline(
        [
            transport.input(),
            stt,
            user_aggregator,
            llm,
            tts,
            transport.output(),
            assistant_aggregator,
        ]
    )
    worker = PipelineWorker(pipeline, params=PipelineParams(enable_metrics=True))
    runner = WorkerRunner(handle_sigint=runner_args.handle_sigint)
    await runner.add_workers(worker)

    @transport.event_handler("on_client_connected")
    async def on_client_connected(transport, client):
        # A fixed greeting is spoken right away, without waiting on the LLM
        await worker.queue_frames([TTSSpeakFrame("Hi, I'm Audrey. How can I help?")])

    @transport.event_handler("on_client_disconnected")
    async def on_client_disconnected(transport, client):
        await runner.cancel()

    await runner.run()


async def bot(runner_args: RunnerArguments):
    transport = await create_transport(runner_args, transport_params)
    await run_bot(transport, runner_args)


if __name__ == "__main__":
    from pipecat.runner.run import main

    main()
```

**4. Talk to it**

Run the agent:

```bash theme={null}
python bot.py
```

Open [http://localhost:7860/client](http://localhost:7860/client), allow the microphone, and press **Connect**. Use headphones, so the agent doesn't hear its own voice. To run the agent in a Daily room or on a phone number, add their transports to `transport_params`; see [Pipecat's transports](https://docs.pipecat.ai/api-reference/server/services/transport/daily).

For the speech-to-text settings in this example, such as `vad_force_turn_endpoint` and `mode`, see [Universal-3.6 Pro on Pipecat](https://assemblyai.com/docs/voice-agents/pipecat-universal-3-5-pro.md).

## Reference agent

> **[AssemblyAI voice agent starter for Pipecat](https://github.com/AssemblyAI-Solutions/assemblyai-pipecat-reference-voice-agent)**
>
> A starter agent that runs every model on AssemblyAI: Streaming TTS,
> Universal-3.6 Pro, and the LLM Gateway. Every model setting is a building
> block with a skill that teaches coding agents such as Claude Code when and how
> to tune it, from the voice to turn detection and noise suppression. It comes
> with tests and a Dockerfile for deploying to Pipecat Cloud. Fork it and make
> it yours.

## Parameters reference

| Parameter | Default | Description |
| - | - | - |
| `api_key` | Required | Your AssemblyAI API key. A [temporary token](https://assemblyai.com/docs/tts/authenticate-with-a-temporary-token.md) opens only one session, so it can't carry the service's reconnects and session renewals. |
| `url` | `wss://tts.us.assemblyai.com/v1/ws` | The Streaming TTS endpoint. Use `wss://tts.eu.assemblyai.com/v1/ws` for the EU data zone. |
| `sample_rate` | The pipeline's output rate (`24000` by default) | Output sample rate in Hz: `8000`, `16000`, `22050`, `24000`, `44100`, or `48000`. |
| `settings` | See below | An `AssemblyAITTSService.Settings` with the session's voice, language, and model. |
| `text_aggregation_mode` | Sentences | Pipecat's `TextAggregationMode`. `TextAggregationMode.TOKEN` streams the LLM's tokens as they arrive instead of sentence by sentence. |

`AssemblyAITTSService.Settings`:

| Setting | Default | Description |
| - | - | - |
| `voice` | Unset: the server's default voice, Jane (`95e643fb-dd55-577f-80ec-134c3118efcc`) | The voice's ID or slug, sent as `voice_id`. See [Choose a voice](#choose-a-voice). |
| `language` | Unset: the voice's own language, otherwise English | A Pipecat `Language`, a language code such as `"es"`, or an English name such as `"spanish"`. See [Set the language](#set-the-language). |
| `model` | Unset: the default model of the language, `assemblyai-tts-1` | The TTS model. |

## Choose a voice

Pass a voice to `voice` by its ID, a UUID. Unset, the session speaks with the server's default voice, Jane (`95e643fb-dd55-577f-80ec-134c3118efcc`). `voice` takes any voice your account can use:

* One of the [AssemblyAI voices](https://assemblyai.com/docs/tts/voices.md#assemblyai-voices), which speak all six supported languages.
* A voice from the AssemblyAI voice library. These speak one language, the language of their recording.
* A voice you [cloned](https://assemblyai.com/docs/tts/cloned-voices.md) in the AssemblyAI dashboard.

The Voice Library in the AssemblyAI dashboard has a **Copy voice ID** action for each voice; [Voices](https://assemblyai.com/docs/tts/voices.md) also shows how to list voices from code. A slug such as `aai-anna` works too, but a voice's ID never changes.

A cloned voice works only on the host of the [data zone](#keep-data-in-the-eu) it was created in.

## Set the language

Set `language` to the language of the text your LLM writes:

```python theme={null}
from pipecat.transcriptions.language import Language

tts = AssemblyAITTSService(
    api_key=api_key,
    settings=AssemblyAITTSService.Settings(
        voice="33a0302a-51cd-5b46-81c9-f4aa8328900f",  # Anna
        language=Language.ES,
    ),
)
```

`language` takes a Pipecat `Language` (`Language.ES`), a language code (`"es"`, `"es-MX"`), or the language's English name (`"spanish"`). The service sends the server the English name the API expects, so you can use the same `Language` values you pass to `AssemblyAISTTService`. See [Languages](https://assemblyai.com/docs/tts/voices.md#languages) for the six supported languages.

Choose a voice that speaks the language. The [AssemblyAI voices](https://assemblyai.com/docs/tts/voices.md#assemblyai-voices) speak all six. A library or cloned voice speaks only the language it was recorded in: leave `language` unset and the session speaks the voice's language. The API doesn't detect language, so an AssemblyAI voice speaks English unless you set `language`.

## Keep data in the EU

`url` sets the data zone. It defaults to the US, `wss://tts.us.assemblyai.com/v1/ws`. To keep text and audio in the EU, point the TTS at the EU host, and the STT and LLM Gateway at theirs:

```python theme={null}
AssemblyAISTTService(..., api_endpoint_base_url="wss://streaming.eu.assemblyai.com/v3/ws")
AssemblyAITTSService(..., url="wss://tts.eu.assemblyai.com/v1/ws")
OpenAILLMService(..., base_url="https://llm-gateway.eu.assemblyai.com/v1")
```

The EU LLM Gateway serves a different set of models; see [Cloud endpoints and data residency](https://assemblyai.com/docs/llm-gateway/cloud-endpoints-and-data-residency.md). For the TTS endpoints, see [Endpoints and data zones](https://assemblyai.com/docs/tts/endpoints-and-data-zones.md).

## Interruptions and context

When the user interrupts, the service sends `Cancel` on the open session and drops the audio of the interrupted reply that is still in flight. The session stays open, so the next reply doesn't wait for a new handshake.

The service always requests [word boundaries](https://assemblyai.com/docs/tts/word-boundaries.md) and turns them into Pipecat's spoken-text frames, timed to the audio. With no extra setting:

* The assistant's transcript follows the agent's speech word by word.
* When the user cuts the agent off, the assistant aggregator saves the reply to the LLM context only up to the word the user heard, so the LLM doesn't think it said things it didn't.

## Latency

* **The session opens when the pipeline starts.** The service connects during setup, so the first reply doesn't wait on the handshake.
* **Greet with a `TTSSpeakFrame`.** A fixed greeting queued in `on_client_connected` is spoken as soon as the user joins, without waiting for the LLM.
* **Keep replies short.** The agent speaks after the first sentence, so a short first sentence starts the audio sooner. To start before a sentence is complete, set `text_aggregation_mode=TextAggregationMode.TOKEN` (from `pipecat.services.tts_service`).

## Sessions and rate limits

AssemblyAI counts TTS sessions per account by [new sessions per minute](https://assemblyai.com/docs/tts/limits.md#session-lifetime), not by sessions open at once. Because the service keeps one session for the whole call, a call usually opens one session.

* **Idle sessions.** A session closes after two minutes with no frame either way, and an agent can be quiet that long while the user talks. The service sends a `KeepAlive` every 30 seconds to keep it open.
* **Long calls.** A session lasts at most an hour. Between replies, the service replaces a session that is about to expire, so a reply is never cut off by the session's end.
* **Errors.** The service reconnects when the server marks an error as retryable, and waits as long as the server's `retry_after_seconds` asks first. An error that can't be retried before the session starts, such as an invalid API key or an unknown voice, marks the service unusable, so the pipeline's `ProcessorUnusablePolicy` or a fallback service can act on it. See [Error handling](https://assemblyai.com/docs/tts/error-handling.md) for the error codes.

## Change the voice during a call

Push a `TTSUpdateSettingsFrame` to change the voice, language, or model:

```python theme={null}
from pipecat.frames.frames import TTSUpdateSettingsFrame

await worker.queue_frame(
    TTSUpdateSettingsFrame(
        delta=AssemblyAITTSService.Settings(
            voice="b009cbfa-2237-59ed-95aa-761ccd2ae37e",  # Michael
            language="fr",
        )
    )
)
```

The API fixes a session's voice, language, and model when it opens, so the service reconnects with the new settings. Each new session counts toward your new sessions per minute.

## Related

* [Universal-3.6 Pro on Pipecat](https://assemblyai.com/docs/voice-agents/pipecat-universal-3-5-pro.md): speech-to-text, turn detection, and interruption settings for Pipecat agents.
* [Voices](https://assemblyai.com/docs/tts/voices.md): the AssemblyAI voices, languages, and how to find a voice's ID.
* [Voice cloning](https://assemblyai.com/docs/tts/cloned-voices.md): clone a voice from a short recording.
* [Word boundaries](https://assemblyai.com/docs/tts/word-boundaries.md): the per-word timings behind word-timed context.
* [Pipecat documentation](https://docs.pipecat.ai/): pipelines, transports, and deployment.