Pipecat

Overview

Pipecat is an open source platform for developers building realtime media applications. In this guide, we’ll show you how to integrate AssemblyAI’s streaming speech-to-text model into your Pipecat voice agent using the Pipeline framework.

Pipecat

View Pipecat’s AssemblyAI STT plugin documentation.

Quick start

Installation

Install the AssemblyAI service from PyPI:

$
pip install "pipecat-ai[assemblyai]"

Authentication

The AssemblyAI service requires an AssemblyAI API key. Set ASSEMBLYAI_API_KEY in your .env file.

Basic usage

Use AssemblyAI STT in a Pipeline:

1
from pipecat.services.assemblyai.stt import AssemblyAISTTService, AssemblyAIConnectionParams
2
3
# Configure service
4
stt = AssemblyAISTTService(
5
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
6
    vad_force_turn_endpoint=False,  # Use AssemblyAI's STT-based turn detection
7
    connection_params=AssemblyAIConnectionParams(
8
        end_of_turn_confidence_threshold=0.7,
9
        min_end_of_turn_silence_when_confident=160,
10
        max_turn_silence=2400,
11
    )
12
)
13
14
# Use in pipeline
15
pipeline = Pipeline([
16
    transport.input(),
17
    stt,
18
    llm,
19
    tts,
20
    transport.output(),
21
])

Configuration

Turn Detection (Key Feature)

AssemblyAI’s new turn detection model was built specifically for voice agents and you can tweak it to fit your use case. It processes both audio and linguistic information to determine an end of turn confidence score on every inference, and if that confidence score is past the set threshold, it triggers end of turn.

This custom model was designed to address 2 major issues with voice agents. With traditional VAD (voice activity detection) approaches based on silence alone, there are situations where the agent wouldn’t wait for a user to finish their turn even if the audio data suggested it. Think of a situation like “My credit card number is____” - if someone is looking that up, traditional VAD may not wait for the user, where our turn detection model is far better in these situations.

Additionally, in situations where we are certain that the user is done speaking like “What is my credit score?”, a high end of turn confidence is returned, greater than the threshold, and triggering end of turn, allowing for minimal turnaround latency in those scenarios.

You can set the vad_force_turn_endpoint parameter within the AssemblyAISTTService constructor:

1
stt = AssemblyAISTTService(
2
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
3
    vad_force_turn_endpoint=False,  # Use AssemblyAI's STT-based turn detection
4
    connection_params=AssemblyAIConnectionParams(
5
        end_of_turn_confidence_threshold=0.7,
6
        min_end_of_turn_silence_when_confident=160,  # in ms
7
        max_turn_silence=2400,  # in ms
8
    )
9
)

Parameter tuning:

• end_of_turn_confidence_threshold: Raise or lower the threshold based on how confident you’d like us to be before triggering end of turn based on confidence score
• min_end_of_turn_silence_when_confident: Increase or decrease the amount of time we wait to trigger end of turn when confident
• max_turn_silence: Lower or raise the amount of time needed to trigger end of turn when end of turn isn’t triggered by a high confidence score

For more information, see our Universal-Streaming end-of-turn detection guide and message-by-message breakdown.

Parameters

Constructor Parameters

Connection Parameters