Real-time speech recognition with Python

Real-time speech recognition in Python is powered by the AssemblyAI Python SDK. The SDK provides a RealTimeTranscriber that handles the complexities of WebSocket connections, and you capture live microphone audio with PyAudio and feed it to the transcriber. This feature, available on our free tier, processes audio as you speak.

We'll install PyAudio, then build an application that streams microphone audio to AssemblyAI's streaming endpoint.

Prerequisites

Before we start writing code, you'll need a few things:

Install the libraries using pip:

pip install assemblyai

Capturing microphone audio requires the pyaudio library. If you don't have it, install it as well:

pip install pyaudio

If PyAudio installation fails, install the PortAudio dependency first:

• macOS: brew install portaudio
• Windows: Download the appropriate .whl file for your system
• Linux: sudo apt-get install libasound-dev portaudio19-dev

Basic speech-to-text with Python

The simplest way to get started is by transcribing a local audio file. This confirms your environment and API key are set up correctly before we move to real-time streaming. While this article focuses on real-time, understanding the basic file upload method is a great first step.

You can use our API to transcribe a file by sending a POST request with the audio file's location. The API then returns a transcript object. This approach is ideal for pre-recorded audio, like interviews or podcasts.

import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"
transcriber = aai.Transcriber()

transcript = transcriber.transcribe("./my-local-audio-file.mp3")
print(transcript.text)

Real-time speech recognition setup

Now that you've seen how basic transcription works, let's build the real-time application. The AssemblyAI Python SDK provides a RealTimeTranscriber that makes it simple to transcribe live audio from a microphone. This is perfect for applications that need immediate voice input, like voice commands or live captioning.

The process involves four main steps:

1. Install the AssemblyAI SDK and import the required modules.
2. Define event handlers to process the transcription results.
3. Create and connect the RealTimeTranscriber.
4. Start streaming audio from the microphone by capturing it with PyAudio and passing each chunk to client.stream().

Connect to AssemblyAI's Realtime API

We'll use the RealTimeTranscriber to connect to AssemblyAI's real-time transcription service. The client works with event handlers, which are functions that you define to process events like the connection opening, receiving a transcript, or encountering an error.

First, let's define our event handlers and set up the client in a main function:

import logging

import pyaudio
from assemblyai.streaming.v3 import (
    BeginEvent,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    TerminationEvent,
    TurnEvent,
)

# Configure your API key
API_KEY = "YOUR_API_KEY"

SAMPLE_RATE = 16000
FRAMES_PER_BUFFER = 800  # 50ms of audio at 16kHz

# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Define event handlers
def on_begin(client: RealTimeTranscriber, event: BeginEvent):
    print(f"Session started: {event.id}")

def on_turn(client: RealTimeTranscriber, event: TurnEvent):
    print(f"{event.transcript} ({event.end_of_turn})")

def on_terminated(client: RealTimeTranscriber, event: TerminationEvent):
    print(f"Session terminated: {event.audio_duration_seconds} seconds of audio processed")

def on_error(client: RealTimeTranscriber, error: RealTimeError):
    print(f"Error occurred: {error}")

def main():
    # Create a RealTimeTranscriber
    client = RealTimeTranscriber(
        RealTimeTranscriberOptions(terminate_timeout=30.0),
        api_key=API_KEY,
    )

    # Attach event handlers
    client.on(RealTimeEvents.Begin, on_begin)
    client.on(RealTimeEvents.Turn, on_turn)
    client.on(RealTimeEvents.Termination, on_terminated)
    client.on(RealTimeEvents.Error, on_error)

    # Connect to the streaming service
    client.connect(
        RealTimeParameters(
            speech_model="universal-3-5-pro",
            sample_rate=SAMPLE_RATE,
            format_turns=True,
        )
    )

    print("Listening to microphone... Press Ctrl+C to stop.")

    # Capture microphone audio with PyAudio and stream it
    pa = pyaudio.PyAudio()
    mic = pa.open(
        format=pyaudio.paInt16,
        channels=1,
        rate=SAMPLE_RATE,
        input=True,
        frames_per_buffer=FRAMES_PER_BUFFER,
    )

    try:
        while True:
            client.stream(mic.read(FRAMES_PER_BUFFER, exception_on_overflow=False))
    except KeyboardInterrupt:
        pass
    finally:
        mic.stop_stream()
        mic.close()
        pa.terminate()
        # Ensure the client disconnects gracefully
        client.disconnect(terminate=True)

if __name__ == "__main__":
    main()

This single script handles everything: setting up the client, defining how to react to different events (like receiving a transcript turn), connecting, and streaming audio. PyAudio captures raw microphone audio in small chunks, and each chunk is passed to client.stream(), giving you full control over the audio pipeline.

Handle errors and connection issues

In a production application, network connections can be unreliable. The SDK's RealTimeTranscriber simplifies error handling. By attaching a function to the RealTimeEvents.Error event, you can gracefully manage unexpected issues without complex try...except blocks for WebSocket exceptions.

In our example, the on_error function will be called automatically if the connection drops or another session error occurs, printing the error to the console. For a production system, you could implement logic here to log the error to a monitoring service or attempt to reconnect.

Run your real-time speech recognition

With the complete script ready, you can run it from your terminal:

python your_script_name.py

Start speaking, and you'll see the transcribed text printed to your console in real time. The on_turn event handler processes each segment of speech as it's finalized. When you're finished, press Ctrl+C to stop. The try...finally block ensures that the microphone is closed and client.disconnect(terminate=True) is called, gracefully closing the session.

Optimize performance and accuracy

The default settings are a good starting point, but you can tune parameters for your specific use case. The sample_rate and turn detection parameters are key.

Here's a breakdown of the main turn detection parameters:

Experiment with these values in the RealTimeParameters object to find the right balance between responsiveness and allowing users to pause naturally.

Next steps with Voice AI

Your real-time speech recognition application is ready for production. Extend it with additional Voice AI capabilities:

Check the API documentation for implementation details. Try our API for free to get started.

Frequently asked questions

How do I fix PyAudio installation issues on different operating systems?

Install PortAudio first: brew install portaudio (macOS), sudo apt-get install libasound-dev portaudio19-dev (Linux), or download a pre-compiled wheel for Windows. This resolves most PyAudio installation issues.

What sample rate and audio format should I use for best accuracy?

Use 16,000 Hz sample rate with 16-bit format (pyaudio.paInt16) for optimal accuracy and performance.

How do I handle websocket disconnections and network errors?

The AssemblyAI Python SDK simplifies error handling. By defining an on_error event handler and attaching it to your RealTimeTranscriber, you can centralize your error logic. The SDK manages the underlying WebSocket connection state, so you don't need to manually catch specific connection errors or implement reconnection logic.

Why is my real-time transcription slow or inaccurate?

For latency, you can tune the turn detection parameters like end_of_turn_confidence_threshold to get faster responses. For accuracy, ensure you are providing a clean audio stream at a 16kHz sample rate. Poor audio quality, incorrect sample rates, or network issues are the most common causes of slow or inaccurate transcription.

Can I use AssemblyAI for offline speech recognition?

AssemblyAI requires an internet connection and doesn't offer offline solutions. You can transcribe local files via file upload, which doesn't need persistent streaming connections.