Now live New models for transcription and text-to-speech. AssemblyAI-STT-3.7 AssemblyAI-TTS-1
Insights & Use Cases

Best speech-to-text API for AI notetakers

What an AI notetaker needs from its speech-to-text API, how streaming speaker labels compare, and how to build one on AssemblyAI-STT-3.7-Realtime.

Abstract green mobius illustration

Written by

Kelsey Foster

Published on

18 November 2026

An AI notetaker makes three promises: it heard every word, it knows who said each one, and it turns the meeting into notes someone can act on. The first two belong to the speech-to-text API underneath. When it mislabels a speaker, the action item lands on the wrong person, and the summary model can't repair that because it never heard the audio.

So the best speech-to-text API for an AI notetaker is the one that gets speaker attribution right on long, messy, multi-person meetings, in real time, at a price that still works when every meeting runs an hour. For live notetakers, our pick is AssemblyAI-STT-3.7-Realtime (assemblyai-stt-3-7-realtime), the streaming model built for notetakers and ambient transcription, at $0.30 per hour.

This post is for teams building a notetaker. If you're choosing a notetaker app to use, our roundup of the best AI notetakers compares the products. Here we cover the layer underneath: what a notetaker needs from its speech-to-text API, how speaker labels are measured, how the field compares today, and a working Python build.

What an AI notetaker needs from a speech-to-text API

Voice agents and notetakers both stream audio, but they stress a model in different ways. An agent hears one caller in short turns and has to decide fast when the caller is done. A notetaker hears four to ten people for an hour, and nobody is waiting on a reply. That changes the checklist.

  1. Speaker labels in the stream. Live notes, "what did Priya just commit to" lookups and in-call action items all need to know who is talking while the meeting is still going. A model that only attributes speakers after the recording ends can't power live features.
  2. Labels you don't have to rewrite. If labels get revised after the fact, your UI has to repaint lines that users already read, and anything you've sent downstream has to be corrected. Labels that are final when the turn ends keep the pipeline simple.
  3. Names, numbers and jargon. Meetings are dense with attendee names, product names, ticket numbers and dates. You want a way to tell the model which terms matter, such as a keyterm list built from the calendar invite.
  4. Many languages in one meeting. Global teams switch languages mid-sentence. The model should follow without a restart.
  5. Long sessions and clear billing. A one-hour meeting is a one-hour WebSocket. Know how you're billed (session time or audio time) and how sessions end cleanly.
  6. A path from transcript to notes. Summaries, decisions and action items come from an LLM pass over the labeled transcript. Fewer moving parts means fewer keys and bills to manage.
  7. Security and data handling. Meetings carry customer names, deal terms and sometimes health or financial details. Look for PII redaction, data-retention controls and contracts your customers' security reviews will ask for.
Stream With Speaker Labels Today

Turn on speaker labels with AssemblyAI-STT-3.7-Realtime and see who said what while the meeting is still going. Start with a free account and clear docs.

Sign up free

Why speaker labels decide notetaker quality

Word error rate (WER) tells you whether the words are right. It says nothing about whether they're attached to the right person. Two other metrics cover that:

  • Diarization error rate (DER) is the share of audio time given to the wrong speaker, missed, or wrongly marked as speech.
  • Concatenated minimum-permutation WER (cpWER) scores words and speaker attribution together. It matches each predicted speaker to the best reference speaker and counts every word that's wrong or under the wrong name.

For a notetaker, cpWER is closer to what users see: a line of text under a name. A model with low WER and high cpWER produces clean sentences attributed to the wrong people. If you want to run the metric on your own meetings, here's how to measure diarization accuracy with cpWER in Python. For the mechanics of labeling speakers live, see our explainer on streaming speaker diarization.

AssemblyAI-STT-3.7-Realtime: the streaming model for notetakers

AssemblyAI-STT-3.7-Realtime is one of two models in the STT 3.7 streaming family. The docs recommend it for notetakers and ambient transcription, and AssemblyAI-STT-3.7-Realtime-Agent (assemblyai-stt-3-7-realtime-agent) for voice agents. Both run on the same Streaming Speech-to-Text API endpoint, so moving between them is a one-string change.

What a notetaker needs AssemblyAI-STT-3.7-Realtime
Model ID assemblyai-stt-3-7-realtime on wss://streaming.assemblyai.com/v3/ws
Price $0.30 per hour of session time
Languages 32
Speaker labels in the stream speaker_labels: true. A speaker_label on every final turn and a speaker on every final word. Labels are final when the turn ends.
Speaker cap Optional max_speakers, 1 to 10
Names and jargon Keyterm prompting (keyterms_prompt), a contextual prompt, and context carryover
Turn endings Content-aware endpointing: punctuation ends a turn, and a turn that ends on an entity is held
Data protection PII redaction in the stream
Live translation Per-turn translation through the llm_gateway connection parameter

Speaker labels: the numbers

From the beginning of a new session, STT-3.7-Realtime's speaker labels are final the moment a turn ends. There's no revision pass and no SpeakerRevision message to wait for, so a live view can show who said what without repainting lines later.

Measured against the field on the same recordings, AssemblyAI-STT-3.7-Realtime has the lowest live diarization error rate (DER) of every realtime provider measured on telephone calls and on multi-party meetings recorded on a single distant microphone, ahead of ElevenLabs Scribe v3 Live and Meta Muse Voice Transcribe:

  • 10.46% DER on CallHome, 40 unscripted telephone calls.
  • 6.30% DER on NOTSOFAR, 129 real meetings recorded on a single distant microphone with up to seven speakers.

Word accuracy holds up too. On short-form English its word error rate is 3.23%, ahead of ElevenLabs Scribe v3 Live, AWS Transcribe Streaming and Deepgram Flux, and on code-switched audio it is 6.67%, first on all five language pairs measured.

How the field compares on streaming speaker labels

Here are the published live diarization results from the AssemblyAI-STT-3.7 launch post. Each vendor's realtime API ran at default settings on the same recordings, with runs from August to November 2026. Lower is better:

System Live DER, CallHome (telephone calls) Live DER, NOTSOFAR (single distant mic meetings)
AssemblyAI-STT-3.7-Realtime 10.46% 6.30%
ElevenLabs Scribe v3 Live 10.78% 9.63%
Meta Muse Voice Transcribe 16.44% 7.34%
Speechmatics Enhanced Realtime 14.58% 25.74%
Azure Realtime 20.21% 19.19%
AWS Transcribe Streaming 27.89% 23.79%
Deepgram Nova-3 30.46% 26.37%
Soniox Realtime v6 18.35% 32.89%

Google Gemini 3.5 Transcribe Live, Cartesia Ink Whisper and Gladia Solaria return no speaker labels on streaming, so they have no score here. The public benchmarks page has the full streaming diarization results. Use both as a shortlist, then test on your own recordings.

Live, post-call, or both

Many notetakers run two passes. They stream during the meeting for live captions and in-call features, then send the recording through pre-recorded transcription afterward for the archive. Our meeting notetaker best practices recommend this hybrid, and our guide to real-time vs. batch transcription covers the trade-offs in depth.

Here's how the two passes add up on your bill, per hour of meeting, at the list prices today:

Pass Model List price per meeting hour
Live, during the meeting AssemblyAI-STT-3.7-Realtime $0.30 (session time)
Post-call, on the recording Universal-3.5 Pro + speaker diarization $0.21 + $0.02 = $0.23
Both passes $0.53

Streaming is billed on how long the WebSocket stays open, not on how much audio you send. Close every session when the meeting ends, and don't leave idle connections running between calls.

Whether you need the second pass depends on your product. If your users read the live transcript and the notes generated from it, start with streaming alone and add the post-call pass when you have a feature that needs it, such as named-speaker identification on the archive.

Build a live notetaker in Python

The build below streams meeting audio to STT-3.7-Realtime with speaker labels on, prints each finished turn with its label, then asks LLM Gateway for notes. It uses the Python SDK (pip install assemblyai==1.10.1 requests). Version 1.10.1 is the first SDK release that lists the 3.7 model IDs. The parameters match the streaming diarization docs.

import os

import requests
from assemblyai.streaming.v3 import (
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    TurnEvent,
)

API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
turns = []


def on_turn(client: RealTimeTranscriber, event: TurnEvent):
    # Partials arrive while someone talks; the final turn carries the label.
    if event.end_of_turn and event.transcript:
        speaker = event.speaker_label or "PENDING"
        turns.append(f"Speaker {speaker}: {event.transcript}")
        print(f"[{speaker}] {event.transcript}")


client = RealTimeTranscriber(
    RealTimeTranscriberOptions(terminate_timeout=30.0),
    api_key=API_KEY,
)
client.on(RealTimeEvents.Turn, on_turn)

client.connect(
    RealTimeParameters(
        speech_model="assemblyai-stt-3-7-realtime",
        sample_rate=16000,
        speaker_labels=True,
        max_speakers=6,  # hard cap; leave headroom above the attendee count
        keyterms_prompt=["Priya Raman", "Kubernetes", "Q3 roadmap"],
    )
)

try:
    for chunk in meeting_audio():  # your source: 16-bit PCM, ~50 ms chunks
        client.stream(chunk)
finally:
    # Finalizes the open turn and closes the billed session.
    client.disconnect(terminate=True)

notes = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers={"authorization": API_KEY},
    json={
        "model": "claude-sonnet-4-6",
        "messages": [{
            "role": "user",
            "content": "Summarize this meeting, list decisions, and list "
                       "action items with the owner's speaker label:\n\n"
                       + "\n".join(turns),
        }],
        "max_tokens": 1000,
    },
    timeout=60,
).json()["choices"][0]["message"]["content"]
print(notes)

meeting_audio() stands in for your capture layer: a meeting bot, a desktop recorder or a microphone. The companion repo implements it for a WAV file streamed at real-time pace, merges consecutive turns from the same speaker, and writes the transcript and notes to disk.

A few details from the docs that matter in production:

  • PENDING labels. A turn with less than about one second of audio comes back labeled PENDING, because there isn't enough audio for a reliable speaker embedding. Short backchannels ("yeah", "mm-hm") are the usual case. Don't assign action items to them.
  • max_speakers is a hard cap. Extra speakers are merged into the closest existing label. Set it a little above the number of attendees on the invite.
  • Labels settle over the session. The first one or two turns can be misassigned while the model builds speaker profiles. Overlapping speech goes to a single speaker.
  • Word-level speakers. Each final word carries its own speaker field, so you can catch a mid-turn interjection without waiting for the next turn.

For the full event sequence, see the real-time meeting assistant example in the docs.

Try It On Your Own Audio

Run a real meeting recording through the playground and check the transcript before you write any code.

Try playground

From transcript to notes

The last step is an LLM pass over the labeled transcript. LLM Gateway gives you an OpenAI-compatible endpoint for many models under the same API key you use for speech-to-text, which keeps the notetaker on one vendor contract. Two prompt habits help:

  • Pass speaker labels through, and tell the model to ignore PENDING when it assigns owners. Once your app maps labels to names (from the invite or a quick "who's who" prompt in the UI), substitute names before the call.
  • Generate notes per section for long meetings. For a two-hour session, summarize in chunks of about 20 minutes, then summarize the summaries.

If you'd rather translate each turn live, the same WebSocket takes an llm_gateway connection parameter, and STT-3.7-Realtime supports it. The translation comes back on the socket as each turn finishes.

How to evaluate a speech-to-text API for your notetaker

Run this on 10 to 20 of your own recordings before you commit to any provider. Use real meetings with crosstalk, accents, far-field microphones and your product's vocabulary.

  1. Score speaker attribution as well as words. Compute cpWER and DER against a hand-labeled reference.
  2. Watch the first two minutes and the last ten. Early-session label stability and long-session drift are where streaming diarization differs most.
  3. Count wrong names and numbers. Then add a keyterm list from the calendar invite and count again.
  4. Test your language mix. Include at least one meeting where people switch languages mid-sentence.
  5. Load-test session starts. Open as many concurrent sessions as your busiest hour needs. On AssemblyAI, streaming limits apply to new sessions per minute: paid accounts start at 200, and the limit rises 10% whenever you use 80% or more of it.
  6. Check the paperwork. PII redaction options, data retention, SOC 2, and a Business Associate Addendum (BAA) if any customer processes protected health information.

For the full field across voice agents, contact centers and captions, see our ranking of the best real-time speech-to-text APIs. For the product side of meeting notes, see the AI notetakers use case.

Final words

Notetaker buyers rarely ask which speech-to-text API sits underneath, but they feel the answer every time a decision gets credited to the wrong person. That's why the speech layer deserves more scrutiny than the summary prompt. STT-3.7-Realtime puts final speaker labels in the stream at $0.30 per hour, so live features and post-call notes can come from the same labeled transcript. On live diarization it leads every realtime provider measured on telephone calls and single-microphone meetings. Even so, the most useful test is your own: twenty real meetings, scored on who said what.

Build Your Notetaker On STT-3.7-Realtime

Get real-time speaker labels in 32 languages at $0.30 per hour, plus LLM Gateway for meeting notes on the same API key.

Sign up free

Frequently Asked Questions (FAQs)

What is the best speech-to-text API for AI notetakers?

For live notetakers, AssemblyAI recommends AssemblyAI-STT-3.7-Realtime (assemblyai-stt-3-7-realtime), a streaming model built for notetakers and ambient transcription with real-time speaker labels in 32 languages at $0.30 per hour. It has the lowest live diarization error rate of every realtime provider measured on telephone calls (10.46% on CallHome) and single-distant-mic meetings (6.30% on NOTSOFAR), ahead of ElevenLabs Scribe v3 Live and Meta Muse Voice Transcribe. Test any API on your own meetings, scored on speaker attribution as well as words.

Is there a speech-to-text API with real-time speaker labels?

Yes. AssemblyAI's Streaming API returns a speaker_label on every final turn when you set speaker_labels: true, and a speaker field on each final word. It's supported on AssemblyAI-STT-3.7-Realtime, AssemblyAI-STT-3.7-Realtime-Agent and Universal-3.6 Pro Realtime, and the docs recommend STT-3.7-Realtime for diarization.

Should an AI notetaker use streaming or pre-recorded transcription?

Use streaming when users need a live transcript, in-call notes or captions, and pre-recorded transcription when you only process recordings after the meeting. Many notetakers run both: streaming during the call and a post-call pass for the archive. With STT-3.7-Realtime's final speaker labels, some products can start with streaming alone.

How much does a speech-to-text API for a notetaker cost?

AssemblyAI-STT-3.7-Realtime costs $0.30 per hour of streaming session time. If you also run a post-call pass, Universal-3.5 Pro is $0.21 per hour plus $0.02 per hour for speaker diarization. Streaming bills on how long the WebSocket stays open, so close each session when the meeting ends.

What's the difference between AssemblyAI-STT-3.7-Realtime and STT-3.7-Realtime-Agent?

STT-3.7-Realtime is built for notetakers, meetings and live captions, at $0.30 per hour. STT-3.7-Realtime-Agent is built for voice agents, adding Voice Focus and a manual turn-detection mode, at $0.45 per hour. Both support speaker labels, keyterm prompting and 32 languages.

How do I build an AI notetaker with AssemblyAI?

Open a streaming session with speech_model: "assemblyai-stt-3-7-realtime" and speaker_labels: true, send 16-bit PCM audio from your meeting bot or recorder, and collect each final turn with its speaker label. When the meeting ends, send the labeled transcript to LLM Gateway for a summary, decisions and action items. The Python example in this post does both steps with the AssemblyAI SDK.