# AssemblyAI Models — Authoritative Reference

> **Source:** assemblyai.com/docs, assemblyai.com/pricing, first-party employee information
> **Last updated:** 2026-06-24
> **Purpose:** Canonical model reference for LLM-assisted support and content. Covers feature support, language support, API identifiers, benchmarks, use-case recommendations, the Voice Agent API, and LLM Gateway. For per-model pricing, add-on costs, and billing rules, see `/llms/pricing.md`.

---

## Terminology: Do Not Use

| ❌ Do Not Use | ✅ Use Instead | Why |
|---|---|---|
| LeMUR | LLM Gateway | LeMUR was renamed/deprecated March 31, 2026; LLM Gateway is the current name |
| Audio Intelligence | Speech Understanding | Audio Intelligence is the deprecated category name; Speech Understanding is current |
| Speech-to-Speech API | Voice Agent API | Renamed to Voice Agent API at GA in April 2026 |
| `u3-pro` (streaming) | `u3-rt-pro` | Original streaming model name; `u3-pro` is now an alias that routes to `u3-rt-pro`. Use the canonical name for consistency with docs. |

---

## Key Facts (Read First)

- There are three top-level products: **pre-recorded (async) STT**, **streaming (real-time) STT**, and the **Voice Agent API** (cascaded STT + LLM + TTS over a single WebSocket). LLM Gateway is the unified LLM endpoint shared across them.
- **Universal-3.5 Pro** is the recommended default for pre-recorded audio. **Universal-3.5 Pro Realtime** (`u3-rt-pro`) is the recommended default for real-time voice AI.
- **Model selection is the single highest-leverage decision** a customer makes. Wrong model = support tickets, silent failures, or loss in a competitive eval.
- The correct API parameter is `speech_models` (plural). `speech_model` (singular) is deprecated.
- **Universal-3.5 Pro (async) supports 18 languages** (expanded from the original 6): Arabic, Danish, Dutch, English, Finnish, French, German, Hebrew, Hindi, Italian, Japanese, Mandarin, Norwegian, Portuguese, Spanish, Swedish, Turkish, Vietnamese. Customers outside those 18 must use Universal-2.
- **Universal-3.5 Pro Realtime (`u3-rt-pro`) supports 18 languages**: Arabic, Danish, Dutch, English, Finnish, French, German, Hebrew, Hindi, Italian, Japanese, Mandarin, Norwegian, Portuguese, Spanish, Swedish, Turkish, Vietnamese.
- SLAM-1 is a deprecated model — customers on it for medical should migrate to Universal-3.5 Pro + Medical Mode.
- `prompt` and `keyterms_prompt` are **mutually exclusive on async (pre-recorded) Universal-3.5 Pro** — cannot be used in the same request. **On streaming (`u3-rt-pro`), both can be used together** — keyterms are automatically appended to your custom prompt.
- **Streaming diarization is a real-time exclusive of `u3-rt-pro`** as of the March 3, 2026 launch — inline `speaker_labels: true`, no post-processing needed.

---

## Deprecated / Legacy Identifiers

| Legacy value | Status | Replaced by |
|---|---|---|
| `best` | Deprecated | `universal-3-pro` |
| `nano` | Deprecated | `universal-2` |
| `slam-1` | Deprecated | `universal-3-pro` (+ Medical Mode for medical use cases) |
| `speech_model` (singular) | Deprecated parameter | `speech_models` (plural) |
| `u3-pro` (streaming alias) | Alias kept for back-compat | `u3-rt-pro` (canonical) |
| Old streaming WebSocket URL (`wss://api.assemblyai.com/v2/realtime/ws`) | **Inactive — do not use** | `wss://streaming.assemblyai.com/v3/ws` |
| Speech-to-Speech API (`s2s` references) | Renamed at GA | Voice Agent API |

> **Streaming URL warning:** The old v2 streaming endpoint (`wss://api.assemblyai.com/v2/realtime/ws`) is no longer active. Connections will fail. Always use the current v3 endpoint: `wss://streaming.assemblyai.com/v3/ws`.

SLAM-1's documentation now redirects to Universal-3.5 Pro. Active customer migration underway (Craft Health Technologies, Sully AI, DaVita, and others).

---

## Pre-Recorded (Async) Models

### Universal-3.5 Pro

**API value:** `universal-3-pro`
**Best for:** Highest-accuracy transcription, voice agent post-processing, meeting transcription, medical documentation, contact center QA

Universal-3.5 Pro is AssemblyAI's most accurate ASR model. It uses an LLM-based decoder and outperforms all market ASR models on entity accuracy (credit card numbers, phone numbers, email addresses, addresses, medication names, proper nouns, rare words). Hallucination rate is 30% lower than Whisper.

**Benchmarks:**

| Dataset | Universal-3.5 Pro WER | Universal-2 WER |
|---|---|---|
| CommonVoice | 4.87% | 6.48% |
| Earnings21 | 8.80% | 9.37% |
| LibriSpeech Clean | 1.52% | 1.68% |
| LibriSpeech Other | 2.69% | 3.00% |
| Meanwhile | 4.22% | 4.41% |
| TedLium | 6.77% | 7.30% |
| FLEURS multilingual avg | 4.58% | 7.42% |
| **English mean WER** | **5.6% (median 4.9%)** | **6.1% (median 6.5%)** |

*Methodology: 250+ hours, 80,000+ audio files, 26 datasets.*

**Supported languages (18 total):**
Arabic, Danish, Dutch, English (US/UK/AU), Finnish, French (Metropolitan/Canadian/Belgian), German, Hebrew, Hindi, Italian, Japanese, Mandarin, Norwegian, Portuguese (Brazilian/European), Spanish (Castilian/MX/AR/CO/CL/Caribbean/Spanglish), Swedish, Turkish, Vietnamese. Regional dialects auto-recognized from base language code.

**Feature support:**

| Feature | Supported | Notes |
|---|---|---|
| General Prompting | ✅ | Up to 1,500 words of natural language instruction (add-on) |
| Keyterms Prompting | ✅ | Up to 1,000 terms (max 6 words each); add-on |
| ⚠️ Prompt + Keyterms together (async) | ❌ | **Mutually exclusive on async** — cannot use both in one pre-recorded request |
| Temperature control | ✅ | 0.0–1.0 (default 0.0); low non-zero values yield ~5% relative improvement |
| Medical Mode | ✅ | Add `"domain": "medical-v1"`; EN, ES, DE, FR |
| Speaker Diarization (standard) | ✅ | Add-on |
| Speaker Diarization (experimental) | ✅ | Add-on; better for high speaker count or difficult audio |
| Speaker Identification | ✅ | Add-on |
| Entity Detection | ✅ | 50+ entity types; add-on |
| Sentiment Analysis | ✅ | Add-on |
| Topic Detection (IAB) | ✅ | Add-on |
| Key Phrases / Auto Highlights | ✅ | English only; add-on |
| Content Safety / Moderation | ✅ | Add-on |
| Translation | ✅ | 100+ target languages; pre-recorded only; add-on |
| Custom Formatting | ✅ | Add-on |
| PII Redaction (text) | ✅ | Add-on |
| PII Redaction (audio) | ✅ | Add-on |
| Profanity Filtering | ✅ | Add-on |
| Custom Spelling | ✅ | |
| Code Switching | ✅ | Across all 18 supported languages |
| Automatic Language Detection | ✅ | 130+ detectable languages; requires 15+ seconds of audio |
| Multichannel | ✅ | Billed per channel |
| Audio Event Tagging | ✅ | Over 100 different audio tags, including `[laughter]`, `[music]`, `[applause]`, `[beep]`, `[silence]`, and more |
| Disfluencies (`disfluencies` param) | ❌ | Use prompting instead (e.g., "tag filler words like um and uh") |
| `remove_audio_tags` | ✅ | Set to `"all"` to strip all `[...]` tags (audio event + speaker tags) from output post-inference. Opt-in. |
| Auto Chapters | ❌ | **Causes silent 500 error on U3 Pro.** Use LLM Gateway instead. |
| Summarization | ❌ | **Same failure behavior as auto_chapters.** Use LLM Gateway instead. |
| Word Boost (`word_boost`) | ❌ | **Silently falls back to Universal-2** if set alongside U3 Pro. Use keyterms prompting instead. |

**Critical gotchas:**
- `auto_chapters` and `summarization` do not return helpful errors on U3 Pro — the job silently hangs and eventually returns a generic 500. This is a top support issue.
- `word_boost` with `speech_models: ["universal-3-pro"]` silently routes to Universal-2. Customers think they're on U3 Pro but are not. Use keyterms prompting instead.
- Using the word "speaker" in a prompt can inject inline speaker attribution labels (`[Speaker:NAME]`) into the transcript text. Use `speaker_labels: true` for diarization instead.
- Silent model routing: if a customer specifies U3 Pro but submits audio in a language outside the 18 supported, the API may silently fall back to Universal-2 without error. Use `speech_models: ["universal-3-pro", "universal-2"]` to make this routing explicit.

---

### Universal-2

**API value:** `universal-2`
**Best for:** 99-language coverage, legacy integrations, features not yet available on U3 Pro (auto_chapters, summarization, disfluencies)

Universal-2 is the go-to model when U3 Pro's 19-language coverage doesn't apply, or when the integration requires features not yet ported to U3 Pro. Described internally as "highly accurate, fastest performing model with support across 99 languages."

**Supported languages:** 99+ (full list at assemblyai.com/docs)

**Language accuracy tiers:**

| Tier | Approx. WER | Example languages |
|---|---|---|
| High (≤10%) | — | English, Spanish, French, German, Indonesian, Italian, Japanese, Dutch, Polish, Portuguese, Russian, Swedish, Turkish, Ukrainian, Catalan |
| Good (10–25%) | — | Arabic, Bulgarian, Mandarin Chinese, Czech, Danish, Finnish, Hebrew, Hindi, Hungarian, Korean, Malay, Norwegian, Romanian, Slovak, Thai, Vietnamese |
| Moderate (25–50%) | — | Afrikaans, Welsh, Persian, Icelandic, Kazakh, Lithuanian, Latvian, Maori, Marathi, Swahili, Tamil |
| Fair (>50%) | — | Amharic, Bengali, Gujarati, Hausa, Javanese, Georgian, Khmer, Kannada, Mongolian, Burmese, Nepali, Punjabi, Somali, Telugu, Yoruba |

**Feature support:**

| Feature | Supported | Notes |
|---|---|---|
| General Prompting | ❌ | Universal-3.5 Pro exclusive |
| Keyterms Prompting | ✅ | Up to 200 terms (add-on) |
| Temperature control | ❌ | Universal-3.5 Pro exclusive |
| Medical Mode | ✅ | EN, ES, DE, FR (optimized for U3 Pro) |
| Word Boost (`word_boost`) | ✅ | Legacy vocabulary boosting |
| Speaker Diarization (standard) | ✅ | Add-on |
| Speaker Diarization (experimental) | ✅ | Add-on |
| Speaker Identification | ✅ | Add-on |
| Entity Detection | ✅ | 50+ entity types; add-on |
| Sentiment Analysis | ✅ | Add-on |
| Topic Detection (IAB) | ✅ | Add-on |
| Key Phrases / Auto Highlights | ✅ | English only; add-on |
| Content Safety / Moderation | ✅ | Add-on |
| Translation | ✅ | 100+ target languages; pre-recorded only; add-on |
| Custom Formatting | ✅ | Add-on |
| PII Redaction (text) | ✅ | 47+ languages; add-on |
| PII Redaction (audio) | ✅ | Add-on |
| Profanity Filtering | ✅ | Add-on |
| Custom Spelling | ✅ | |
| Code Switching | ✅ | Up to 2 language codes; one must be English; best pairs: EN/ES, EN/DE |
| Automatic Language Detection | ✅ | 130+ languages; requires 15+ seconds of audio |
| Multichannel | ✅ | Billed per channel |
| Audio Event Tagging | ❌ | Universal-3.5 Pro only (async) |
| Disfluencies / Filler Words | ✅ | Returns filler word tokens in transcript |
| Auto Chapters | ✅ | Deprecated — use LLM Gateway |
| Summarization | ✅ | Deprecated — use LLM Gateway |

**Language routing tip:** Use `speech_models: ["universal-3-pro", "universal-2"]` to get U3 Pro for its 18 supported languages and automatic fallback to Universal-2 for all others. The `speech_model_used` field in the response indicates which model processed the request.

---

### SLAM-1 (Deprecated)

**API value:** `slam-1` — **do not use for new integrations**
**Status:** Deprecated — migrate to Universal-3.5 Pro + Medical Mode

SLAM-1 was AssemblyAI's LLM-based decoder model, historically used for medical transcription and telephony. Universal-3.5 Pro supersedes it with better entity accuracy, prompting, and Medical Mode. Documentation now redirects to Universal-3.5 Pro.

SLAM-1–specific capability not replicated elsewhere:
- Audio event tags (`<hold music>`, `<beep>`, `<background noise>`) — used in some telephony QA workflows. Universal-3.5 Pro has a different audio event tagging format (with 100+ tags).

---

## Streaming (Real-Time) Models

All streaming models use WebSocket. v3 endpoint: `wss://streaming.assemblyai.com/v3/ws`

---

### Universal-3.5 Pro Realtime

**API value:** `u3-rt-pro` (alias `u3-pro` accepted for back-compat)
**Launched:** March 3, 2026 (GA).
**Best for:** Voice agents, real-time agent assist, contact center live coaching, any real-time application requiring highest accuracy on EN/ES/DE/FR/IT/PT

This is the flagship streaming model and the definitive recommendation for all voice AI / voice agent use cases. Per internal GTM directive: *"If you see any voice agent prospects evaluating us and using universal-streaming, we will lose. We need to make sure every voice agent customer is only evaluating us on universal-3-pro."* (Dylan Fox, Head of Sales)

**Supported languages (18):** Arabic, Danish, Dutch, English, Finnish, French, German, Hebrew, Hindi, Italian, Japanese, Mandarin, Norwegian, Portuguese, Spanish, Swedish, Turkish, Vietnamese — with native code switching.

**Latency:** P50 ~150ms post-VAD, P90 ~240ms post-VAD; sub-200ms end-to-end is the launch headline; P50 time-to-complete-transcript ~250ms

**Feature support:**

| Feature | Supported | Notes |
|---|---|---|
| Keyterms Prompting | ✅ | Included at no extra cost; up to 100 terms; updatable mid-stream |
| General Prompting | ✅ (beta) | Add-on |
| Medical Mode | ✅ | EN, ES, DE, FR |
| Speaker Diarization (real-time) | ✅ | **Streaming-exclusive capability launched with U3.5 Pro Realtime.** Inline `speaker_labels: true`. No post-processing required. Add-on. |
| Entity Recognition | ✅ | Credit cards, phone numbers, emails, addresses, names |
| Audio Event Tagging | ✅ | Non-speech detection (100+ tags) |
| Disfluency / verbatim control | ✅ | Via prompting |
| Mid-stream config updates | ✅ | UpdateConfiguration messages (prompt, keyterms, silence params) |
| End-of-turn detection | ✅ | Punctuation-based (replaces deprecated confidence threshold) |
| Code-switching (intra-utterance) | ✅ | Native across all 6 supported languages in a single stream |

**Turn detection parameters:**
- `min_turn_silence`: default 100ms (triggers end-of-turn punctuation check)
- `max_turn_silence`: default 1000ms (forces turn end regardless)
- `vad_threshold`: default 0.3

**Why it wins voice agent evals:**
- Best entity accuracy: credit card numbers, phone numbers, email addresses, physical addresses, proper nouns
- Short utterance handling ("yes," "no," "mmhmm") — the #1 pain point for voice agent customers
- Optimized word emission timing for low-latency turn detection
- Native code switching across 6 supported languages (EN/ES/DE/FR/IT/PT) in a single stream
- Real-time speaker diarization with no post-processing
- Keyterms included for domain-specific vocabulary

**Known friction points:**
- `language_code` parameter is **silently ignored** — language is auto-handled; do not rely on it
- `end_of_turn_confidence_threshold` is **deprecated** — migrating from Universal-Streaming requires rewriting turn detection logic (punctuation-based, not confidence-based)
- Low `min_turn_silence` values can split entities (e.g., phone numbers interrupted mid-sequence)
- Hallucinations on very short (3–10s), low-entropy utterances — flagged by Decagon and HotelPlanner during alpha
- Latency is slightly higher than Universal-Streaming English at P75/P90/P95 — the trade-off for significantly better accuracy
- German–English code switching had reported issues during early access (Retell)
- Phone number formatting: spoken digits may be reformatted ("993-541-1436" instead of spoken-word form) — flagged by Concentrix
- Emission latency: U3 Pro only emits finals after a silence threshold, which can feel slower than competitors for live note-taker products (raised by Granola and Notion in Apr 2026 pipeline calls)

**Customer examples:** Decagon, Lorikeet, Bland AI, Phonely, Simple AI, Vapi, Synthflow, Grotto, Zowie, Retell, Sierra, LiveKit integrations

---

### Universal-Streaming English

**API value:** `universal-streaming-english`
**Best for:** English-only real-time applications where lowest latency is the priority; legacy integrations from before U3.5 Pro Realtime launched

**Supported languages:** English only

| Feature | Supported | Notes |
|---|---|---|
| Keyterms Prompting | ✅ | Up to 100 terms (add-on) |
| General Prompting | ❌ | |
| Medical Mode | ✅ | English only |
| Speaker Diarization | ✅ | Add-on |

**Note:** This was the default streaming model for voice agents before Universal-3.5 Pro Realtime launched. New voice agent customers should be on `u3-rt-pro`. Existing customers on this model should be proactively migrated.

---

### Universal-Streaming Multilingual

**API value:** `universal-streaming-multilingual`
**Best for:** Real-time multilingual applications covering the 6-language set where U3.5 Pro Realtime is not available or not needed; broader compatibility use cases

**Supported languages:** English, Spanish, German, French, Portuguese, Italian — code switching per turn (not intra-utterance like `u3-rt-pro`)

| Feature | Supported | Notes |
|---|---|---|
| Keyterms Prompting | ✅ | Up to 100 terms (added Dec 2025) |
| General Prompting | ❌ | |
| Medical Mode | ✅ | EN, ES, DE, FR |
| Speaker Diarization | ✅ | Add-on |
| Automatic Language Detection | ✅ | Returns `language_code` and `language_confidence` per utterance |
| Built-in formatting | ✅ | Punctuation + capitalization included by default |

---

## Voice Agent API

**Docs:** https://www.assemblyai.com/docs/voice-agents/voice-agent-api
**WebSocket endpoint:** `wss://agents.assemblyai.com/v1/ws`
**Launched:** April 2026 (formerly **Speech-to-Speech API** during alpha)
**Architecture:** **Cascaded** — STT → LLM → TTS in a single persistent WebSocket. *Not* a unified speech-to-speech model. AssemblyAI designed a custom STT-first cascade around `u3-rt-pro`, including an ensemble of STT streams with different prompts to extract richer information for turn-taking and accuracy.
**Built on:** Universal-3.5 Pro Realtime (`u3-rt-pro`) for STT, LLM Gateway for reasoning, text-to-speech running on self-hosted LiveKit infrastructure.
**Compliance:** PCI-certified.

**Authentication:** `Authorization: Bearer YOUR_API_KEY` HTTP upgrade header, or temporary token as query parameter for client-side apps.

**Audio:** PCM16 at 24 kHz sample rate (input + output).

**What it bundles:**
- Speech-to-text (Universal-3.5 Pro Realtime)
- LLM reasoning (any LLM Gateway model)
- Text-to-speech (18 English voices including regional variants; 16 multilingual voices spanning Hindi, Mandarin, Russian, German, French, Korean, Japanese, Italian, Spanish)
- Turn detection (server-side end-of-turn detection)
- Interruption handling — **semantic barge-in** built in: back-channels like "uh-huh" do not interrupt the agent, but explicit interruptions like "wait, stop" do.
- Tool calling — register tools and handle `tool.call` / `tool.result` events
- Single WebSocket — no stitching together five separate APIs

**Configuration parameters:**
- **System Prompt:** custom instructions for agent behavior
- **Greeting:** initial message to user
- **Voice Output:** selectable TTS voice
- **Server-side noise cancellation:** enabled by default
- **Tool registration:** declarative tool schema, runtime tool result return

**Session management:** Sessions are preserved for **30 seconds** after disconnection. Reconnect using `session.resume` with the previous `session_id`.

**Best for:** The simplest path to a production voice agent. Use it when you want AssemblyAI to manage the orchestration end-to-end.

**When to use this vs. building your own stack:**

| Use Voice Agent API | Use Universal-3.5 Pro Realtime + bring your own |
|---|---|
| You want a single connection, end-to-end | You want to swap LLMs, TTS providers, or orchestrators independently |
| You don't want to manage LiveKit/Pipecat directly | You're already deeply integrated with LiveKit, Pipecat, Vapi, etc. |
| You need PCI-certified end-to-end out of the box | You have your own compliance posture |
| You want AssemblyAI's tuned turn-taking + barge-in defaults | You need fully custom orchestration / tool routing |
| You want the included noise cancellation + voice catalog | You're using your own TTS / noise stack |

**Roadmap:**
- **Twilio integration** (Q2 2026): direct Twilio SIP and voice connectivity, phone integration without a customer-managed Twilio account.

**Comparison vs. OpenAI Realtime API:** Same single-WebSocket abstraction, but optimized for speech accuracy on real-world entities (numbers, addresses, names) and cost predictability via per-hour billing.

For pricing, see `/llms/pricing.md`.

---

## LLM Gateway

**Relaunched:** April 13, 2026
**Endpoint:** Single unified endpoint, authenticated with your existing AssemblyAI API key. Provider-agnostic — call any model from one endpoint.
**Use cases:** Drives Voice Agent API reasoning, post-call summarization, chapter generation, agentic workflows, any LLM call your AssemblyAI app needs to make.

**What's new in the April 2026 relaunch:**
- **Automatic fallback across providers** — provider outage routing with zero added latency, no client-side retry logic
- **Real-time streaming with tool calling** — supported on `gpt-5-nano`, `kimi-k2.5`, and `qwen3` models
- **Structured JSON outputs** — supported on Claude 4.5+ models
- **Prompt caching** — across Anthropic, OpenAI, and Google
- **New providers added:** Alibaba Qwen 3, Moonshot AI Kimi K2.5
- **Built-in JSON repair** post-processing
- **20+ production models** at GA, now 25+

### Supported model families (May 2026)

**Anthropic Claude:**
- `claude-opus-4-7` (Claude Opus 4.7)
- `claude-opus-4-6` (Claude Opus 4.6)
- `claude-opus-4-5-20251101` (Claude Opus 4.5)
- `claude-sonnet-4-6` (Claude Sonnet 4.6)
- `claude-sonnet-4-5-20250929` (Claude Sonnet 4.5)

**OpenAI GPT:**
- `gpt-5.5`, `gpt-5.2`, `gpt-5.1`, `gpt-5`, `gpt-5-mini`, `gpt-5-nano`
- `gpt-4.1`
- `gpt-oss-120b`, `gpt-oss-20b`

**Google Gemini:**
- `gemini-3-flash-preview`
- `gemini-3.1-flash-lite-preview` (US-only)
- `gemini-2.5-pro`, `gemini-2.5-flash`, `gemini-2.5-flash-lite`

**Alibaba Cloud Qwen:**
- `qwen3-next-80b-a3b`
- `qwen3-32B`

**Moonshot AI Kimi:**
- `kimi-k2.5`

### Feature matrix

| Feature | Supported on |
|---|---|
| Basic chat completions | All models |
| Streaming responses | All current OpenAI, plus the listed Claude / Gemini / Qwen / Kimi models |
| Tool / function calling | All current models; **real-time streaming + tool calling** specifically on `gpt-5-nano`, `kimi-k2.5`, `qwen3` |
| Structured JSON outputs | Claude 4.5+, OpenAI, Google |
| Prompt caching | Anthropic, OpenAI, Google |
| Multi-turn with context persistence | All models |
| Agentic workflows | All models |
| Automatic provider fallback | All models (configured at request time) |
| Built-in JSON repair | All models |

### Deprecated LLM Gateway models

| Model | Status | Notes |
|---|---|---|
| `gpt-4o-latest` | **Removed** — first LLM Gateway deprecation | Followed OpenAI's Feb 17, 2026 deprecation. Silently removed from the gateway. |
| Claude Haiku 3.5 | **Deprecated** | Removed from Anthropic's models page entirely. |
| `claude-haiku-4-5-20251001` (Claude 4.5 Haiku) | **Deprecated** | Do not use for new integrations. |

> **Migration:** Old `auto_chapters` and `summarization` features (deprecated on Universal-2, broken on Universal-3.5 Pro) should be replaced by LLM Gateway calls — typically Claude Sonnet 4.6 or GPT-5.1 are good defaults for chapter/summary generation from a transcript.

---

## Full Compatibility Matrix

### Pre-Recorded (Async)

| Feature | Universal-3.5 Pro | Universal-2 | SLAM-1 (deprecated) |
|---|---|---|---|
| Languages | 18 (Arabic, Danish, Dutch, EN, Finnish, French, German, Hebrew, Hindi, Italian, Japanese, Mandarin, Norwegian, Portuguese, Spanish, Swedish, Turkish, Vietnamese) | 99+ | EN (primary) |
| General Prompting | ✅ (1,500 words) | ❌ | ❌ |
| Keyterms Prompting | ✅ (1,000 terms) | ✅ (200 terms) | ❌ |
| Temperature control | ✅ | ❌ | ❌ |
| Medical Mode | ✅ | ✅ | ❌ |
| Word Boost | ❌ (silent fallback to U2) | ✅ | ✅ |
| Speaker Diarization (std) | ✅ | ✅ | ✅ |
| Speaker Diarization (exp) | ✅ | ✅ | — |
| Speaker Identification | ✅ | ✅ | ✅ |
| Entity Detection | ✅ (50+ types) | ✅ (50+ types) | ✅ |
| Sentiment Analysis | ✅ | ✅ | ✅ |
| Topic Detection (IAB) | ✅ | ✅ | ✅ |
| Key Phrases | ✅ (EN only) | ✅ (EN only) | ✅ |
| Content Moderation | ✅ | ✅ | ✅ |
| Translation | ✅ (100+ langs) | ✅ (100+ langs) | ✅ |
| Custom Formatting | ✅ | ✅ | ✅ |
| PII Redaction (text) | ✅ | ✅ (47+ langs) | ✅ |
| PII Redaction (audio) | ✅ | ✅ | ✅ |
| Profanity Filtering | ✅ | ✅ | ✅ |
| Custom Spelling | ✅ | ✅ | — |
| Code Switching | ✅ (18 langs) | ✅ (99 langs, 2 at a time, one must be EN) | — |
| Auto Language Detection | ✅ (130+) | ✅ (130+) | — |
| Multichannel | ✅ | ✅ | — |
| Audio Event Tagging | ✅ (100+ tags) | ❌ | ✅ (different format) |
| Disfluencies (`disfluencies` param) | ❌ (use prompting) | ✅ | ✅ |
| Auto Chapters | ❌ (500 error!) | ✅ (deprecated) | — |
| Summarization | ❌ (500 error!) | ✅ (deprecated) | — |

### Streaming

| Feature | U3.5 Pro Realtime | Universal-Streaming EN | Universal-Streaming ML |
|---|---|---|---|
| API value | `u3-rt-pro` (alias: `u3-pro`) | `universal-streaming-english` | `universal-streaming-multilingual` |
| Languages | Arabic, Danish, Dutch, EN, Finnish, French, German, Hebrew, Hindi, Italian, Japanese, Mandarin, Norwegian, Portuguese, Spanish, Swedish, Turkish, Vietnamese (18) | EN only | EN, ES, DE, FR, PT, IT (6) |
| Keyterms | ✅ Included (100 terms) | ✅ (add-on, 100 terms) | ✅ (100 terms) |
| General Prompting | ✅ Beta (add-on) | ❌ | ❌ |
| Medical Mode | ✅ | ✅ (EN only) | ✅ |
| Speaker Diarization (real-time) | ✅ (add-on; inline `speaker_labels: true`) | ✅ (add-on) | ✅ (add-on) |
| Auto Language Detection | ✅ (built-in) | ❌ | ✅ (built-in) |
| Audio Event Tagging | ✅ (100+ tags) | ❌ | ❌ |
| Formatted output | ✅ | ✅ | ✅ (default) |
| Mid-stream config updates | ✅ | ❌ | ❌ |

---

## Speaker Diarization — Speaker Limits

Available on all pre-recorded models and on U3.5 Pro Realtime (real-time, inline) plus the other streaming models. Speaker count limits vary by audio duration:

| Audio duration | Max speakers |
|---|---|
| 0–2 minutes | No limit |
| 2–10 minutes | 10 speakers |
| 10+ minutes | 30 speakers |

Override with `speaker_options` → `min_speakers_expected` and `max_speakers_expected`.

**Warning:** Setting `max_speakers_expected` too high reduces accuracy. US and EU regions only.

---

## Use-Case Recommendations

Derived from customer support conversations, AE/CSM Slack threads, and GTM guidance. This is what AssemblyAI customers actually use in production.

### Voice Agents / Real-Time AI Assistants

**Two paths:**

1. **Voice Agent API** (single WebSocket, end-to-end) — use when you want AssemblyAI to manage STT + LLM + TTS + turn detection + barge-in for you.
2. **`u3-rt-pro` + your own orchestrator** (LiveKit, Pipecat, Vapi, custom) — use when you need to swap LLMs/TTS independently or already have orchestration.

For raw STT in a voice agent stack, `u3-rt-pro` is the only correct choice — best entity accuracy, short-utterance handling, native code switching across the 6 supported languages (EN/ES/DE/FR/IT/PT), real-time diarization, and keyterms included.

- Customer examples: Decagon, Bland AI, Phonely, Vapi, Synthflow, Lorikeet, Sierra

**Common mistake:** Customers default to Universal-Streaming because it was the prior default. Results in losing competitive evals against Deepgram.

---

### Contact Centers / Call Centers

**Real-time (agent assist, live coaching):** Universal-3.5 Pro Realtime
**Post-call QA / conversation intelligence:** Universal-3.5 Pro (async)
**99-language or legacy requirements:** Universal-2 (async)

Many contact center customers use a **hybrid approach**: streaming for live agent assist + async for post-call analysis with deeper features (diarization, entity detection, sentiment, PII). Medical/clinical contact centers should layer in Medical Mode.

- Customer examples: Concentrix, CloudCall, Global Telesourcing, Calabrio, AmplifAI
- Note: It is not obvious on the website that AssemblyAI handles low-quality 8kHz telephony audio well. Proactively address in contact center evals.

---

### Meeting Transcription / Conversation Intelligence (CI)

**Recommended: Universal-3.5 Pro (async)**

Default for all post-call and meeting recording use cases. Highest accuracy + diarization + prompting for custom output formatting.

- Use `speaker_labels: true` for speaker attribution — not prompt instructions
- Do NOT use `auto_chapters` or `summarization` — use LLM Gateway for those (these features still work on Universal-2 but are deprecated)
- Customer examples: Granola, TLDV, Descript, Fireflies, Dovetail, Metaview, Glyphic, Fyxer, Bluedot, Atlassian, RecallAI

---

### Medical / Healthcare / Ambient Listening

**Recommended: Universal-3.5 Pro (async) + Medical Mode (`"domain": "medical-v1"`)**

Medical Mode reduces Missed Entity Rate on drug names, conditions, procedures, and clinical terminology by ~20%. Available on Universal-3.5 Pro and Universal-2 (optimized for U3 Pro).

- **Enable:** add `"domain": "medical-v1"` to your existing API config — no model switch required
- **Language support:** EN, ES, DE, FR
- **Streaming:** Medical Mode also available on U3.5 Pro Realtime, Universal-Streaming EN, Universal-Streaming ML
- Migrating from SLAM-1? U3 Pro replaces it with better accuracy + prompting support
- Customer examples: DaVita, Sully AI, Heidi Health, Magentus Healthcare, Deepscribe, Knowtex, Chapter (Medicare), NMDP

---

### Multilingual / Global Applications

**Async:**
- 18 supported languages (Arabic, Danish, Dutch, EN, Finnish, French, German, Hebrew, Hindi, Italian, Japanese, Mandarin, Norwegian, Portuguese, Spanish, Swedish, Turkish, Vietnamese): **Universal-3.5 Pro**
- All other languages: **Universal-2**
- Mixed: `speech_models: ["universal-3-pro", "universal-2"]` — routes to U3 Pro for its 18 languages, falls back to U2 for everything else

**Streaming:**
- 18 supported languages (Arabic, Danish, Dutch, EN, Finnish, French, German, Hebrew, Hindi, Italian, Japanese, Mandarin, Norwegian, Portuguese, Spanish, Swedish, Turkish, Vietnamese), voice agents: **Universal-3.5 Pro Realtime**
- 6-language set (EN/ES/DE/FR/IT/PT), broader compat: **Universal-Streaming Multilingual**

**Note:** Universal-3.5 Pro covers the same 18 languages for both async and real-time (streaming). Universal-Streaming Multilingual remains a 6-language model (EN/ES/DE/FR/IT/PT) for broader-compatibility use cases.

---

### Podcasts / Media / Broadcast

**Recommended: Universal-3.5 Pro (async)**

High-quality studio audio with proper nouns, brand names, technical vocabulary — the profile that U3 Pro's entity accuracy is optimized for. Use keyterms prompting for show-specific vocabulary.

For non-English archives or 99-language requirements: **Universal-2**.

---

### Compliance / Regulated Industries (PII, Content Moderation)

All pre-recorded models support PII redaction (audio and text), content moderation, and profanity filtering. Choose model based on accuracy and language needs first, then layer guardrails. The Voice Agent API is PCI-certified end-to-end.

- HIPAA BAA: available, sign in minutes without a sales call
- EU data residency: `api.eu.assemblyai.com` — data stays in EU for GDPR compliance
- PCI-DSS: Voice Agent API is PCI-certified

---

## Model Selection Decision Tree

```
What kind of audio?
│
├── Voice agent (end-to-end, single WebSocket)
│   └── Voice Agent API (built on u3-rt-pro + LLM Gateway + TTS)
│
├── Real-time STT (bring-your-own orchestrator)
│   └── Language in EN/ES/DE/FR/IT/PT?
│       └── YES → Universal-3.5 Pro Realtime (u3-rt-pro)   ← voice agents, agent assist
│
└── Pre-recorded (async)
    ├── Language in U3 Pro's 19?
    │   └── YES → Universal-3.5 Pro (default)
    │           ├── Need medical accuracy?          → + Medical Mode add-on
    │           ├── Need auto_chapters/summarization? → Use LLM Gateway
    │           └── Need filler word detection?     → Use prompting
    │
    └── Language outside those 19?
        └── YES → Universal-2
                └── Need auto_chapters or summarization? → Universal-2 (native, but deprecated; prefer LLM Gateway)
```

---

## Common Mistakes & How to Fix Them

| Mistake | What Happens | Fix |
|---|---|---|
| `speech_model` (singular) | Deprecated param — may cause errors or be silently ignored | Use `speech_models` (plural) |
| `auto_chapters` + Universal-3.5 Pro | Job hangs → silent 500 error | Use Universal-2 or LLM Gateway |
| `summarization` + Universal-3.5 Pro | Same as above | Use Universal-2 or LLM Gateway |
| `word_boost` + Universal-3.5 Pro | Silent fallback to Universal-2 | Use keyterms prompting instead |
| `disfluencies` param + Universal-3.5 Pro | Feature not supported, silently ignored | Use prompting; or switch to Universal-2 |
| `prompt` + `keyterms_prompt` together (async) | One silently ignored or error | Mutually exclusive on async — use one or the other. On streaming (`u3-rt-pro`), both are supported together. |
| "speaker" in a U3 Pro prompt | Inline speaker labels injected into transcript text | Use `speaker_labels: true` for diarization |
| `language_code` set on `u3-rt-pro` streaming | Silently ignored — language is auto-handled | Remove the parameter; rely on auto-detection |
| Voice agent on Universal-Streaming | Loses accuracy evals vs. Deepgram / competitors | Migrate to Universal-3.5 Pro Realtime |
| `max_speakers_expected` set too high (diarization) | Reduces speaker attribution accuracy | Set to actual expected max, or omit |
| Language detection on <15 seconds of audio | Detection may fail or return low confidence | Ensure 15+ seconds of spoken audio before relying on detection |
| Adopting `claude-haiku-4-5-20251001` in LLM Gateway | Deprecated | Use `claude-sonnet-4-6` or `claude-opus-4-7` for new integrations |
| Building your own STT+LLM+TTS pipeline when you don't need to swap parts | Engineering overhead, 5 APIs to debug | Use Voice Agent API for the simplest production path |

---

## Frequently Asked Questions

**Which model should I use for a voice agent?**
For raw STT: Universal-3.5 Pro Realtime (`u3-rt-pro`) — the only streaming model with the entity accuracy, short-utterance handling, and turn detection quality needed to win voice agent evals. For end-to-end: the Voice Agent API, which bundles `u3-rt-pro` + LLM Gateway + TTS in one WebSocket.

**What languages does U3 Pro support?**
**Async (`universal-3-pro`):** 18 — Arabic, Danish, Dutch, English, Finnish, French, German, Hebrew, Hindi, Italian, Japanese, Mandarin, Norwegian, Portuguese, Spanish, Swedish, Turkish, Vietnamese.
**Streaming (`u3-rt-pro`):** 18 — Arabic, Danish, Dutch, English, Finnish, French, German, Hebrew, Hindi, Italian, Japanese, Mandarin, Norwegian, Portuguese, Spanish, Swedish, Turkish, Vietnamese.

**What's the difference between `u3-pro` and `u3-rt-pro`?**
`u3-pro` was the original streaming name; `u3-rt-pro` is the canonical name at GA. `u3-pro` is now an alias and routes to `u3-rt-pro`. Use `u3-rt-pro` for consistency with current docs.

**Can I use `auto_chapters` with Universal-3.5 Pro?**
No. It silently causes stuck jobs and 500 errors. Use Universal-2 if you need it natively (though it's deprecated), or use LLM Gateway (e.g., Claude Sonnet 4.6 or GPT-5.1) to generate chapters from a U3 Pro transcript.

**Can I use `prompt` and `keyterms_prompt` together?**
**Pre-recorded (async):** No — they are mutually exclusive.
**Streaming (`u3-rt-pro`):** Yes — both can be used together. The keyterms are automatically appended to your custom prompt.

**Can I get real-time speaker labels?**
Yes. As of March 3, 2026, `u3-rt-pro` supports real-time speaker diarization. Set `speaker_labels: true`. Inline labels arrive with each turn — no post-processing required.

**Can I mix models with automatic language detection (ALD)?**
Yes. Use `speech_models: ["universal-3-pro", "universal-2"]` with ALD. The API routes to U3 Pro for its 18 languages and falls back to U2 for all others. The `speech_model_used` field in the response tells you which model was used.

**How do I enable Medical Mode?**
Add `"domain": "medical-v1"` to your existing API config. No model switch required. Supported on Universal-3.5 Pro (async + streaming), Universal-2 (async), Universal-Streaming EN, and Universal-Streaming ML. Supports EN, ES, DE, FR.

**I was on SLAM-1 for medical. What should I migrate to?**
Universal-3.5 Pro + Medical Mode. U3 Pro delivers better entity accuracy than SLAM-1 on medical terminology, with the added benefit of prompting support.

**What is the difference between Universal-Streaming and Universal-3.5 Pro Realtime?**
Universal-Streaming was the prior-generation model. Universal-3.5 Pro Realtime delivers significantly higher entity accuracy, better short utterance handling, native prompting (beta), keyterms included, real-time speaker diarization, and superior turn detection. Universal-3.5 Pro Realtime covers 18 languages; Universal-Streaming Multilingual covers 6 (EN/ES/DE/FR/IT/PT); Universal-Streaming English is EN-only. All new voice agent customers should use `u3-rt-pro`. Turn detection is also architecturally different (punctuation-based vs. confidence-based), so migration requires code changes.

**Does Universal-3.5 Pro Realtime support the same languages as async?**
Yes — both async and real-time Universal-3.5 Pro support the same 18 languages.

**Which model handles low-quality telephony / 8kHz audio?**
Universal-3.5 Pro (streaming or async) handles low-quality 8kHz telephony audio well, but this is not prominently documented. Mention proactively in contact center and phone agent evals. Note: as of early 2026, U3 Pro slightly underperforms Nova-3 on one CallHome telephony benchmark but outperforms on voice agent datasets.

**Does filler word / disfluency detection work on Universal-3.5 Pro?**
The `disfluencies` parameter is Universal-2 only. On Universal-3.5 Pro, use a prompt to handle filler words (e.g., "Remove filler words like um, uh, and ah" or "Tag all filler words in the transcript").

**What's the correct parameter name?**
`speech_models` (plural). Always set this explicitly — model defaults may differ between free-tier and paid accounts, and relying on defaults can cause surprises after upgrading.

**Is Voice Agent API a speech-to-speech model?**
No. It's a cascaded architecture (STT → LLM → TTS) wrapped in a single WebSocket. AssemblyAI built the cascade STT-first around `u3-rt-pro`, including an ensemble of STT streams with different prompts to extract richer information for turn-taking and accuracy.

**Should I use Voice Agent API or LiveKit + STT?**
Use the Voice Agent API when you want one connection end-to-end and don't need to swap LLM or TTS independently. Use `u3-rt-pro` + LiveKit (or Pipecat, Vapi, etc.) when you have your own orchestration or need fully custom routing.

**What new LLM Gateway features launched in April 2026?**
Automatic provider fallback (zero added latency), real-time streaming with tool calling on `gpt-5-nano`/`kimi-k2.5`/`qwen3`, structured JSON outputs on Claude 4.5+, prompt caching across Anthropic/OpenAI/Google, and new providers (Qwen 3, Kimi K2.5).

**What LLM Gateway models are deprecated?**
`gpt-4o-latest` (silent removal — first deprecation), Claude Haiku 3.5, and `claude-haiku-4-5-20251001` (Claude 4.5 Haiku). Do not use any of these for new integrations.