LLM Gateway

Run open and frontier LLMs on your voice data

From raw utterance to structured answer—fast open models for the cleanup pass, frontier models for the reasoning pass, one API key for both.

Speech-to-Text

Hi, Marissa. Um, quick recap on where we landed, uh, with the Q3 vendor review. So, uh, we got through 6 of the 8— oh no, 5 of the 8 assessments. Um, and the last 3, they’re waiting on security sign-off. Um, the big thing is that pricing came back higher than, than we modeled, something like, uh, between 12— no, 15% over. So I think we need to revisit The budget line before we commit to anything. Can you pull the original forecast for me? And I’ll put 30 minutes on the calendar.

LLM Gateway

Clean up dictated speech. Remove fillers (um, uh, like, you know), stutters, repetitions, false starts; on self-corrections keep only the final version. Fix punctuation, capitalization, sentence and paragraph breaks. Render spoken commands ("period", "new paragraph") as punctuation.

Final output

Hi Marissa, quick recap on where we landed with the Q3 vendor review. We got through five of the eight assessments. The last three are waiting on security sign-off. The big thing is that pricing came back higher than we modeled, something like 15% over. I think we need to revisit the budget line before we commit to anything. Can you pull the original forecast for me? I'll put 30 minutes on the calendar.

539 ms · 90 tokens · <$0.0001

Models

Open and frontier models, one endpoint

53 models from 6 providers, all OpenAI-compatible—switch by changing one string.

LLM Gateway models with context window, input, output and cached token rates per 1 million tokens, capabilities, recommended tasks, serving providers and regions.
					Capabilities	Best for	Providers	Regions
`anthropic/claude-opus-5` New	200K	$5.00/M	$25.00/M	Read:$0.50/M Write:$6.25/M	Tools JSON Streaming	—	AWS Bedrock	USGlobal
`google/gemini-3.8-flash` New	1M	$0.75/M	$3.75/M	Read:$0.07/M Write:—	Tools JSON Streaming	Correction	Vertex	USEUGlobal
`openai/gpt-5.6-sol` New	270K	$4.00/M	$20.00/M	Read:$0.40/M Write:$5.00/M	Tools JSON Streaming	Extraction	Open AI	USGlobal
`google/gemma-4-31b` New	256K	$0.14/M	$0.40/M	Read:— Write:—	Tools JSON Streaming	DictationSummarization	AWS Bedrock	US
`google/gemini-3.7-flash` New	1M	$0.75/M	$3.75/M	Read:$0.07/M Write:—	Tools JSON Streaming	Correction	Vertex	USEUGlobal
`nvidia/nemotron-lightning-3p5-30b-a3b`	262K	$0.05/M	$0.20/M	Read:$0.01/M Write:—	Tools JSON Streaming	—	Fireworks	US
`openai/gpt-5-nano`	400K	$0.05/M	$0.40/M	Read:$0.005/M Write:—	Tools JSON Streaming	—	Open AI	US
`nvidia/nemotron-3-nano-30b-a3b`	256K	$0.06/M	$0.24/M	Read:— Write:—	Tools JSON Streaming	—	AWS Bedrock	USEU
`nvidia/nemotron-nano-9b-v2`	128K	$0.06/M	$0.23/M	Read:— Write:—	Tools JSON Streaming	—	AWS Bedrock	USEU
`openai/gpt-oss-20b`	131K	$0.07/M	$0.30/M	Read:— Write:—	Tools	—	AWS Bedrock	US
`anthropic/claude-haiku-5-5`	1M	$0.10/M	$0.50/M	Read:$0.01/M Write:$0.13/M	Tools JSON Streaming	DictationSummarization	AWS Bedrock	USEUGlobal
`assemblyai/qwen3.5-4b-32k-fast`	33K	$0.10/M	$0.50/M	Read:— Write:—	Streaming	—	AssemblyAI	USEU
`google/gemini-2.5-flash-lite`	1M	$0.10/M	$0.40/M	Read:$0.01/M Write:—	Tools JSON Streaming	DictationSummarization	Vertex	USEUGlobal
`openai/gpt-6-luna`	922K	$0.10/M	$0.50/M	Read:$0.01/M Write:$0.13/M	Tools JSON Streaming	DictationExtractionSummarizationCorrection	AWS BedrockOpen AI	USGlobal
`alibaba/qwen3-32B`	200K	$0.15/M	$0.60/M	Read:— Write:—	Tools JSON Streaming	—	AWS Bedrock	USEU
`alibaba/qwen3-next-80b-a3b`	200K	$0.15/M	$1.20/M	Read:— Write:—	Tools JSON Streaming	Dictation	AWS Bedrock	USEU
`nvidia/nemotron-3-super-120b-a12b`	256K	$0.15/M	$0.65/M	Read:— Write:—	Tools JSON Streaming	DictationSummarization	AWS Bedrock	USEU
`openai/gpt-oss-120b`	131K	$0.15/M	$0.60/M	Read:— Write:—	Tools JSON	—	AWS Bedrock	US
`zai/glm-5.3-flash`	1M	$0.15/M	$0.50/M	Read:$0.03/M Write:—	Tools JSON Streaming	DictationSummarizationCorrection	Digital OceanFireworks	US
`openai/gpt-5.6-luna`	270K	$0.20/M	$1.20/M	Read:$0.02/M Write:$0.25/M	Tools JSON Streaming	DictationExtractionSummarizationCorrection	Open AI	USEUGlobal
`google/gemini-3.1-flash-lite`	1M	$0.25/M	$1.50/M	Read:$0.03/M Write:—	Tools JSON Streaming	—	Vertex	USEUGlobal
`openai/gpt-5-mini`	400K	$0.25/M	$2.00/M	Read:$0.03/M Write:—	Tools JSON Streaming	—	Open AI	US
`deepseek/deepseek-v4.1-flash`	1M	$0.30/M	$1.20/M	Read:$0.006/M Write:—	Tools JSON Streaming	—	Digital OceanFireworks	US
`google/gemini-2.5-flash`	1M	$0.30/M	$2.50/M	Read:$0.03/M Write:—	Tools JSON Streaming	—	Vertex	USEUGlobal
`google/gemini-3.5-flash-lite`	1M	$0.30/M	$2.50/M	Read:$0.03/M Write:—	Tools JSON Streaming	Summarization	Vertex	USEUGlobal
`minimax/minimax-m3`	512K	$0.30/M	$1.20/M	Read:$0.06/M Write:—	Tools JSON Streaming	—	Fireworks	US
`zai/glm-5.3`	1M	$0.95/M	$3.40/M	Read:$0.20/M Write:—	Tools JSON Streaming	—	Digital OceanFireworks	US
`anthropic/claude-haiku-4-5-20251001`	200K	$1.00/M	$5.00/M	Read:$0.10/M Write:$1.25/M	Tools JSON Streaming	—	AWS Bedrock	USEUGlobal
`google/gemini-2.5-pro`	200K	$1.25/M	$10.00/M	Read:$0.13/M Write:—	Tools JSON Streaming	—	Vertex	USEUGlobal
`google/gemini-3.5-flash`	1M	$1.25/M	$9.00/M	Read:$0.13/M Write:—	Tools JSON Streaming	—	Vertex	USEUGlobal
`meta/muse-spark-open`	1M	$1.25/M	$4.25/M	Read:$0.15/M Write:—	Tools JSON Streaming	—	Fireworks	US
`openai/gpt-5`	400K	$1.25/M	$10.00/M	Read:$0.13/M Write:—	Tools JSON Streaming	Extraction	Open AI	US
`openai/gpt-5.1`	400K	$1.25/M	$10.00/M	Read:$0.13/M Write:—	Tools JSON Streaming	—	Open AI	US
`google/gemini-3.6-flash`	1M	$1.50/M	$7.50/M	Read:$0.15/M Write:—	Tools JSON Streaming	Correction	Vertex	USEUGlobal
`openai/gpt-5.2`	400K	$1.75/M	$14.00/M	Read:$0.17/M Write:—	Tools JSON Streaming	—	Open AI	US
`anthropic/claude-sonnet-5-5`	1M	$2.00/M	$10.00/M	Read:$0.10/M Write:$2.50/M	Tools JSON Streaming	Extraction	AWS Bedrock	USGlobal
`google/gemini-4-argon`	1M	$2.00/M	$10.00/M	Read:$0.10/M Write:—	Tools JSON Streaming	—	Vertex	USEUGlobal
`openai/gpt-4.1`	1M	$2.00/M	$8.00/M	Read:$0.50/M Write:—	Tools Streaming	—	Open AI	US
`openai/gpt-5.6-terra`	270K	$2.00/M	$12.00/M	Read:$0.20/M Write:$2.50/M	Tools JSON Streaming	—	Open AI	USGlobal
`openai/gpt-6-sol`	922K	$2.00/M	$10.00/M	Read:$0.20/M Write:$2.50/M	Tools JSON Streaming	ExtractionCorrection	AWS BedrockOpen AI	USGlobal
`openai/gpt-6.1-sol`	922K	$2.00/M	$10.00/M	Read:$0.10/M Write:$2.50/M	JSON Streaming	ExtractionCorrection	AWS BedrockOpen AI	US
`xai/grok-4.7`	500K	$2.00/M	$6.00/M	Read:$0.50/M Write:—	Tools JSON Streaming	—	AWS Bedrock	USGlobal
`anthropic/claude-sonnet-4-5-20250929`	200K	$3.00/M	$15.00/M	Read:$0.30/M Write:$3.75/M	Tools JSON Streaming	—	AWS Bedrock	USEUGlobal
`anthropic/claude-sonnet-4-6`	200K	$3.00/M	$15.00/M	Read:$0.30/M Write:$3.75/M	Tools JSON Streaming	—	AWS Bedrock	USEUGlobal
`anthropic/claude-sonnet-5`	200K	$3.00/M	$15.00/M	Read:$0.30/M Write:$3.75/M	Tools JSON Streaming	—	AWS Bedrock	USGlobal
`moonshot-ai/kimi-k3`	1M	$3.00/M	$15.00/M	Read:$0.30/M Write:$3.75/M	Tools JSON Streaming	—	AWS BedrockFireworks	USGlobal
`anthropic/claude-opus-5-5`	1M	$4.00/M	$20.00/M	Read:$0.20/M Write:$5.00/M	Tools JSON Streaming	Extraction	AWS Bedrock	USEUGlobal
`anthropic/claude-opus-4-5-20251101`	200K	$5.00/M	$25.00/M	Read:$0.50/M Write:$6.25/M	Tools JSON Streaming	—	AWS Bedrock	USGlobal
`anthropic/claude-opus-4-6`	200K	$5.00/M	$25.00/M	Read:$0.50/M Write:$6.25/M	Tools JSON Streaming	—	AWS Bedrock	USGlobal
`anthropic/claude-opus-4-7`	1M	$5.00/M	$25.00/M	Read:$0.50/M Write:$6.25/M	Tools JSON Streaming	—	AWS Bedrock	USGlobal
`anthropic/claude-opus-4-8`	1M	$5.00/M	$25.00/M	Read:$0.50/M Write:$6.25/M	Tools JSON Streaming	—	AWS Bedrock	US
`openai/gpt-5.5`	272K	$5.00/M	$30.00/M	Read:$0.50/M Write:—	Tools JSON Streaming	—	Open AI	USGlobal
`openai/gpt-6-astra`	270K	$10.00/M	$50.00/M	Read:$1.00/M Write:$12.50/M	Tools JSON Streaming	—	Open AIAWS Bedrock	US

Production grade

Easiest, most reliable way to call multiple LLMs

Ship faster, spend less on tokens, and stop losing users to provider outages.

0% markup

Pay provider rates, not gateway rates. Competing gateways add 5% or more to every call.

Automatic fallbacks

Configure backup models per request. When a provider errors or stalls, your call still goes through.

Security by simplicity

Pay the exact same price as calling the model provider directly. No markup, no hidden fees, no minimum commitment. We make it simple.

OpenAI-compatible

Drop into any OpenAI SDK. Change a base URL and a model string, and everything keeps working.

Voice-native

Your LLM calls run where your transcription does. One less network hop on every turn.

Models worth using

Frontier models from OpenAI, Anthropic, and Google, plus open models we host ourselves. New ones added the day they launch.

Voice Standard Bench

Choose the right model for every voice task

See the full results

Picking from 50+ models is hard, so we scored each on real transcripts for performance, cost, and speed.

Picks by task and lengthRecommended, fastest and budget

Transcript length	30 secsVoicemail	2 minsShort call	15 minsSupport call	60 minsMeeting
RecommendedHighest quality	GPT-5.6 Sol94 quality	Nemotron Lightning 3.5 30B A3B100 quality	GPT-6 Luna85 quality	GPT-6.1 Sol85 quality
FastestLowest median response	Qwen3.5 4B Fast0.28 s median · 80 quality	Qwen3.5 4B Fast0.60 s median · 51 quality	Qwen3.5 4B Fast1.49 s median · 33 quality	Gemini 3.5 Flash Lite2.86 s median · 59 quality
BudgetLowest cost per hour of audio	Nemotron 3 Nano 30B A3B$0.002 per hour · 73 quality	Nemotron 3 Nano 30B A3B$0.002 per hour · 58 quality	Nemotron 3 Nano 30B A3B$0.001 per hour · 49 quality	Nemotron 3 Nano 30B A3B$0.001 per hour · 52 quality

Quality is 0 to 100. Response time is the median across every call at that length, and cost is per hour of audio at list price. Open the full results to filter every chart by task and length.

Playground

Compare models before you write any code

Use our playground to upload a file, run a prompt over the transcript, and switch models to see how the answers differ.

Try it out

Common questions

: AssemblyAI's LLM Gateway is a single, OpenAI-compatible API that routes requests to GPT, Claude, Gemini, Qwen, and other frontier models—with automatic fallbacks, no markup, and zero data retention available on every call.
: You pay the same price as the underlying provider—zero markup. No additional fees from AssemblyAI to use the Gateway.
: GPT (OpenAI), Claude (Anthropic), Gemini (Google), Qwen (Alibaba), and other frontier and open-weights models, including ones we host ourselves. New models are added the day they launch.
: Yes. Any OpenAI SDK or HTTP client works—change the base URL and API key, and the rest of your code is unchanged.
: Yes. EU customers can route requests through EU-hosted infrastructure for GDPR compliance. EU residency is on by default for traffic originating in the EU.
: Configure a primary model and any number of fallbacks. If the primary fails or rate-limits, the Gateway transparently retries on the next model in your list—no code changes required.
: Yes. Pass a header on each request, or configure it project-wide, to opt into zero data retention.