Introducing Voice Standard Bench: 50+ models, ranked by performance, speed and cost on real audio

How we test 50+ models

To find out where a cheaper LLM is good enough, we test models on the inputs voice products actually send: real Speech-to-Text output, with filler words, false starts, and the occasional misheard word, at the lengths those products see in production.

Each model runs tasks like:

• Information extraction: pull specific facts, such as names, dates, and order numbers, from the transcript into a structured answer.
• Summarization: write a summary that covers the conversation accurately and concisely.
• Text correction: fix the wording of a transcript without changing what was said.

Each task runs files at about 30 seconds, 2 minutes, 15 minutes, and 60 minutes. Every model gets the same transcripts and the same prompt.

Two judge models, Claude Opus 5.5 and GPT-5.6 Sol, score each answer against a fixed rubric for that task, with every metric normalized to a 0–100 scale, and we return model response time and cost alongside quality to help build a robust picture of each model’s strengths and weaknesses. Researchers on our team reviewed hundreds of sample scores to identify issues and iterate on the model grader prompts.

If you’re interested in the full methodology, or want to benchmark Voice Tasks with your own audio files using our Voice Standard Bench framework, read the full methodology post here.

Where open models win, and where they don't

If you only read one thing: start with the model in the column that matches what you are optimizing for, then test it on your own transcripts.

Take-home picksTop quality, lowest price and lowest latency by task

Task	Top qualityHighest mean score	Lowest priceWithin 8 points of the top	Lowest latencyWithin 8 points of the top
Summarization	Opus 5.598 quality	GPT-6 Luna94 quality	GLM-5.3 Flash91 quality
Information extraction	Opus 587 quality	GPT-6 Luna86 quality	GPT-4.179 quality
Text correction	GPT-5.6 SolGPT-6 AstraGPT-6.1 Sol91 quality	gemma-4-31b84 quality	GPT-4.184 quality
Dictation	Opus 4.6Gemini 3.5 Flash99 quality	gemma-4-31b97 quality	GPT-4.193 quality
OverallEvery task, weighted equally	GPT-6 Astra92 quality	GPT-6 Luna90 quality	GPT-4.184 quality

Quality is 0 to 100, the model's mean score across transcript lengths. The price and latency picks are limited to models within 8 points of the top score, so a cheap or fast model that fails the task cannot win the column. Picks are from the current run and move when it does. Open the full results to filter every chart by task and length.

In the current run, the top scorer costs 55 to 65 times more per hour of audio than a model within two quality points of it. Only on hour-long transcripts does that premium narrow, to about 5×. Open models, though, surprisingly rarely win on cost. GPT-6 Luna, at its default reasoning setting, scores within a point of the best open model at every transcript length and costs between 0.4 and 1.6 cents per hour of audio, so the open models that undercut it save fractions of a cent, and past two minutes they give up at least 10 quality points to do it. The true cost of GPT-6 Luna is latency.

Open models win on speed for short transcripts. On a 30-second input, such as a voicemail or a dictated note, gemma-4-31b answers in under a second at 93 quality against GPT-6 Luna's 1.7 seconds, and on a 2-minute call it answers in 1.5 seconds at 89 against 2.4. Qwen3.5 4B Fast is the fastest model in the run at every length up to 15 minutes. The same families hosted elsewhere take 10 to 20 seconds to respond. That said, as transcripts approach meeting length, larger models pull ahead on quality and the speed gap closes.

Picks by task and lengthRecommended, fastest and budget

Information extractionSummarizationText correctionDictation cleanup

Transcript length	30 secsVoicemail	2 minsShort call	15 minsSupport call	60 minsMeeting
RecommendedHighest quality	GPT-5.6 Sol94 quality	Nemotron Lightning 3.5 30B A3B100 quality	GPT-6 Luna85 quality	GPT-6.1 Sol85 quality
FastestLowest median response	Qwen3.5 4B Fast0.28 s median · 80 quality	Qwen3.5 4B Fast0.60 s median · 51 quality	Qwen3.5 4B Fast1.49 s median · 33 quality	Gemini 3.5 Flash Lite2.86 s median · 59 quality
BudgetLowest cost per hour of audio	Nemotron 3 Nano 30B A3B$0.005 per hour · 73 quality	Nemotron 3 Nano 30B A3B$0.003 per hour · 58 quality	Nemotron 3 Nano 30B A3B$0.001 per hour · 49 quality	Nemotron 3 Nano 30B A3B$0.001 per hour · 52 quality

Quality is 0 to 100. Response time is the median across every call at that length, and cost is per hour of audio at list price. Open the full results to filter every chart by task and length.

Here's where to start for three common workloads:

• Extracting details from short calls. For names, dates, or order numbers from calls of a minute or two, start with the fastest and budget picks. The quality gap to the largest models is small at this length, and the savings add up quickly at high call volume. See short-call extraction results.
• Call-center summaries. Typical support calls run 2 to 15 minutes. Compare the recommended and budget picks in the 15-minute column, and decide whether the quality difference is worth the cost difference at your volume. See call summary results.
• Meeting notes. For hour-long recordings, start with the recommended pick. Larger models are clearly ahead at this length, and an incomplete summary of a long meeting is expensive to fix by hand. See meeting summary results.

A cheap model only saves money if it's reliable

Quality scores aren't the whole picture. Voice Standard Bench also reports format failures (the model was asked for a structured answer and returned something unusable), refusals, timeouts, and error rates for every combination.

Models that fail the reliability check still appear in the charts with their scores, but they aren't recommended in the grid. Some models give strong answers most of the time but return an unusable format often enough that they need more work on prompts or output handling before they reach users.

Start with our picks, then test on your transcripts

Our transcripts aren't yours. Your callers have different accents, your domain has its own vocabulary, your prompts ask for different output, and your cost limit is your own. Any of these dimensions can change which model works best, so leverage Voice Standard Bench as a starting point for which two or three models to try first.

Which one you ship should come from your own data, so take these recommendations and test them on your own voice data with LLM Gateway.

How to run Voice Standard Bench on your own transcripts

Voice Standard Bench is open source. Download the code from GitHub, point it at your own transcripts, and run a private version with the same tasks, rubric, and graders, scored on your data. You get the same quality, speed, and cost results as the public benchmark, but for your callers, your vocabulary, and your prompts.

Every model in the benchmark is available through LLM Gateway with your existing API key, so the full run needs no new accounts. If you'd rather start with a quick check on a few candidates, switching models only means changing the model parameter. Comparing three picks takes three requests:

import requests

API_KEY = "YOUR_ASSEMBLYAI_API_KEY"
transcript = open("call.txt").read()  # text from your Speech-to-Text request

# Replace with the picks from the grid cell for your task and length
models = ["MODEL_ID_1", "MODEL_ID_2", "MODEL_ID_3"]

for model in models:
    response = requests.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        headers={"authorization": API_KEY},
        json={
            "model": model,
            "messages": [
                {"role": "user", "content": f"Summarize this call in five bullet points:\n\n{transcript}"}
            ],
        },
    )
    print(model, response.json()["choices"][0]["message"]["content"], sep="\n")

New to AssemblyAI? Create a free account to get an API key. It works for Speech-to-Text and every model on LLM Gateway.

You can also compare models without writing code:

• Playground compare mode runs one prompt against several models and shows the answers side by side.
• The Speech-to-Text playground sends a transcript straight to an LLM Gateway model. Upload your own audio, get the transcript, and run it through your shortlist.
• "Try in Gateway" links on the results page open the playground with that model and a prompt for that task already loaded. Replace our transcript with one of yours and run it.

Frequently asked questions