How to choose an LLM for production voice tasks

Don't pick one LLM for your company. Pick one per workload.

That sounds like extra work, but the numbers make the case. In the current Voice Standard Bench run, the top-scoring model costs 15 to 77 times more per hour of audio than a model within two quality points of it. On hour-long transcripts, that premium narrows to about 4×. So the right model for a 30-second voicemail and the right model for an hour-long meeting are often different models, and paying frontier prices for both is how LLM bills get out of hand.

Here's the framework: name the job, measure your transcript lengths, set a latency budget, price it per hour of audio, check reliability and API features, check where your data goes, then shortlist two or three models with Voice Standard Bench and prove them on your own transcripts. A benchmark tells you which models to try first. Your data tells you which one to ship.

Why general-purpose leaderboards don't answer the voice question

Most LLM benchmarks grade reasoning, coding, and knowledge over clean, typed text. Voice workloads don't look like that.

Speech-to-text output is messy. It has filler words, false starts, self-corrections, and the occasional misheard word. Inputs range from a few hundred tokens for a 30-second voicemail to well over 10,000 tokens for an hour-long meeting. And because the LLM step runs on every call or file, volume turns cost per hour of audio into a first-order concern instead of a rounding error.

General-purpose indexes don't measure any of that. A model near the bottom of one can be the best-value pick for pulling facts out of a short call, and a model that tops coding charts can still drop action items from a long meeting.

That's the gap Voice Standard Bench fills. It runs every model on LLM Gateway against real speech-to-text output at four transcript lengths, scores the answers, and reports quality, median response time, and cost per hour of audio side by side. If you want the details, the methodology post covers the rubrics, judges, and transcript set. The rest of this guide shows how to use those results to make a decision.

Step 1: Name the job

"Summarize this call" and "fix this dictation" stress a model in completely different ways. So start by writing down exactly what each LLM call in your pipeline does. Most voice workloads fall into five buckets:

• Text correction and dictation cleanup. Remove disfluencies and false starts, fix punctuation and casing, and follow spoken formatting commands, without changing what the speaker meant. Voice Standard Bench scores text correction on faithfulness and improvement. Dictation is scored mechanically against a known-correct output: exact match, word error rate, punctuation, capitalization, and formatting commands.
• Summarization. Call summaries, meeting notes, CRM updates. Scored on faithfulness (no hallucinated claims), coverage (did it keep the decisions and action items?), and conciseness.
• Information extraction. Names, dates, order numbers, action items, returned as structured JSON for downstream systems. Scored on groundedness and completeness.
• Classification. Call reason, sentiment, QA scoring. Usually short outputs, often high volume.
• Agent reasoning with tool calling. A live voice agent deciding what to say and which function to call next.

One honest caveat: Voice Standard Bench covers post-processing tasks: summarization, information extraction, text correction, and dictation. It doesn't test tool calling or live agent turns. If you're choosing a model for a voice agent, use the rest of this framework (latency, reliability, and feature support matter even more there), but don't read the bench scores as agent rankings.

Also, if one workload does several jobs, score it on the one that matters most. A model's overall score rewards versatility; its per-task score tells you about its specialty.

Step 2: Measure your transcript lengths

This is the axis general guides miss, and it's the main finding of Voice Standard Bench: transcript length changes which model you should use.

The bench groups transcripts into four lengths that match common production traffic: about 30 seconds (voicemails, dictated notes), 2 minutes (short calls), 15 minutes (support calls), and 60 minutes (meetings). Small, fast models score close to the top on short inputs and fall off as transcripts approach meeting length. Here's how the picks compare, using the overall quality score from the run published October 13, 2026:

Read it this way. Under about two minutes, a model that costs a fraction of a cent per hour of audio is within a few points of the leader. At 15 minutes the gap starts to open, and you have to decide whether the difference is worth paying for at your volume. At meeting length, larger models pull ahead, and an incomplete summary of an hour-long meeting is expensive to fix by hand.

So pull a histogram of your actual transcript durations before you pick anything. If 90% of your calls are under two minutes, you're shopping in a different aisle than a meeting-notes product.

Step 3: Set a latency budget

How long can the user wait? Answer that before you look at quality scores.

A live feature, like formatting dictation as someone speaks or enriching a streaming transcript, has a few hundred milliseconds to a second. A summary that lands in the CRM after the call ends can take several seconds and nobody notices. Those two budgets point to very different shortlists.

The bench reports median response time for every model at every length, and the spread is wide:

• Qwen3.5 4B Fast is the fastest model in the run at every length up to 15 minutes: 0.28 seconds median at 30 seconds, 0.60 at 2 minutes, and 1.49 at 15 minutes.
• gemma-4-31b answers a 30-second input in under a second, against GPT-6 Luna's 1.7 seconds. On a 2-minute call it's 1.5 seconds against 2.4.
• The same open-model families hosted elsewhere take 10 to 20 seconds to respond.

That last point matters more than it looks. Where a model is served can change response time as much as which model it is. When you compare latency, compare the deployment you'll actually call, and look at the slow tail (p95), not just the median. A model with a fast median can still stall often enough to hurt a live experience.

Step 4: Price per hour of audio, not per token

Token prices are hard to reason about for voice. Cost per hour of audio isn't.

The conversion is simple: tokens per hour of speech × price per token, for both input and output. A 30-second voicemail is a few hundred input tokens. An hour-long meeting is well over 10,000. Voice Standard Bench does this math for you and reports every model's cost per hour of audio at list price, for every task and length.

Two things jump out. First, the spread is enormous: that 15–77× premium for the top score, which narrows to about 4× at meeting length, when the larger models are actually earning their price. Second, low-cost proprietary models are competitive. GPT-6 Luna costs about half a cent per hour of audio in the current run and scores within a point of the best open model at every length.

A few pricing details to get right when you compare:

• LLM Gateway passes through provider rates with 0% markup.
• Listed prices are for global routing. In-region (US or EU) routing is 10% higher for models that carry a regional surcharge. For those models, add "model_region": "global" to a request to get the listed rate. Global routing is documented for Claude; it has no effect on models without a surcharge, such as Qwen or Gemma.
• Check current rates on the pricing page; models are added often.

For a full worked example (10,000 support calls a month, four models, per-call costs), see how to cut LLM costs on voice workloads. The short version: don't ask whether the cheaper model is cheaper. Ask whether it's good enough for this task at this length.

Step 5: Check reliability and API features, not just quality

A cheap model only saves money if it's reliable.

A model that gives a great answer 95% of the time and returns malformed JSON the other 5% will cost you more in retries, error handling, and broken downstream records than it saves on tokens. That's why Voice Standard Bench reports format failures, refusals, timeouts, and error rates for every model, task, and length. Models that fail its reliability check still appear in the charts, but they're never recommended.

Then check what the model can actually do through the API. Feature support varies more than people expect, and the model list in the docs spells it out per model:

• Qwen3.5 4B Fast supports max_tokens, temperature, and stream. No tool calling, no structured output. That makes it a strong fit for rewrite work like dictation cleanup and the wrong pick for JSON extraction or agents.
• GPT-6.1 Sol supports structured output (response_format) and reasoning effort, but doesn't list tools, so it can't drive an agent loop.
• GPT-4.1 supports tools but doesn't list response_format.

If your workload needs tool calling and structured outputs, filter on those parameters before you look at a single score. And whatever you pick, add automatic LLM fallbacks on a different provider, so one provider's bad day doesn't become your outage.

Step 6: Check where your data goes

The company that made a model and the provider that serves it aren't always the same. For data handling, the provider is what matters, because that's who processes your transcript.

On LLM Gateway, Qwen3.5 4B Fast is the one model AssemblyAI hosts on its own GPUs. Other models are served through AWS Bedrock, Google Vertex, OpenAI, Fireworks, or DigitalOcean, and the docs list the provider for each. A few other checks belong on your list:

• Retention. Zero data retention is available per request (with a header) or project-wide.
• Region. LLM Gateway has US and EU endpoints. Claude and most Gemini models are available in the EU; most OpenAI models are US-only. If you need EU residency, that alone can rule out part of your shortlist.
• Lifecycle. Models retire, and a retirement forces a migration whether you planned one or not. The docs list retirement dates; GPT-4.1, for example, shows January 14, 2027. That's worth knowing because GPT-4.1 is currently the lowest-latency pick within 8 points of the top on four of the five rows in the table below. Treat every retirement as a chance to re-run your comparison, not just to move to the direct successor.

Step 7: Shortlist with Voice Standard Bench, then test on your own transcripts

Now you have constraints: a task, a length, a latency budget, a cost ceiling, the features you need, and the regions you can use. Use them to filter the interactive results down to two or three candidates.

A few rules make the grid easier to read:

• Each cell names a recommended (highest quality), fastest (lowest median response), and budget (lowest cost per hour) pick for that task and length.
• In the take-home table, the price and latency picks must score within 8 points of the top model, so a cheap or fast model that fails the task can't win the column.
• Scores within 2–3 points of each other are effectively tied. When two models are that close, let price and latency break the tie.
• If your audio falls between two lengths, use the longer one.

The same scores are available programmatically. The LLM Gateway models endpoint returns quality, per-task quality, cost per hour of audio, and latency for each model, so you can choose a model with a script instead of by eye (the docs' "Choose a model" guide shows how).

Then test. Our transcripts aren't yours. Your callers have different accents, your domain has its own vocabulary, your prompts ask for different output, and your cost limit is your own. Any of those can change which model wins. There are three ways to test, from quickest to most thorough:

1. Playground compare mode. Upload a file, run one prompt across several models, and compare the answers side by side. No code.
2. "Try in Gateway" links. Every cell on the results page opens the playground with that model and task prompt loaded. Swap in your transcript and run it.
3. The open-source benchmark. Voice Standard Bench is on GitHub. Point it at your own transcripts to get the same quality, speed, and cost results, scored on your data.

Or do it in a few lines of code. LLM Gateway is OpenAI-compatible, so comparing three models means changing one string three times. This script runs the same cleanup prompt on the same transcript across a shortlist and prints latency and output for each:

import time
from openai import OpenAI

client = OpenAI(
    api_key="<YOUR_API_KEY>",
    base_url="https://llm-gateway.assemblyai.com/v1",
)

# Your shortlist from Voice Standard Bench
models = ["qwen3.5-4b-32k-fast", "gemma-4-31b", "gpt-6-luna"]

with open("transcript.txt") as f:
    transcript = f.read()

prompt = (
    "Clean up this transcript. Remove filler words and false starts, "
    "fix punctuation and casing, and keep the speaker's meaning exactly.\n\n"
    + transcript
)

for model in models:
    start = time.perf_counter()
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=1000,
    )
    elapsed = time.perf_counter() - start
    print(f"--- {model}: {elapsed:.2f}s ---")
    if response.usage:
        print(f"tokens in/out: {response.usage.prompt_tokens}/{response.usage.completion_tokens}")
    print(response.choices[0].message.content)

Run it on 20 to 30 real transcripts from your highest-volume workload, not one hand-picked example. Once you've chosen, keep a backup on a different provider. The fallback goes in the same request:

response = client.chat.completions.create(
    model="gemma-4-31b",
    messages=[{"role": "user", "content": prompt}],
    max_tokens=1000,
    extra_body={
        "fallbacks": [{"model": "gpt-5.6-luna"}],
        "fallback_config": {"depth": 1},
    },
)
print(response.model)  # the model that actually answered

A starting-point table

If you want somewhere to begin, here's the take-home table from the Voice Standard Bench run published October 13, 2026. Quality is 0 to 100, averaged across transcript lengths. Picks move with each run, so check the live results before you commit.

No model takes more than one column in any row. That's the whole argument for choosing per workload.

One more option for dictation: if you'd rather not run a cleanup model yourself, the Dictation API bundles transcription and cleanup in one call.

Production checklist

Once a model is in production, the decision isn't finished. Keep these habits:

• One model per workload. Dictation cleanup and meeting summaries share an API key, not a model.
• Fallbacks on a different provider. A fallback on the same upstream doesn't help during that provider's outage.
• Log model, latency, and cost on every call. The response tells you which model actually answered, which matters once fallbacks fire.
• Re-run your comparison when models change. New models land on LLM Gateway often, and the cheapest good-enough model shifts every few months. A retirement date is a scheduled reason to look.
• Re-check reliability after any prompt change. A prompt that works on one model can break another model's output format.

The benchmark is the start, not the answer

Choosing an LLM for voice isn't one decision. It's a small decision per workload, made against constraints most general guides ignore: how long your transcripts run, how long your users wait, and what an hour of audio costs at your volume.

The useful shift is to treat model choice as something you revisit, not something you settle. The task, length, and latency axes don't change much. The models do, every few weeks. Teams that keep a small test set of their own transcripts and a one-command way to score it will keep finding cheaper, faster picks. Teams that chose once will keep paying for the model they chose.

Frequently asked questions

How do I choose the right LLM for my use case?

For voice workloads, choose on six things: the task, your typical transcript length, your latency budget, cost per hour of audio, reliability and API feature support, and where your data is processed. Use a voice-specific benchmark such as Voice Standard Bench to shortlist two or three models that fit those constraints. Then test the shortlist on 20 to 30 of your own transcripts before you ship.

What is the best LLM for voice agents?

It depends on your latency budget and whether the model supports tool calling, so there isn't one answer. Voice Standard Bench measures post-processing tasks like summarization and extraction, not live agent turns, so don't read its scores as agent rankings. For agents, filter for models that support tools, compare median and p95 latency, and test on real conversations from your product.

Does transcript length change which LLM I should use?

Yes. In Voice Standard Bench, small, fast models score close to the top on transcripts under about two minutes and fall behind at meeting length, where larger models pull ahead.

How do I evaluate an LLM on my own call transcripts?

Send the same prompt and the same set of real transcripts to each candidate model and compare quality, latency, and cost. On LLM Gateway that means changing one model string per request, or using playground compare mode with no code. For a full evaluation with the same tasks, rubrics, and judges as the public benchmark, run the open-source Voice Standard Bench code on your own transcripts.

Why don't general LLM leaderboards work for voice tasks?

General-purpose indexes mostly test reasoning and coding on clean, typed text. Voice workloads run on speech-to-text output full of fillers and false starts, at lengths from 30 seconds to an hour, where cost per hour of audio matters. A model's rank on a general index often doesn't predict how well it summarizes a support call or cleans up a dictated note.

How much does it cost to run an LLM on an hour of audio?

It ranges from a fraction of a cent to a few dollars per hour of audio, depending on the model, the task, and the transcript length. In the current Voice Standard Bench run, GPT-6 Luna costs about half a cent per hour of audio, while the top-scoring model costs 15 to 77 times more than a model within two quality points of it. LLM Gateway charges provider rates with 0% markup at global routing; in-region routing is 10% higher for models with a regional surcharge.