How to cut LLM costs on voice workloads without losing quality

If you run a voice product at scale, your LLM bill is probably top of mind. In production workflows, LLM volume often grows from the tasks that happen after the transcript lands: cleaning up dictation, summarizing calls, pulling out names, dates, and order numbers. That work runs on every file, and teams often send all of it to a frontier model by default.

Most of it doesn't need one. Yesterday we released Voice Standard Bench, which scores every model on LLM Gateway on the tasks voice products run most, at four transcript lengths, with quality, speed, and cost side by side. This post turns those results into a playbook: how to find the expensive workloads in your stack, pick cheaper candidates, and prove quality holds on your own transcripts before you switch.

Where LLM spend goes in a voice pipeline

A typical voice pipeline makes one or more LLM calls per file or per call:

• Text correction and dictation cleanup: remove filler words and false starts, fix casing and punctuation.
• Summarization: call summaries, meeting notes, CRM updates.
• Information extraction: names, dates, amounts, action items, structured JSON for downstream systems.
• Classification: call reason, sentiment, next step, QA scoring.

Cost scales with two things: volume and transcript length. A 30-second voicemail is a few hundred tokens. An hour-long meeting is well over 10,000 tokens of input. Multiply either by your monthly call count and the model's price per token, and the choice of model quickly becomes the biggest lever on the bill.

And here's the thing: the work itself is relatively narrow. Rewriting a transcript or pulling five fields out of a support call doesn't take frontier-level reasoning. It takes a model that reads messy speech-to-text output reliably and follows a format.

The low-cost models on LLM Gateway today

LLM Gateway gives you 50+ models from OpenAI, Anthropic, Google, and other providers through one OpenAI-compatible API, using the same AssemblyAI API key you use for speech-to-text. Alongside the frontier models, it includes a set of open-weight and low-cost models. List prices per 1 million tokens as of October 8, 2026 (available models):

For comparison, Claude Sonnet 5 lists at $3 / $15, Claude Opus 5.5 at $4 / $20, and GPT-5.6 Sol at $4 / $20 per 1M tokens. Many proprietary models also carry a +10% regional surcharge. Several of the models above are 20–60x cheaper per token than those.

Two of these deserve a note. Qwen3.5 4B Fast is the one model AssemblyAI hosts itself, set up for fast rewrite tasks. In our launch testing it averaged about 612 ms on voice rewrite tasks (details). Gemma 4 31B is a larger open-weight model from Google that handles a wider range of tasks.

Match the model to the task and the transcript length

Price per token only matters if the cheaper model does the job. That's what Voice Standard Bench measures, and the main finding is simple: transcript length changes which model you should use.

Here's how the picks compare at each length (Voice Standard Bench quality score, with cost per hour of audio in parentheses):

The practical takeaway: don't pick one model for your whole pipeline. Pick one per workload. A dictation cleanup step and a meeting summary step have very different needs, even if they share an API key.

Worked example: summarizing 10,000 support calls a month

Here's what the price difference looks like at list price. Assumptions: 10,000 support calls a month, about 15 minutes each, so roughly 3,000 tokens of transcript plus a 200-token prompt (3,200 input tokens). Each summary is about 300 output tokens.

That is a 25x to 30x difference between Sonnet 5 and the two cheapest rows on the same workload, before any regional surcharge. At 10,000 calls it's a rounding error. At 1 million calls a month it's $14,100 versus $470 to $568.

The question isn't whether the cheaper model is cheaper. It's whether it's good enough for this task at this length. That's the column you fill in with benchmark results and then confirm on your own data.

A cheap model only saves money if it's reliable

Quality scores don't tell the whole story. A model that gives a great answer 95% of the time but returns malformed JSON the other 5% will cost you more in retries, error handling, and broken downstream records than it saves.

The quality score counts every failure: a structured answer that came back in the wrong format, a refusal, a timeout, an error. A model that gives a great answer four times in five lands mid-table, not at the top. Check the quality score for your task before you move a structured-extraction workload, and check where the model is served from: the same open model answers in under a second from AWS Bedrock or AssemblyAI and in 10 to 20 seconds from smaller hosts.

Two practical safeguards:

• Set up automatic LLM fallbacks so a failed or slow request retries on a second model.
• Ramp traffic gradually. Some third-party-served models can hit provider capacity limits under sudden load, even with higher rate limits on your account.

The migration playbook: four steps

1. Audit where your tokens go

List every LLM call in your pipeline. For each one, note the task (correction, summary, extraction, classification), the typical transcript length, monthly volume, and current model. Sort by monthly cost. The top two or three workloads are usually most of the bill.

2. Shortlist two or three candidates per workload

Open the Voice Standard Bench grid, find the cell for your task and length, and note the recommended, fastest, and budget picks. If your workload needs JSON, check the candidates' scores on information extraction, where a malformed answer counts as a failure.

3. Test on your own transcripts

Our transcripts aren't yours. Your callers have different accents, your domain has its own vocabulary, and your prompts ask for different output. Run your real prompts on a sample of real transcripts across your shortlist. On LLM Gateway, switching models means changing one parameter:

from openai import OpenAI

client = OpenAI(
    base_url="https://llm-gateway.assemblyai.com/v1",
    api_key="YOUR_ASSEMBLYAI_API_KEY",
)
transcript = open("call.txt").read()  # text from your speech-to-text request

# Your current model plus the shortlist from the benchmark grid
for model in ["claude-sonnet-5", "gemma-4-31b", "gpt-6-luna"]:
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "user", "content": f"Summarize this call in five bullet points:\n\n{transcript}"}
        ],
    )
    print(model, response.choices[0].message.content, sep="\n")

For a full evaluation with the same tasks, rubric, and graders as the public benchmark, run the open-source Voice Standard Bench code against your own transcript set. It reports quality, latency, and cost for your data.

4. Switch, then keep measuring

Move one workload at a time. Keep the old model as a fallback for the first few weeks and spot-check outputs. Re-run your comparison when new models land on LLM Gateway, because the cheapest good-enough model changes every few months.

A good time to do this: model retirements

Model retirements force a migration anyway, so they're the cheapest moment to re-evaluate. Claude Sonnet 4.5 on LLM Gateway is scheduled to retire on November 30, 2026. If you're on it, don't just move to its direct successor by default. Check the benchmark grid for your task and length first. A cheaper model may cover the workload at the quality you need.

The real optimization is per workload

Routing, caching, and batching all help. But for voice products, the biggest lever is simpler: stop paying frontier prices for narrow tasks on short transcripts, and save the larger models for the work that needs them. The benchmark tells you where to look. Your own transcripts tell you what to ship.

If you're processing at high volume and want help mapping workloads to models, talk to our team. For more on LLM Gateway itself, see What is LLM Gateway? and our comparison with OpenRouter and LLM Gateway.io. For hands-on examples, try automatic summarization with LLMs in Python or extracting call insights with LLMs.

Frequently asked questions

How do I reduce LLM costs without losing quality?

Match each workload to the smallest model that meets your quality bar for that task and transcript length. Use a task-specific benchmark like Voice Standard Bench to shortlist candidates, test them on your own transcripts, and switch one workload at a time with a fallback in place.

Are open-weight models good enough for summarizing transcripts?

Often, yes, for short and mid-length transcripts. In Voice Standard Bench, small models score close to much larger ones on transcripts under about two minutes, then fall behind at meeting length. For hour-long recordings, larger models are usually worth the cost.

What is the cheapest model on LLM Gateway?

As of October 8, 2026, Nemotron Lightning 3.5 30B A3B has the lowest list price at $0.05 input and $0.20 output per 1M tokens. The cheapest model isn't always the cheapest choice, though. Check its quality and reliability for your task first.

Does transcript length change which LLM I should use?

Yes. It's the central finding of Voice Standard Bench. Small, fast models do well on short transcripts like voicemails and quick calls, but quality drops as transcripts approach meeting length, where larger models pull ahead.

How do I test a cheaper model on my own data?

Send the same transcripts and prompts to several models through LLM Gateway by changing the model parameter, then compare outputs. For a full evaluation, run the open-source Voice Standard Bench code on your own transcript set.

What should I do when a model I use is retired?

Treat it as a chance to re-evaluate, not just upgrade. Check the benchmark results for your task and transcript length, test two or three candidates on your own data, and pick the cheapest model that meets your quality bar.