Why small LLMs win on short transcripts and lose at meeting length

On a 30-second voicemail or a 2-minute call, a small, fast LLM scores within a few points of the biggest models and answers in about a second. On an hour-long meeting, it falls behind, and the larger model is worth paying for.

That's the short version of the SLM vs LLM question for voice, and it comes from measurement, not intuition. In the current Voice Standard Bench run, the top-scoring model costs 15 to 77 times more per hour of audio than a model within two quality points of it. On hour-long transcripts, that premium narrows to about 4×, because that's where the bigger models actually earn it.

Here's the data: which small models hold up, the one we host ourselves, where each one breaks, why, and how to route your traffic by transcript length.

What counts as a small LLM

There's no official cutoff, but the common convention is that small language models (SLMs) run from a few hundred million to around 10 billion parameters. Large language models are everything above that, up to the frontier models from OpenAI, Anthropic, and Google.

Model names add a wrinkle. Mixture-of-experts models like "Nemotron 3 Nano 30B A3B" have 30 billion parameters in total but activate only about 3 billion per token. They're cheap and fast to serve, so in practice they compete with small models on price and speed even though they aren't small on paper.

So in this post, "small" means two things. Qwen3.5 4B Fast is a true small model. The rest are small open models: cheap, fast-to-serve open-weight models, some of them mixture-of-experts, that you'd consider for the same jobs. Here are the ones on LLM Gateway worth knowing:

Prices are list prices as of October 20, 2026. These open models carry no regional surcharge, though Nemotron 3 Nano has separate US and EU rates; check the pricing page for current rates and the model list in the docs for every model's parameters and context window.

The finding in one chart

Voice Standard Bench runs every model on LLM Gateway against real speech-to-text output at four lengths: about 30 seconds, 2 minutes, 15 minutes, and 60 minutes. Here's the overall quality score (0 to 100, every task weighted equally) for four models across those lengths, from the run published October 13, 2026:

Two patterns stand out. gemma-4-31b sits within three to five points of the leader on short inputs, then drops more than 20 points by 15 minutes. And Qwen3.5 4B Fast, the smallest model here, falls steadily from the start. Size and speed buy you a lot on short transcripts. They stop buying you much once the input gets long.

Scores within 2–3 points are effectively tied, so read the gaps, not the exact numbers. The methodology explains how each score is built.

The smallest model we host: Qwen3.5 4B Fast

Qwen3.5 4B Fast is the open-source Qwen3.5 4B model, hosted by AssemblyAI on our own GPUs in a latency-optimized configuration with a 32,768-token context window. It costs $0.10 per million input tokens and $0.50 per million output tokens.

Where it wins: speed. It's the fastest model in Voice Standard Bench at every length up to 15 minutes: 0.28 seconds median on 30-second transcripts, 0.60 seconds at 2 minutes, and 1.49 seconds at 15 minutes. In our launch testing on voice rewrite tasks, it averaged 612 ms, 1.9× faster than GPT-4.1 at 94% lower cost per hour of audio. On 2-minute transcripts it costs about $0.003 per hour of audio.

What it's built for: short, rewrite-style work where a user is waiting. That means dictation cleanup, transcript rewrites into emails or notes, live formatting while someone is still talking, and compressing a conversational turn into one line. Be clear about the quality trade, though: on the bench it scores 62 on text correction and 81 on dictation at 30 seconds, against 93 and 98 for gemma-4-31b. Pick it when your latency budget is under about 500 ms; otherwise, route rewrites to a larger model.

Where it falls off: longer, harder reading. Its information-extraction score is 80 on 30-second inputs, 51 at 2 minutes, and 33 at 15 minutes. Its overall score drops from 74 to 31 between the shortest and longest transcripts. We host this model, and we're publishing where it struggles anyway, because you need that to route traffic well.

There's also a hard limit on what it can do through the API. It supports max_tokens, temperature, and stream only. No tool calling, no structured output. So it's the wrong pick for JSON extraction or voice agents, no matter how short the input.

Small open models that hold up longer

If you need more than a rewrite but still want small-model speed and price, three open models are worth a look, each with a different strength.

gemma-4-31b (AWS Bedrock) is the speed-and-quality pick on short transcripts. It answers a 30-second input in under a second at 93 quality, against GPT-6 Luna's 1.7 seconds, and a 2-minute call in 1.5 seconds against 2.4. It's also the lowest-price model within 8 points of the top for text correction (84). The catch is length: it falls to 64 at 15 minutes.

GLM-5.3 Flash (DigitalOcean, Fireworks) is the lowest-price model within 8 points of the top for summarization (91) and dictation (96). At 15 minutes it scores in the mid-70s (74), well behind the leaders there.

Nemotron 3 Nano 30B A3B (AWS Bedrock) costs about $0.002 or less per hour of audio at every length. But on information extraction it scores 73, 58, 49, and 52 across those lengths. It's a good reminder that the cheapest model isn't the cheapest choice if it can't do the job.

Why small models fall off at meeting length

The obvious guess is that long transcripts overflow a small model's context window. That's not it. An hour-long meeting is well over 10,000 input tokens, which still fits comfortably inside Qwen3.5 4B Fast's 32,768-token window.

The scores point somewhere else: what the task asks the model to keep track of. Look at what the rubrics reward.

• Summarization coverage asks whether the summary captures the topics, decisions, and action items a listener would consider the point of the call. The methodology describes it as measuring the amount of information a model can keep track of.
• Extraction completeness asks whether the output captures the action items, decisions, and named entities the transcript actually contains.

On a 30-second voicemail, there's one caller, one request, maybe one callback number. Almost any competent model gets all of it. On an hour-long meeting, there are dozens of decisions and owners spread across the conversation, with topics that come back 40 minutes later. Every item a model drops costs it on coverage and completeness. Smaller models drop more of them as transcripts grow, and the scores show it.

Two caveats keep this honest. First, this is what the scores show, not a full account of the mechanism, so test it on your own long transcripts before you rule a model in or out. Second, dictation was only measured at 30 seconds and 2 minutes, because dictation sessions are short. The meeting-length results come from summarization, extraction, and text correction.

Where a model is hosted matters as much as its size

Size isn't the only thing that makes a model fast. Serving matters just as much.

In Voice Standard Bench, the same open-model families hosted elsewhere take 10 to 20 seconds to respond. That's the difference between a feature that feels live and one that feels broken. Qwen3.5 4B Fast gets its sub-second responses from a deployment tuned for exactly this work, not only from having 4 billion parameters.

Hosting also decides where your data goes. The provider that serves a model is the one that processes your transcript. Qwen3.5 4B Fast runs on AssemblyAI infrastructure; the other models in this post run through AWS Bedrock, DigitalOcean, or Fireworks. The docs list the provider for every model, so check it alongside the score.

The counterexample: low-cost and proprietary

"Small" and "open" aren't the same as "cheap", and the best value in the bench isn't an open model.

GPT-6 Luna scores 95, 92, 87, and 85 across the four lengths. Small open models are close to it on short transcripts and fall well behind past two minutes. It costs about half a cent per hour of audio in the current run and scores within a point of the best open model at every length. So the open models that undercut it on price save fractions of a cent, and past two minutes they give up at least 10 quality points to do it.

What GPT-6 Luna costs you is latency. On short inputs it's slower than gemma-4-31b and much slower than Qwen3.5 4B Fast. Which makes the real decision clearer: on short transcripts, a small open model is a latency choice more than a cost choice. On long transcripts, it's rarely the right choice at all.

Route by transcript length in production

You don't have to pick one model. Route each request by task and length. Here's a starting rule based on the current run:

• Under about 2 minutes: use the fastest pick that fits the task. Qwen3.5 4B Fast for rewrites and dictation cleanup when you need responses in under about 500 ms (otherwise a larger model); gemma-4-31b for short summaries and extraction.
• 2 to 15 minutes: compare the recommended and budget picks at your volume. GPT-6 Luna scores within two points of the leader at 15 minutes (87 vs 89) at a fraction of the cost.
• Meeting length: start with GPT-6 Luna. It scores 85 at 60 minutes for about $0.004 per hour of audio, against GPT-6.1 Sol's 87 at about $0.09 per hour. That's within the 2–3-point range we treat as a tie, so let price decide. The top 60-minute score is GPT-6 Astra's 88.

If a transcript falls between two lengths, route it with the longer one. And always set a fallback on a different provider; automatic fallbacks take one extra field. Here's the rule as code, using the OpenAI SDK pointed at LLM Gateway. It passes an AssemblyAI transcript ID so the Gateway injects the transcript text into the prompt for you:

from typing import List, Tuple
from openai import OpenAI

WORDS_PER_MIN = 150      # typical conversational speech
TOKENS_PER_WORD = 1.3    # rough English average

# Context windows from the LLM Gateway model list
CONTEXT = {
    "qwen3.5-4b-32k-fast": 32_768,
    "gemma-4-31b": 256_000,
    "gpt-6-luna": 922_000,
    "gpt-6.1-sol": 922_000,
}


def estimate_tokens(duration_s: float) -> int:
    return int(duration_s / 60 * WORDS_PER_MIN * TOKENS_PER_WORD)


def pick_model(task: str, duration_s: float) -> Tuple[str, List[str]]:
    """Return (primary, fallbacks) by task and transcript length."""
    if duration_s <= 120:
        if task == "rewrite":
            return "qwen3.5-4b-32k-fast", ["gemma-4-31b"]
        return "gemma-4-31b", ["gpt-5.6-luna"]
    if duration_s <= 900:
        return "gpt-6-luna", ["gpt-5.6-luna"]
    return "gpt-6-luna", ["gemini-4-argon"]


def fits(model: str, duration_s: float, prompt_tokens: int = 300, max_tokens: int = 1000) -> bool:
    if model not in CONTEXT:
        return True
    return estimate_tokens(duration_s) + prompt_tokens + max_tokens < CONTEXT[model]


def run(client: OpenAI, transcript_id: str, duration_s: float, task: str, instruction: str) -> str:
    model, fallbacks = pick_model(task, duration_s)
    if not fits(model, duration_s):
        raise ValueError(f"Transcript is too long for {model}")
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": instruction + "\n\n{{ transcript }}"}],
        max_tokens=1000,
        extra_body={
            "transcript_id": transcript_id,
            "fallbacks": [{"model": m} for m in fallbacks],
        },
    )
    print(f"routed to {model}, answered by {response.model}")
    return response.choices[0].message.content


client = OpenAI(
    api_key="<YOUR_API_KEY>",
    base_url="https://llm-gateway.assemblyai.com/v1",
)

print(run(client, "<YOUR_TRANSCRIPT_ID>", duration_s=95, task="summary",
          instruction="Summarize this call in three bullet points."))

A few notes on the code. transcript_id and fallbacks aren't standard OpenAI arguments, so they go in extra_body. The {{ transcript }} tag must be written exactly like that, with the spaces. Several of these models don't accept temperature, so the request leaves it out. And the thresholds are a starting point: if your 2-to-15-minute traffic clusters near the short end, test gemma-4-31b there too. If you're processing your own transcripts from a different source, swap transcript_id for the text itself in the message.

For the dollar math and a step-by-step migration playbook, see how to cut LLM costs on voice workloads. For task-specific starting code, there are tutorials to summarize meeting transcripts and extract call insights.

Test it on your own transcripts

Our transcripts aren't yours. Your callers have different accents, your domain has its own vocabulary, and your prompts ask for different output. Any of those can move the point where a small model stops being good enough. A benchmark tells you which two or three models to try first, not which one to ship.

Three ways to find your own crossover point:

1. Playground compare mode. Upload a short call and a long one, run the same prompt through a small model and a large one, and compare the answers side by side.
2. "Try in Gateway" links. Every cell on the full results page opens the playground with that model and task prompt loaded.
3. The open-source benchmark. Voice Standard Bench is on GitHub. Run it on your own transcripts, bucketed by your own lengths.

Small models are a length decision

The SLM vs LLM debate usually gets framed as a question of capability: can a small model do the job? For voice, the better question is how long the job is.

Under two minutes, small and open models are close enough to the leaders that speed and price should decide. Past 15 minutes, they aren't, and no amount of savings makes up for a meeting summary that misses half the decisions. The teams that get the most out of small models won't be the ones that pick a winner. They'll be the ones that measure their transcript lengths and route each request to the model that fits it.

Frequently asked questions

What is the difference between an SLM and an LLM?

A small language model (SLM) typically has from a few hundred million to around 10 billion parameters, while a large language model (LLM) has far more, up to today's frontier models. SLMs are faster and cheaper to run; LLMs handle longer, more complex inputs better. On voice transcripts, Voice Standard Bench shows small models scoring close to large ones under two minutes and falling behind at meeting length.

Should I use an SLM or an LLM for my voice app?

It depends on transcript length and your latency budget. For transcripts under about two minutes, a small or open model is usually close enough in quality and much faster. For 15-minute calls, compare both at your volume, and for hour-long meetings, start with a larger model.

Are small language models good enough for summarizing call transcripts?

Yes for short calls, often not for long ones. In Voice Standard Bench, small open models score within a few points of the leaders on calls under two minutes, but they fall well behind by 15 minutes. Test your own call lengths before you switch.

What is the fastest LLM for dictation cleanup?

Qwen3.5 4B Fast is the fastest model in Voice Standard Bench at every transcript length up to 15 minutes, with a 0.28-second median on 30-second inputs. In launch testing on voice rewrite tasks, it averaged 612 ms. It's hosted by AssemblyAI on LLM Gateway and built for rewrite-style work like dictation cleanup. Speed is the trade: at 30 seconds it scores 62 on text correction and 81 on dictation, against 93 and 98 for gemma-4-31b, so pick it for latency budgets under about 500 ms and route rewrites to a larger model otherwise.

Why does LLM accuracy drop on long transcripts?

Long transcripts contain more decisions, owners, and details that a model has to keep track of, and smaller models drop more of them as length grows. It usually isn't a context-window problem: an hour-long meeting is well over 10,000 tokens but fits inside a 32,768-token window. Voice Standard Bench's coverage and completeness scores capture exactly what gets dropped.

Which small models can I run through AssemblyAI's LLM Gateway?

LLM Gateway includes Qwen3.5 4B Fast, which AssemblyAI hosts itself, plus small open models such as gemma-4-31b, Nemotron 3 Nano 30B A3B, GLM-5.3 Flash, and GPT OSS 20B, alongside frontier models from OpenAI, Anthropic, and Google. All of them use the same AssemblyAI API key and OpenAI-compatible API, so switching models means changing one string.