The real cost of self-hosting open-source speech-to-text

Open-source speech-to-text got good. That part isn't marketing spin, and pretending otherwise is how you lose an engineer's trust in the first paragraph.

Whisper Large-v3 transcribes clean English about as well as anything you can buy. Qwen3-ASR and NVIDIA's Parakeet and Canary models are genuinely strong. Mistral shipped Voxtral. And the checkpoints are free — pip install, download the weights, point it at an audio file, done. Spin one up on Baseten, Modal, or Fireworks and you've got a working transcription endpoint in an afternoon. For a growing number of teams, that afternoon ends with a reasonable question: why are we paying an API for this?

It's a fair question. Here's the more useful one: what does it actually cost to run that endpoint in production for the next two years?

Because the sticker price of an open model is zero, and the total cost of ownership is not. The gap between those two numbers is where most self-hosting decisions quietly go wrong.

What "free" actually buys you

A model checkpoint is one component of a transcription product. It is not the product.

When you call a managed speech-to-text API, the price covers the model plus the parts you'd otherwise have to build, operate, and keep alive yourself: the inference infrastructure, the accuracy work on messy real-world audio, speaker diarization, entity formatting, streaming, language handling, and the on-call rotation for when any of it breaks at 2am. Download a raw checkpoint and you own every one of those. The weights are free. The system around them is the job.

So the honest comparison isn't "$0 vs. $0.21 an hour." It's "your engineering org vs. $0.21 an hour." Let's break down what lands on your side of that trade.

Hidden cost #1: the GPU bill nobody quotes you

Open weights don't come with a price tag, but GPUs do. To serve transcription at production quality and latency you're renting accelerators — and you pay for them whether or not audio is flowing through them.

This is the detail that surprises teams. Utilization is rarely high. Traffic is spiky, models sit warm waiting for requests, and a GPU idling at 15% still bills at 100%. Serverless platforms like Modal soften the idle problem but hand you a cold-start problem instead: scale to zero and your next request waits for a container and a multi-gigabyte model to load. Keep instances warm to avoid that and you're back to paying for idle. Autoscaling a GPU fleet against bursty voice traffic is a real, ongoing engineering project, not a checkbox.

A managed API bills per second of audio — AssemblyAI's pricing runs $0.15–$0.21/hr for pre-recorded transcription with no minimums and no idle time — because absorbing that utilization math is the vendor's problem, not yours.

Hidden cost #2: everything the checkpoint doesn't do

Here's where the "it's basically the same model" argument falls apart. Raw ASR checkpoints give you a transcript. Production voice products need a lot more than a transcript.

Speaker diarization. Who said what. Most open checkpoints don't do it at all, and the ones that gesture at it don't hold up on the audio that matters — short back-and-forth turns, interruptions, overlapping speech. Getting diarization right is its own research problem; AssemblyAI's latest models produce the transcript and the speaker boundaries jointly, optimized for concatenated word error rate rather than the looser metrics that make demos look fine and production look broken.

Entity accuracy. In the real world people say credit card numbers, email addresses, medication names, and account IDs — the exact tokens where a single wrong character makes the transcript worthless. A base model transcribes the words; it doesn't reliably format an alphanumeric confirmation code spelled out over a phone line.

Streaming. Most open checkpoints are async-only. If you're building anything live — a voice agent, real-time captions, in-call intelligence — you get to build the streaming layer yourself: chunking, partial hypotheses, endpointing, the works. Real-time transcription is a different engineering problem than batch, not a config flag.

Code-switching, redaction, formatting. Multilingual speakers switch languages mid-sentence. Recordings need PII redacted. Numbers, dates, and punctuation need to render like a human wrote them. Every one of these is a feature you either buy once or build and maintain forever.

You can build all of it. The question is whether reimplementing diarization and a streaming stack is the best use of your team versus shipping your actual product.

Hidden cost #3: accuracy on audio that isn't clean

Benchmarks are run on clean audio. Your users are not clean audio.

They're on a phone in a moving car, in a call center with three conversations bleeding together, using product names and industry jargon no model saw in training, in accents the leaderboard underweighted. This is exactly where open checkpoints tend to slip — and where one specific failure mode gets expensive: hallucination. Whisper is known to invent fluent, confident text during silence or noise. It's not a rounding error; in AssemblyAI's own testing, Universal-3 Pro's hallucination rate runs about 30% lower than Whisper's, and on published head-to-head benchmarks, open models trail badly on the audio that resembles production — Voxtral Mini, for instance, posts nearly 18% word error rate on code-switched speech versus under 8% for a managed flagship. A transcript that's wrong in a plausible way is more dangerous than one that's obviously garbled, because nothing downstream flags it.

Closing that gap yourself means fine-tuning, building an eval set from your own audio, and re-running it every time you touch the model. That's a standing capability you'd be committing to own. And word error rate alone won't even tell you if you're winning — you need to measure the errors that actually break your product.

Hidden cost #4: reliability is now your pager

The afternoon demo runs on one GPU serving one request. Production is a different animal.

Concurrency at real volume, retries and graceful degradation, latency under load, model-version upgrades without downtime, region failover, monitoring, uptime — this is the operational surface of a speech platform, and self-hosting means it's yours. Someone carries the pager. A managed platform runs hundreds of millions of inference calls a month with unlimited concurrency and no rate limits, because reliability at scale is the product it's selling. When you self-host, reliability at scale is a headcount question.

When self-hosting is the right call

None of this means "never self-host." There are real cases where owning the stack is the correct decision:

You're doing research or a prototype where model control matters more than production polish. You're running massive offline batch jobs where latency is irrelevant and you can keep GPUs pinned at high utilization, which fixes the economics. You have a hard data-isolation requirement that rules out any external call — though note that a managed platform can meet strict needs too, through self-hosted deployment in your own VPC, EU data residency, and a signable Business Associate Addendum for teams handling protected health information. Or you have an ML platform team that wants to own this and has the cycles to.

If one of those is you, self-hosting is defensible. If none of them is you, the "free" model is the most expensive line item on the roadmap — it just doesn't show up on the invoice.

The number that actually matters

The mistake isn't choosing open source. The mistake is comparing the wrong two numbers.

The comparison that matters isn't the model's price against the API's price — it's the fully loaded cost of running your own speech stack, GPUs and engineers and eval pipelines and on-call included, against a per-second bill where someone else owns all of it. Run that math honestly and the answer flips more often than the afternoon-demo instinct suggests. Cheap models are real. Free ones don't exist.

If you want to see where a managed model lands on your actual audio before you commit engineering time to either path, start with a free API key and run it against a batch of your worst recordings — the noisy ones, the accented ones, the jargon-heavy ones. That's the benchmark that decides it.

Frequently asked questions

Is open-source speech-to-text really free?

Open-source speech-to-text models are free to download but not free to run in production. Model weights like Whisper, NVIDIA Parakeet, Qwen3-ASR, and Mistral's Voxtral carry no license fee, but you still pay for the GPUs that serve them, the engineering time to add production features, and the on-call burden to keep them running. The real comparison isn't "$0 vs. an API bill" — it's your fully loaded infrastructure and engineering cost versus a managed per-second price.

How much does it cost to self-host an open-source speech-to-text model?

Self-hosting cost is driven by GPU time, not the model. A single on-demand cloud GPU in the A100/H100 class typically runs $2–4+ per hour and bills whether or not it's actively transcribing, so a model idling at 15% utilization still costs you 100%. On top of that you add engineering time to build diarization, streaming, and formatting, plus ongoing monitoring and model upgrades. By contrast, a managed API like AssemblyAI charges $0.15–$0.21 per hour of audio for pre-recorded transcription, billed per second with no idle cost.

Is self-hosted Whisper cheaper than a speech-to-text API?

It depends almost entirely on GPU utilization. Self-hosted Whisper can be cost-effective for large offline batch jobs that keep a GPU near 100% busy, but for spiky or low-volume workloads the idle GPU time usually makes a per-second managed API cheaper. Whisper also lacks production features and shows higher hallucination rates on real-world audio — AssemblyAI's Universal-3 Pro runs about 30% fewer hallucinations than Whisper. Benchmark both on your own audio and include engineering and infrastructure time, not just the sticker price.

What features do open-source speech-to-text models lack out of the box?

Raw open-source checkpoints transcribe audio but omit most of what production apps need. Commonly missing pieces include speaker diarization (who said what), real-time streaming, PII redaction, entity accuracy for data like credit cards and medication names, code-switching across languages, and consistent formatting. Each one has to be built, integrated, and maintained yourself, whereas a managed API ships them as part of the model or a single parameter.

When does self-hosting open-source speech-to-text make sense?

Self-hosting is a good fit when you have high, steady GPU utilization or strict control requirements. The clearest cases are large offline batch jobs that keep GPUs busy, research and prototyping where you need to modify the model, strict data-isolation rules, and teams that already run a dedicated ML platform. For most real-time or customer-facing products that need accuracy, features, and scale without an ops burden, a managed API is usually the lower-cost and faster path to production.

How much does a managed speech-to-text API cost?

AssemblyAI's pre-recorded speech-to-text starts at $0.15–$0.21 per hour of audio, billed per second with no minimums, and streaming transcription ranges from $0.15 to $0.45 per hour. There are no idle-GPU charges, no rate limits, and unlimited concurrency, so you pay only for the audio you actually process. You can benchmark it against your own recordings with free credits and an API key before committing to either path.