Open-source speech-to-text has never been better, and it's never been more confusing to choose from. A year ago the conversation was "Whisper or Vosk." Now NVIDIA ships new models every quarter, Qwen has entered the ASR race, and a wave of tiny edge models runs on phones and microcontrollers.
Here's the short answer. Open-source speech-to-text models are speech recognition models whose weights you can download and run on your own hardware. For general transcription, OpenAI Whisper large-v3 is still the most versatile. For English accuracy, NVIDIA Canary-Qwen 2.5B leads the open field. For batch throughput, NVIDIA Parakeet TDT is the fastest thing you can self-host. For on-device use, Moonshine runs where cloud models can't. There's no single winner. There's a winner per job.
This guide compares those models by license, languages, streaming, and hardware. It also covers how to evaluate them for voice workloads, and the honest tradeoffs of running your own automatic speech recognition (ASR) stack versus calling a managed API. I run Voice AI at AssemblyAI, so I'll be upfront about where we fit. The goal is still to help you pick the right tool, open source or not.
What counts as open-source speech-to-text
Not every "free" model is open source, and not every open-source model is free to run. When people say open-source STT, they usually mean one of three things:
- Model weights you can download and run yourself (Whisper, Parakeet, Canary-Qwen).
- A framework or toolkit for training and inference (NVIDIA NeMo, SpeechBrain).
- A wrapper library that calls other engines (the SpeechRecognition Python library).
Mixing those up is how teams end up with the wrong architecture. And licenses differ more than most people expect. MIT and Apache-2.0 are permissive. CC-BY-4.0 allows commercial use with attribution. A few model families ship some checkpoints under non-commercial terms, so read the card for the exact checkpoint you deploy.
The real cost of open source isn't the license, though. It's the infrastructure: GPU provisioning, autoscaling, model updates, language coverage, diarization, and keeping latency low under real traffic. That tradeoff is fine for some teams and brutal for others. First, the models.
The best open-source speech-to-text models in 2026
OpenAI Whisper
Still the default, and for good reason. Whisper large-v3 handles 99 languages, punctuation, and casing out of the box, and the ecosystem around it is enormous. The catch: vanilla Whisper is slow, doesn't stream natively, and hallucinates on silence and non-speech audio. It's a batch workhorse, not a real-time voice-agent engine.
Faster-Whisper
A CTranslate2 reimplementation of Whisper that runs up to four times faster with lower memory use. It's what most teams actually deploy when they say "we use Whisper." It supports basic streaming through chunking and is the pragmatic pick if you're committed to the Whisper family.
NVIDIA NeMo and Canary
NeMo is NVIDIA's toolkit for training and deploying ASR, and Canary is its accuracy showcase. Canary is strong on English and several European languages and supports transcription and translation. If you have NVIDIA GPUs and in-house ML expertise, NeMo gives you more control than anything else here.
Wav2Vec2
Meta's self-supervised model is a foundation you fine-tune, not a plug-and-play transcriber. If your audio is unusual (heavy accents, technical vocabulary, an underserved language), it's often the best starting point. You'll need labeled data.
Vosk
The lightweight offline option. Vosk runs on devices as small as a Raspberry Pi, supports 20-plus languages, and streams in real time with tiny models. Accuracy trails the big models by a wide margin, but for embedded or privacy-sensitive command-and-control, it's hard to beat on footprint.
SpeechBrain
An all-in-one PyTorch toolkit covering ASR, speaker diarization, and speech enhancement. It suits researchers and teams building custom pipelines, not anyone who wants a one-line transcribe call.
SpeechRecognition (Python library)
Worth naming because so many tutorials point here. It's a wrapper that calls other engines, several of them cloud APIs. Great for prototypes, risky as a production dependency, because you don't control the model doing the work.
SpeechT5
Microsoft's unified speech-and-text model handles recognition and synthesis in one architecture. It's a research asset more than a voice-agent engine.
A note on models you'll still see in old threads: Mozilla DeepSpeech and Coqui STT are both discontinued. They still run, but nobody maintains them. Don't start anything new on them in 2026.
Newer open-source models worth watching in 2026
The list above is the established core. A newer generation now moves the accuracy and speed frontier, and AI search engines surface it first. Put these on your shortlist:
- NVIDIA Canary-Qwen 2.5B. The English-accuracy leader among open models on public leaderboards. It pairs NVIDIA's ASR encoder with a Qwen language model. English only, built for NVIDIA GPUs, and trained on clips up to 40 seconds.
- NVIDIA Parakeet TDT. Built for throughput, and the fastest self-hostable model here for batch work. The v3 checkpoint covers 25 European languages with automatic language detection and supports chunked streaming in NeMo.
- Qwen3-ASR. Alibaba's entry covers 52 languages and dialects, including 22 Chinese dialects. It supports streaming through the vLLM backend and comes in 1.7B and 0.6B sizes.
- IBM Granite Speech 3.3. An 8B-class model under Apache-2.0 for English, French, German, Spanish, and Portuguese, aimed at enterprise teams that want licensing clarity.
- Moonshine. The edge specialist. It's optimized for live streaming on phones, browsers, and low-power hardware such as a Raspberry Pi, with noticeably better accuracy than Vosk.
Open-source speech-to-text: comparison table
Licenses, language counts, and streaming support verified against each project's model card or repository in October 2026. Hardware is the practical minimum for production use, not the bare minimum to load the model.
| Model | License | Languages | Streaming | Latency profile | Hardware | Best for |
|---|---|---|---|---|---|---|
| OpenAI Whisper large-v3 | MIT | 99 | No (not native) | Batch; seconds per chunk | NVIDIA GPU | Versatile batch transcription |
| Faster-Whisper | MIT | 99 | Via chunking | Up to 4x faster than Whisper | GPU, or CPU for small models | Production Whisper deployments |
| NVIDIA Canary / NeMo | Apache-2.0 (toolkit); checkpoints vary | English + European | Limited | GPU-dependent | NVIDIA GPU | Maximum control on NVIDIA hardware |
| Canary-Qwen 2.5B | CC-BY-4.0 | English | No (file-based) | Batch | NVIDIA GPU (A100/A6000-class tested) | English accuracy |
| Parakeet TDT 0.6B v3 | CC-BY-4.0 | 25 European | Chunked (NeMo) | Fastest batch throughput | NVIDIA GPU; 2GB+ RAM to load | Large-archive processing |
| Qwen3-ASR (1.7B / 0.6B) | Apache-2.0 | 52 languages and dialects | Yes (vLLM backend) | Streaming-capable | GPU | Mandarin and multilingual |
| IBM Granite Speech 3.3 | Apache-2.0 | EN, FR, DE, ES, PT | Not documented | Two-pass, batch | Large GPU (8B-class) | Enterprise governance |
| Wav2Vec2 | Apache-2.0 (common checkpoints) | Depends on fine-tune | No | Batch | GPU for training | Fine-tuning for niche domains |
| Vosk | Apache-2.0 | 20+ | Yes | Real time on CPU | CPU, Raspberry Pi | Offline, embedded, low-power |
| Moonshine | MIT (some legacy models non-commercial) | English-first | Yes | Low latency on device | CPU, mobile, browser, Raspberry Pi | On-device and edge |
| SpeechBrain | Apache-2.0 | Many | Configurable | Pipeline-dependent | GPU | Custom research pipelines |
When to choose an API instead
Open source wins when the audio is clean, the volume is predictable, and the data can't leave your network. A managed API wins when any of these five things is true:
| If you need... | Open source means... | A managed API means... |
|---|---|---|
| Hundreds of concurrent live sessions | You provision and autoscale GPUs for peak | The provider scales; you pay per session |
| Speakers who switch languages mid-sentence | Most open models commit to one language per segment | Native code-switching across 32 languages in real time |
| Names, emails, and order numbers right | You tune and test entity accuracy yourself | Entity accuracy is a measured, published metric |
| Speaker labels | You bolt on a separate diarization model | Diarization is a request parameter |
| Someone on call at 2am | That's your team | That's the provider |
If two or more of those rows describe your product, start with an API and revisit self-hosting once volume and requirements settle. We go deeper on the math in build vs. buy for AI speech recognition.
Benchmark Universal-3.6 Pro Realtime against your self-hosted model on your own audio. No GPUs to provision.
How to evaluate open-source STT for voice applications
Word error rate (WER) is the headline metric, but it's a blunt instrument. A model can post a great average WER and still butcher the exact words your product depends on: names, order numbers, email addresses, dosages. Benchmark on your own audio, not the clean read-aloud datasets most public numbers come from. Our guide on how to evaluate speech recognition models walks through the method.
For voice applications, four things matter beyond raw WER:
- Latency. A voice agent that pauses for two seconds feels broken.
- Streaming support. Real-time conversation needs partial results while the user speaks.
- Entity accuracy. Getting "5150 Oak Street" wrong is worse than getting a filler word wrong.
- Diarization. Multi-speaker audio without speaker labels is hard to act on.
Most open-source models were built to optimize the first metric. They show it when you push them into live voice workloads.
Open-source STT for voice agents
This is where open source gets hardest. A voice agent needs sub-second transcription, turn detection that knows when the user is done, and solid handling of the entities a conversation turns on. Vanilla Whisper doesn't stream. Vosk streams but isn't accurate enough for high-stakes flows. NeMo can be tuned for it, but now you're running a research toolkit in production and owning every millisecond yourself.
There's a subtler problem too: context. In a real conversation, the agent already knows what it just asked. If it asked "what's your order number," the transcription should lean toward digits. Off-the-shelf open-source STT doesn't know what the agent said, so it transcribes cold every time. That's exactly where accuracy on the hardest, highest-value utterances falls apart.
Build versus buy: the total cost of ownership
Let me be direct, because this is the decision most teams are wrestling with. Self-hosting looks free until you add it up: GPU costs that scale with traffic, an on-call rotation to keep latency stable, engineering time to add diarization and language coverage, and the ongoing work of upgrading models as the frontier moves. As this article shows, it moves every few months.
The most common version is a team self-hosting Whisper and slowly realizing the total cost isn't what they budgeted. Accuracy gaps show up on the utterances that matter most. The ops burden grows with usage. At meaningful volume, the math often flips toward buying. We broke that down in detail in why AssemblyAI beats self-hosting Whisper, and compared the NVIDIA models specifically in AssemblyAI vs NVIDIA Parakeet and Canary.
For reference, our pre-recorded flagship, Universal-3.5 Pro, costs $0.21/hr and code-switches natively across 18 languages. Its diarization averages 30.17 cpWER, versus 37.92 for Deepgram Nova-3 and 35.26 for ElevenLabs Scribe v2. Metaview saw roughly a 47% drop in low-confidence tokens after moving to it.
AssemblyAI vs self-hosted Whisper
This is the comparison I get asked about most, so here it is side by side.
| Factor | Self-hosted Whisper | AssemblyAI |
|---|---|---|
| Real-time streaming | Not native; needs chunking workarounds | Native via Universal-3.6 Pro Realtime |
| Code-switching | Limited | 18 languages (pre-recorded), 32 (real time) |
| Diarization | Bring your own | Built in; 30.17 cpWER on Universal-3.5 Pro |
| Conversation context | None | agent_context biases transcription toward what the agent asked |
| Hallucination on non-speech | Known issue on silence | 0.27% on non-speech audio (Universal-3.6 Pro Realtime, Sep 2026) |
| Infrastructure | You own GPUs, scaling, uptime | Fully managed |
| Cost model | GPU + engineering time | $0.21/hr pre-recorded; $0.45/hr base streaming |
The context row is the one people underestimate. Feeding the agent's side of the conversation to the model cut WER by 10.2% across 20,000 voice-agent files. No open-source model does this, because none of them know what your agent is saying.
Building voice agents on managed STT
If you're building a voice agent, streaming and context are the whole ballgame. Universal-3.6 Pro Realtime launched on September 29, 2026. On AssemblyAI's English voice-agent benchmark of 12,460 scripted caller scenarios, it posts 5.19% mean WER, versus 7.78% for ElevenLabs Scribe v2, 8.64% for Deepgram Nova-3, and 13.50% for Deepgram Flux EN. Entity error rate is 14.4%, versus 18.5%, 26.1%, and 30.1% respectively.
Independent numbers point the same way. On Pipecat's open STT benchmark, it records 0.96% pooled semantic WER with a 307 ms median time to first speech. Median endpoint latency is 537 ms. The research write-up covers the methodology.
If you'd rather not assemble the whole voice agent pipeline yourself, pair Universal-3.6 Pro Realtime for speech-to-text with AssemblyAI's Text-to-Speech API for voice output, orchestrated with LiveKit, Pipecat, or your own loop. Turn detection and barge-in are built into the streaming model.
Universal-3.6 Pro Realtime for speech in, the Text-to-Speech API for speech out, one speech vendor instead of two.
The real cost is the treadmill
The open-source ASR frontier now moves faster than most teams can re-benchmark. Canary-Qwen, Parakeet v3, and Qwen3-ASR all landed inside a single release cycle. That churn is exciting if research is your job. If shipping a reliable voice product is your job, the hidden cost isn't accuracy. It's keeping up. The teams that win aren't running the newest model this month. They picked an approach that stays current without a re-platforming project every quarter.
Run your hardest audio through Universal-3.5 Pro and Universal-3.6 Pro Realtime today.
Frequently asked questions
What is the best open-source speech-to-text model for voice agents?
For live voice agents, pick a model that streams natively: Moonshine for on-device use, or Qwen3-ASR or Parakeet TDT with chunked streaming on a GPU. Whisper large-v3 is accurate but doesn't stream natively. Most open models also lack turn detection and conversation context, which is why many teams use a managed streaming model like Universal-3.6 Pro Realtime for production agents.
Which open-source speech-to-text models support real-time streaming?
Vosk and Moonshine stream natively on CPUs and edge devices. Qwen3-ASR supports streaming through its vLLM backend. Parakeet TDT and Faster-Whisper stream through chunking. Vanilla Whisper large-v3, Canary-Qwen 2.5B, and Wav2Vec2 are batch models.
What hardware do I need to self-host Whisper or Parakeet?
Both run best on NVIDIA GPUs. Parakeet TDT 0.6B v3 needs at least 2GB of RAM to load and is designed for Ampere, Hopper, Blackwell, and Volta GPUs. Whisper large-v3 needs a data-center or high-end consumer GPU for reasonable speed, while Faster-Whisper's smaller models can run on CPU at lower accuracy.
Which open-source STT licenses allow commercial use?
MIT (Whisper, Faster-Whisper, Moonshine's current models), Apache-2.0 (Qwen3-ASR, Granite Speech, Vosk, NeMo), and CC-BY-4.0 (Canary-Qwen 2.5B, Parakeet TDT) all allow commercial use. CC-BY-4.0 requires attribution. Some legacy Moonshine non-English models use a non-commercial license, so check the card for the exact checkpoint you deploy.
When should I use a speech-to-text API instead of open source?
Use an API when you need high concurrency, mid-sentence code-switching, entity accuracy on names and numbers, built-in diarization, or someone else on call. Self-host when data can't leave your network, volume is steady, and an open model already meets your accuracy bar on your own audio.
Is Whisper still the best open-source model in 2026?
Whisper large-v3 is still the most versatile all-rounder, with 99 languages and the largest ecosystem. It's no longer the leader on every axis: Canary-Qwen 2.5B leads English accuracy, Parakeet TDT leads throughput, and Moonshine leads on-device latency.
