Speech-to-text is an AI-driven technology that converts spoken audio into written text automatically. A deep learning model processes the audio signal, maps sound to words, and returns formatted, punctuated text. It powers transcription, live captioning, voice agents, and searchable audio archives — the top models now transcribe real-world conversations at under 7% word error rate.
That's the short answer. The rest of this guide goes deep: how the technology actually works, where it breaks, what separates a hobby project from production infrastructure, and how to choose a model that won't cost you customers when it mishears a phone number. Whether you're an engineer shipping a feature or a leader sizing up a build, you'll leave knowing exactly what to look for.
Table of contents
- What is speech-to-text technology?
- Real-time, sync, and async speech-to-text
- Speech-to-text vs. related technologies
- How does speech-to-text work?
- Speech-to-text for voice agents
- Common challenges and limitations
- Benefits of speech-to-text
- Applications of speech-to-text
- Free vs. paid speech-to-text
- How to choose the right tool
- Speech-to-text pricing
- Getting started
- Frequently asked questions
What is speech-to-text technology?
Speech-to-text (also called automatic speech recognition, or ASR) automatically converts spoken words into written text with high accuracy. It takes audio — from a microphone or a file — runs it through a speech recognition model trained on massive audio-transcription datasets, and returns clean, formatted text. Good systems handle multiple speakers, several languages in the same recording, and domain-specific vocabulary.
The market reflects how central this has become: analysts project the speech-to-text space will reach roughly $73 billion by 2031. But the number that matters to builders is accuracy on your audio, not a headline forecast.
Types of speech-to-text engines
Not every engine solves the same problem. A few distinctions shape which one fits your use case.
Cloud-based vs. on-premise
Cloud APIs deliver the highest accuracy because they run large models on managed infrastructure — you send audio, you get text, someone else handles the GPUs. On-premise (or self-hosted) deployment keeps audio inside your own environment, which matters for regulated data, at the cost of running the infrastructure yourself.
Open-source vs. proprietary
Open-source models like Whisper give you full control and offline capability, but you own the accuracy tuning, scaling, and maintenance. Proprietary APIs trade that control for higher real-world accuracy, built-in features (diarization, sentiment, redaction), and support. For a deeper breakdown, see our roundup of the top free speech-to-text APIs and open-source engines.
Real-time, sync, and async speech-to-text
How you deliver audio to a speech-to-text system changes everything about latency and cost. There are three modes:
| Method | Best for | Key characteristic |
|---|---|---|
| Real-time (streaming) | Voice agents, live captioning | Low-latency WebSocket connection; partial transcripts return in a few hundred milliseconds |
| Sync | Short clips — dictation, single commands | One HTTP request, response in roughly a hundred milliseconds |
| Async (batch) | Long or bulk audio — podcasts, meetings, call archives | Post a file or URL, poll for the completed transcript |
Most production systems use more than one. A meeting-intelligence product might transcribe recordings async and caption the live call over streaming. For a hands-on streaming walkthrough, see real-time transcription in Python.
Speech-to-text vs. related technologies
People use "speech-to-text," "voice recognition," and "speaker recognition" interchangeably. They're not the same thing.
| Technology | What it does | Question it answers | Applications |
|---|---|---|---|
| Speech-to-text | Converts speech into written text | "What was said?" | Transcription, captioning, search |
| Voice recognition | Understands and acts on spoken commands | "What action?" | Virtual assistants, smart home |
| Speaker recognition | Identifies who is speaking by voice | "Who is speaking?" | Security, personalization, diarization |
If you're weighing these terms, our guide to AI voice recognition covers the overlap in detail.
How does speech-to-text work?
Under the hood, converting audio to text runs through three stages.
1. Audio preprocessing
The system transcodes, normalizes, and segments the incoming sound — cleaning up the signal so the model gets a consistent input regardless of source.
2. Deep learning speech recognition model
A neural model trained on enormous audio-transcript datasets maps acoustic signals to the most probable sequence of words, using surrounding context to disambiguate. This is where modern accuracy gains live. AssemblyAI's flagship async model, Universal-3.5 Pro (model ID universal-3-5-pro), adds native code-switching across 18 languages and contextual prompting so you can bias the model toward the terms that matter for your domain.
3. Text formatting
Raw model output gets punctuation, capitalization, and entity formatting — turning "my number is five five five one two one two" into "My number is 555-1212."
Run your own audio through Universal-3.5 Pro and read the transcript. Free API credit to start—no credit card required.
Factors affecting speech-to-text accuracy
Technical factors: audio quality, file format (uncompressed beats heavily compressed), and recording setup all shape the result. Speaker factors: speaking style, accents and dialects, and specialized vocabulary. Contextual factors: overlapping speakers, code-switching between languages, and background context.
A concrete benchmark makes this real: on AssemblyAI's English voice-agent benchmark, Universal-3.6 Pro Realtime hits a 5.19% word error rate, versus 7.78% for ElevenLabs Scribe v2, 8.64% for Deepgram Nova-3, and 13.50% for Deepgram Flux EN. Accuracy isn't a rounding error — it's the difference between an agent that works and one that frustrates. See the full numbers on our benchmarks page.
Speech-to-text for voice agents
Voice agents are the most demanding application of speech-to-text, because a mistake isn't a typo in a transcript — it's a broken conversation.
The multi-provider challenge
The traditional way to build a voice agent stitches together three vendors: speech-to-text, an LLM, and text-to-speech. Every hop adds latency, integration surface, and a separate bill.
What voice agents require from speech-to-text
- Low latency — end-to-end response around a second
- Turn detection — knowing when a speaker has actually finished a thought, not just paused
- Interruption handling — detecting when the user cuts in
- Entity accuracy — proper nouns, account numbers, medical terms right the first time
On entity accuracy, Universal-3.6 Pro Realtime posts a 14.4% entity error rate against 30.1% for Deepgram Flux EN — with 10.9% on names, 10.0% on codes and IDs, and 2.4% on phone numbers. And its end-of-turn detection combines semantic context with voice activity rather than waiting on raw silence like traditional VAD; on Pipecat's open STT benchmark, the final transcript lands a median 307ms after the speaker stops.
The unified API approach
AssemblyAI's Voice Agent API collapses the stack into a single WebSocket — speech-to-text, LLM, and text-to-speech — built on Universal-3.6 Pro Realtime, where agent_context cuts WER a further 10.2%. It's flat $4.50/hr, roughly 4x cheaper than OpenAI's Realtime API at about $18/hr. We think of ourselves as invisible infrastructure: not the agent, the platform you build it on. For the bigger picture, read our pillar on AI voice agents.
Ready to try it? Talk to a live voice agent built on the API, or get your free API key.
Common challenges and limitations
Accuracy limitations: even the best models struggle with rare terminology and heavy accents — the final mile of edge cases is hard.
Environmental challenges: word error rates can jump from near-zero to 50%+ in noisy, multi-speaker rooms.
Contextual understanding: models don't understand meaning the way humans do; homophones and ambiguous phrasing still trip them up.
Bias and fairness: one widely cited study found a 16-percentage-point accuracy gap between Black and white speakers — a reminder that training data and evaluation matter. If you're evaluating models yourself, start with how to evaluate speech recognition models and why word error rate alone is broken.
Benefits of speech-to-text technology
- Cutting the hours spent on manual transcription and note-taking
- Opening content up through real-time captioning and accessibility support
- Shortening handle times and lifting resolution rates in customer service
- Slashing the cost of documentation versus human transcription
- Turning large volumes of spoken data into analyzable text
- Strengthening compliance and record-keeping with accurate, timestamped records
- Working across devices and slotting into software you already run
Applications of speech-to-text technology
Personal use cases include dictation and note-taking, real-time captioning for accessibility, and voice commands through assistants like Siri, Alexa, and Google Assistant.
Business applications span customer service and call centers, meeting transcription, voice agents, and applying LLMs to spoken data for summarization and Q&A. Speech Understanding features — sentiment, topic detection, entity detection, and PII redaction — turn a raw transcript into structured insight, and the LLM Gateway lets you run that transcript through GPT, Claude, or Gemini with one API.
Real-world example: Commure, which builds ambient clinical documentation, integrated AssemblyAI's models for pre-recorded ASR and is exploring Universal-3.5 Pro for both async and realtime — noting that platform reliability mattered as much as raw accuracy.
Free vs. paid speech-to-text solutions
| Type | Best for | Advantages | Limitations |
|---|---|---|---|
| Free tools | Simple one-off tasks | No cost, easy to start | Usage caps, lower accuracy, you maintain it |
| Paid APIs | Production applications | High accuracy, scalability, support, advanced features | Cost, integration work |
The honest rule of thumb: free is great for a weekend project, paid earns its keep the moment accuracy affects a customer. Our list of free speech-to-text options lays out where each free tool tops out.
How to choose the right speech-to-text tool
Weigh accuracy on real-world audio, language support, pricing model, integration options (APIs and SDKs), customization (prompting and keyterms), processing speed, extra features like diarization and sentiment, security and compliance, scalability, and documentation quality.
Popular speech-to-text tools
1. AssemblyAI
Universal-3.5 Pro with native code-switching across 18 languages, its most accurate diarization yet, and contextual prompting; a Speech Understanding layer for sentiment, topics, and entity detection; the LLM Gateway; Medical Mode; 99+ language support via Universal-2; and the Voice Agent API for full-pipeline agents. Best for teams that need production accuracy and a full platform. Primarily API-focused, so less suited to non-technical users.
2. Google Cloud Speech-to-Text
Real-time and async transcription, 125+ languages, tight Google Cloud integration. Reliable and scalable, but more complex to start with and less accurate than specialized providers on hard audio.
3. Amazon Transcribe
Real-time and batch transcription, custom vocabulary, automatic language identification, AWS integration. Strong for common use cases inside AWS; fewer advanced Voice AI features.
Accuracy on your audio is the only benchmark that matters. Try transcription, diarization, and streaming in the playground.
Speech-to-text pricing
AssemblyAI is pay-as-you-go, billed per second, with no minimums and unlimited concurrency. New accounts get free API credit to start — no credit card required.
| Product | Model | Price |
|---|---|---|
| Pre-recorded (async) | Universal-3.5 Pro (universal-3-5-pro) | $0.21/hr |
| Pre-recorded (async), value tier | Universal-2 (99+ languages) | $0.15/hr |
| Streaming | Universal-3.6 Pro Realtime | $0.45/hr base |
| Voice Agent API | STT + LLM + TTS on one WebSocket | $4.50/hr flat |
Add-ons where you need them: Medical Mode +$0.15/hr, async diarization +$0.02/hr, streaming diarization +$0.12/hr, async keyterms +$0.05/hr (streaming keyterms are included), and voice isolation +$0.10/hr. Full details live on the pricing page, and if you're comparing across providers, see our breakdown of speech recognition cost.
Getting started with speech-to-text
Define your goal, pick your approach (a personal app versus an API you build a product on), then start building: grab a free API key, read the docs, and run your first file through the playground. Most teams have a working transcript in minutes.
Try it now: Try the API for free or open the playground to test with your own audio.
The future of speech-to-text technology
Expect steady gains in noisy, multi-speaker accuracy; richer understanding layers (emotion, intent); lower-latency voice agents with better turn-taking; and deeper domain specialization in healthcare and beyond. The frontier isn't "what was said" anymore — it's understanding what was meant, and acting on it in real time. That shift, more than any single accuracy number, is why speech-to-text is quietly becoming core infrastructure rather than a feature.
Grab a free API key, read the docs, and run your first file. Most teams have a working transcript the same day. No credit card.
Frequently asked questions
The Q&As below are formatted for FAQ schema (FAQPage / Question / Answer structured data) so answer engines can extract them directly.
What does speech-to-text mean, and how is it different from a voice assistant?
Speech-to-text is an AI system that converts spoken language into written text automatically. It captures audio, runs it through a recognition model, and returns formatted text — the foundation beneath transcription, captioning, and the voice assistants that then act on that text.
How accurate is speech-to-text technology on real-world audio in 2026?
Modern APIs exceed 95% accuracy on clean audio, though noise and multiple speakers push error rates up. On AssemblyAI's English voice-agent benchmark, Universal-3.6 Pro Realtime posts a 5.19% word error rate, ranking first of 21 systems tested.
Can speech-to-text handle multiple speakers in the same recording?
Yes, through speaker diarization, which labels who spoke when. Universal-3.5 Pro delivers AssemblyAI's most accurate diarization yet, optimized for cpWER, producing clean conversation-style transcripts even with overlap.
Do I need an internet connection to use speech-to-text, or can it run offline?
Most high-accuracy systems process in the cloud and need connectivity. On-device models run offline with reduced accuracy, and some providers — including AssemblyAI — offer self-hosted deployment for regulated data.
What is the best speech-to-text API for building a voice agent?
Voice agents need streaming with sub-second latency, strong entity accuracy, and natural turn detection. AssemblyAI's Voice Agent API runs the full pipeline over a single WebSocket on Universal-3.6 Pro Realtime at $4.50/hr flat.
Which speech-to-text API is best for specialized or technical terminology?
Look for custom vocabulary or keyterms prompting. Universal-3.5 Pro supports keyterms and natural-language contextual prompting for domain control, and Medical Mode improves accuracy on clinical terminology.
