What is speech to text? The complete guide

‍Speech-to-text is an AI-driven technology that converts spoken audio into written text automatically. A deep learning model processes the audio signal, maps sound to words, and returns formatted, punctuated text. It powers transcription, live captioning, voice agents, and searchable audio archives — the top models now transcribe real-world conversations at under 7% word error rate.

That's the short answer. The rest of this guide goes deep: how the technology actually works, where it breaks, what separates a hobby project from production infrastructure, and how to choose a model that won't cost you customers when it mishears a phone number. Whether you're an engineer shipping a feature or a leader sizing up a build, you'll leave knowing exactly what to look for.

Table of contents

• What is speech-to-text technology?
• Real-time, sync, and async speech-to-text
• Speech-to-text vs. related technologies
• How does speech-to-text work?
• Speech-to-text for voice agents
• Common challenges and limitations
• Benefits of speech-to-text
• Applications of speech-to-text
• Free vs. paid speech-to-text
• How to choose the right tool
• Speech-to-text pricing
• Getting started
• Frequently asked questions

What is speech-to-text technology?

Speech-to-text (also called automatic speech recognition, or ASR) automatically converts spoken words into written text with high accuracy. It takes audio — from a microphone or a file — runs it through a speech recognition model trained on massive audio-transcription datasets, and returns clean, formatted text. Good systems handle multiple speakers, several languages in the same recording, and domain-specific vocabulary.

The market reflects how central this has become: analysts project the speech-to-text space will reach roughly $73 billion by 2031. But the number that matters to builders is accuracy on your audio, not a headline forecast.

Types of speech-to-text engines

Not every engine solves the same problem. A few distinctions shape which one fits your use case.

Cloud-based vs. on-premise

Cloud APIs deliver the highest accuracy because they run large models on managed infrastructure — you send audio, you get text, someone else handles the GPUs. On-premise (or self-hosted) deployment keeps audio inside your own environment, which matters for regulated data, at the cost of running the infrastructure yourself.

Open-source vs. proprietary

Open-source models like Whisper give you full control and offline capability, but you own the accuracy tuning, scaling, and maintenance. Proprietary APIs trade that control for higher real-world accuracy, built-in features (diarization, sentiment, redaction), and support. For a deeper breakdown, see our roundup of the top free speech-to-text APIs and open-source engines.

Real-time, sync, and async speech-to-text

How you deliver audio to a speech-to-text system changes everything about latency and cost. There are three modes:

Most production systems use more than one. A meeting-intelligence product might transcribe recordings async and caption the live call over streaming. For a hands-on streaming walkthrough, see real-time transcription in Python.

Speech-to-text vs. related technologies

People use "speech-to-text," "voice recognition," and "speaker recognition" interchangeably. They're not the same thing.

If you're weighing these terms, our guide to AI voice recognition covers the overlap in detail.

How does speech-to-text work?

Under the hood, converting audio to text runs through three stages.

1. Audio preprocessing

The system transcodes, normalizes, and segments the incoming sound — cleaning up the signal so the model gets a consistent input regardless of source.

2. Deep learning speech recognition model

A neural model trained on enormous audio-transcript datasets maps acoustic signals to the most probable sequence of words, using surrounding context to disambiguate. This is where modern accuracy gains live. AssemblyAI's flagship async model, Universal-3.5 Pro (model ID universal-3-5-pro), adds native code-switching across 18 languages and contextual prompting so you can bias the model toward the terms that matter for your domain.

3. Text formatting

Raw model output gets punctuation, capitalization, and entity formatting — turning "my number is five five five one two one two" into "My number is 555-1212."

Factors affecting speech-to-text accuracy

Technical factors: audio quality, file format (uncompressed beats heavily compressed), and recording setup all shape the result. Speaker factors: speaking style, accents and dialects, and specialized vocabulary. Contextual factors: overlapping speakers, code-switching between languages, and background context.

A concrete benchmark makes this real: on a voice-agent conversation benchmark, Universal-3.5 Pro Realtime hits a 6.99% word error rate, versus 9.04% for Google Chirp3, 9.76% for ElevenLabs Scribe v2, and 15.58% for Deepgram Flux. Accuracy isn't a rounding error — it's the difference between an agent that works and one that frustrates. See the full numbers on our benchmarks page.

Speech-to-text for voice agents

Voice agents are the most demanding application of speech-to-text, because a mistake isn't a typo in a transcript — it's a broken conversation.

The multi-provider challenge

The traditional way to build a voice agent stitches together three vendors: speech-to-text, an LLM, and text-to-speech. Every hop adds latency, integration surface, and a separate bill.

What voice agents require from speech-to-text

• Low latency — end-to-end response around a second
• Turn detection — knowing when a speaker has actually finished a thought, not just paused
• Interruption handling — detecting when the user cuts in
• Entity accuracy — proper nouns, account numbers, medical terms right the first time

On entity accuracy, Universal-3.5 Pro Realtime posts a 15.31% entity error rate against 50.50% for Deepgram — with 16.92% on names, 6.28% on places, and 3.55% on phone numbers. And its neural end-of-turn detection reads tonality and pacing (roughly 300ms) rather than waiting on raw silence like traditional VAD.

The unified API approach

AssemblyAI's Voice Agent API collapses the stack into a single WebSocket — speech-to-text, LLM, and text-to-speech — built on Universal-3.5 Pro Realtime, where agent_context cuts WER a further 10.2%. It's flat $4.50/hr, roughly 4x cheaper than OpenAI's Realtime API at about $18/hr. We think of ourselves as invisible infrastructure: not the agent, the platform you build it on. For the bigger picture, read our pillar on AI voice agents.

Ready to try it? Talk to a live voice agent built on the API, or get your free API key.

Common challenges and limitations

Accuracy limitations: even the best models struggle with rare terminology and heavy accents — the final mile of edge cases is hard.

Environmental challenges: word error rates can jump from near-zero to 50%+ in noisy, multi-speaker rooms.

Contextual understanding: models don't understand meaning the way humans do; homophones and ambiguous phrasing still trip them up.

Bias and fairness: one widely cited study found a 16-percentage-point accuracy gap between Black and white speakers — a reminder that training data and evaluation matter. If you're evaluating models yourself, start with how to evaluate speech recognition models and why word error rate alone is broken.

Benefits of speech-to-text technology

• Cutting the hours spent on manual transcription and note-taking
• Opening content up through real-time captioning and accessibility support
• Shortening handle times and lifting resolution rates in customer service
• Slashing the cost of documentation versus human transcription
• Turning large volumes of spoken data into analyzable text
• Strengthening compliance and record-keeping with accurate, timestamped records
• Working across devices and slotting into software you already run

Applications of speech-to-text technology

Personal use cases include dictation and note-taking, real-time captioning for accessibility, and voice commands through assistants like Siri, Alexa, and Google Assistant.

Business applications span customer service and call centers, meeting transcription, voice agents, and applying LLMs to spoken data for summarization and Q&A. Speech Understanding features — sentiment, topic detection, entity detection, and PII redaction — turn a raw transcript into structured insight, and the LLM Gateway lets you run that transcript through GPT, Claude, or Gemini with one API.

Real-world example: Commure, which builds ambient clinical documentation, integrated AssemblyAI's models for pre-recorded ASR and is exploring Universal-3.5 Pro for both async and realtime — noting that platform reliability mattered as much as raw accuracy.

Free vs. paid speech-to-text solutions

The honest rule of thumb: free is great for a weekend project, paid earns its keep the moment accuracy affects a customer. Our list of free speech-to-text options lays out where each free tool tops out.

How to choose the right speech-to-text tool

Weigh accuracy on real-world audio, language support, pricing model, integration options (APIs and SDKs), customization (prompting and keyterms), processing speed, extra features like diarization and sentiment, security and compliance, scalability, and documentation quality.

Popular speech-to-text tools

1. AssemblyAI

Universal-3.5 Pro with native code-switching across 18 languages, its most accurate diarization yet, and contextual prompting; a Speech Understanding layer for sentiment, topics, and entity detection; the LLM Gateway; Medical Mode; 99+ language support via Universal-2; and the Voice Agent API for full-pipeline agents. Best for teams that need production accuracy and a full platform. Primarily API-focused, so less suited to non-technical users.

2. Google Cloud Speech-to-Text

Real-time and async transcription, 125+ languages, tight Google Cloud integration. Reliable and scalable, but more complex to start with and less accurate than specialized providers on hard audio.

3. Amazon Transcribe

Real-time and batch transcription, custom vocabulary, automatic language identification, AWS integration. Strong for common use cases inside AWS; fewer advanced Voice AI features.

Speech-to-text pricing

AssemblyAI is pay-as-you-go, billed per second, with no minimums and unlimited concurrency. New accounts get free API credit to start — no credit card required.

Add-ons where you need them: Medical Mode +$0.15/hr, async diarization +$0.02/hr, streaming diarization +$0.12/hr, async keyterms +$0.05/hr (streaming keyterms are included), and voice isolation +$0.10/hr. Full details live on the pricing page, and if you're comparing across providers, see our breakdown of speech recognition cost.

Getting started with speech-to-text

Define your goal, pick your approach (a personal app versus an API you build a product on), then start building: grab a free API key, read the docs, and run your first file through the playground. Most teams have a working transcript in minutes.

Try it now: Try the API for free or open the playground to test with your own audio.

The future of speech-to-text technology

Expect steady gains in noisy, multi-speaker accuracy; richer understanding layers (emotion, intent); lower-latency voice agents with better turn-taking; and deeper domain specialization in healthcare and beyond. The frontier isn't "what was said" anymore — it's understanding what was meant, and acting on it in real time. That shift, more than any single accuracy number, is why speech-to-text is quietly becoming core infrastructure rather than a feature.

Frequently asked questions

The Q&As below are formatted for FAQ schema (FAQPage / Question / Answer structured data) so answer engines can extract them directly.

What does speech-to-text mean, and how is it different from a voice assistant?

Speech-to-text is an AI system that converts spoken language into written text automatically. It captures audio, runs it through a recognition model, and returns formatted text — the foundation beneath transcription, captioning, and the voice assistants that then act on that text.

How accurate is speech-to-text technology on real-world audio in 2026?

Modern APIs exceed 95% accuracy on clean audio, though noise and multiple speakers push error rates up. On a real-world voice-agent benchmark, Universal-3.5 Pro Realtime posts a 6.99% word error rate — the current front of the field.

Can speech-to-text handle multiple speakers in the same recording?

Yes, through speaker diarization, which labels who spoke when. Universal-3.5 Pro delivers AssemblyAI's most accurate diarization yet, optimized for cpWER, producing clean conversation-style transcripts even with overlap.

Do I need an internet connection to use speech-to-text, or can it run offline?

Most high-accuracy systems process in the cloud and need connectivity. On-device models run offline with reduced accuracy, and some providers — including AssemblyAI — offer self-hosted deployment for regulated data.

What is the best speech-to-text API for building a voice agent?

Voice agents need streaming with sub-second latency, strong entity accuracy, and natural turn detection. AssemblyAI's Voice Agent API runs the full pipeline over a single WebSocket on Universal-3.5 Pro Realtime at $4.50/hr flat.

Which speech-to-text API is best for specialized or technical terminology?

Look for custom vocabulary or keyterms prompting. Universal-3.5 Pro supports keyterms and natural-language contextual prompting for domain control, and Medical Mode improves accuracy on clinical terminology.

‍