How to choose the best speech-to-text API for voice agents
Choose the right speech-to-text API for voice agents. Learn the latency, accuracy, and integration requirements that actually matter for real conversations.
Standard speech-to-text benchmarks don't predict real-world performance. While metrics like Word Error Rate give you a baseline, they miss what actually matters in production—accuracy on the short, entity-dense utterances a voice agent lives on, latency under load, and how the model behaves when the network gets ugly. A recent builder survey found that 76% of participants consider speech-to-text accuracy a non-negotiable requirement, right alongside latency under load and behavior on the critical tokens your application depends on.
This guide covers how to choose a streaming speech-to-text API for voice agents: the evaluation criteria that actually matter, a decision matrix you can score providers against, how the leading real-time APIs compare on accuracy, and what implementation looks like from proof of concept to scale. For a broader introduction, see our complete guide to AI voice agents.
How to choose the best STT API for voice agents (the short version)
To choose the best speech-to-text API for voice agents, score providers on six criteria: accuracy on short utterances and entities, latency and end-of-turn detection, language and code-switching coverage, streaming reliability and network resilience, pricing model, and diarization. Voice agents fail on entities and turn-taking long before they fail on average WER, so weight those first. On our internal ranking, Universal-3.5 Pro Realtime leads: 6.99% pooled word error rate on real agent conversations, end-of-turn detection landing around 300ms, 18 languages with mid-sentence code-switching, and context features that cut WER by 10.2% on voice-agent audio.
Decision matrix: what to score and why
What is a real-time speech-to-text API?
A real-time speech-to-text API converts spoken audio into text as it's captured—streaming transcription results back to your application in milliseconds rather than after a recording finishes. Unlike batch APIs that process pre-recorded files, real-time APIs maintain a persistent connection (typically a WebSocket) to the audio source, enabling live captioning, voice commands, and conversational AI applications where every millisecond of delay affects the user experience.
Real-time vs. batch: the core distinction
There are two primary types of speech-to-text APIs:
- Batch APIs: Process pre-recorded audio files and return complete transcripts after processing. Ideal for podcasts, video files, and recorded meetings.
- Streaming APIs: Process live audio in real time, essential for voice commands, live captioning, and conversational AI agents.
Streaming APIs make decisions with limited context, while batch APIs use entire files for better accuracy. As this guide explains, batch processing can see the full context of a recording, which often leads to the highest possible accuracy. This affects both pricing and integration complexity.
How streaming transcription works
Real-time speech-to-text relies on a persistent connection—typically a WebSocket—to stream audio data to the API as it's spoken. The API processes these audio chunks instantly and returns a stream of transcripts.
During this process, the API sends partial transcripts—words it thinks it heard based on the current audio—and then finalizes them into immutable transcripts once it has enough context. For interactive applications like voice agents, the system must also handle intelligent endpointing to determine when a user has finished speaking, rather than just pausing to think.
The key technical considerations for streaming include:
- Connection management: Maintaining stable WebSocket connections and handling reconnection gracefully
- Audio format handling: Supporting various sample rates, bit depths, and encoding formats
- Partial vs. final results: Understanding when transcripts are tentative versus committed
- Latency optimization: Minimizing the delay between speech and transcript availability
Key features and capabilities to evaluate
Key features determine speech-to-text API performance for your specific use case. Focus on accuracy, latency, language support, and advanced processing capabilities rather than marketing claims.
Core transcription features
- Accuracy: The most fundamental requirement. A survey of builders found that 76% consider speech-to-text accuracy a non-negotiable requirement for voice agents. Look for benchmarks on your specific use case—medical transcription accuracy differs vastly from casual conversation accuracy.
- Speed and latency: How quickly does the API return a transcript? For real-time applications, low latency is non-negotiable. Batch processing speed affects user wait times and system throughput.
- Language support: Does the API support the languages, dialects, and accents of your user base? Some APIs excel at American English but struggle with international accents and mid-sentence code-switching.
Advanced processing capabilities
- Speaker diarization: Can the model distinguish between multiple speakers and label who said what? Essential for meeting transcription and call analytics.
- Automatic punctuation and casing: Does the transcript include proper punctuation and capitalization for readability? This dramatically affects transcript usability.
- Number formatting: How does the API handle spoken numbers? Consistent formatting matters for addresses, phone numbers, and financial data.
Customization and intelligence features
- Keyterm prompting: Can you provide a list of domain-specific jargon, unique names, or product terms to improve their recognition accuracy? This is also known as word boosting.
- Context features: Can the model carry conversation history and take the agent's own question as a hint? Universal-3.5 Pro Realtime does both—rolling conversation memory is on by default, and agent_context lets you pass the agent's question so the model hears the reply through that lens.
- Entity detection: Does the API automatically identify important information like dates, locations, or person names? This enables downstream processing without additional NLP steps.
- Sentiment analysis: Can the system detect emotional tone in speech? This is valuable for customer service and sales applications.
What makes real-time speech-to-text different for voice agents
Voice agent speech-to-text requires sub-300ms latency, intelligent endpointing, and real-time processing—capabilities that standard transcription APIs lack. Unlike batch transcription, where speed is a convenience, voice agents need instant responses to maintain conversational flow. Human conversation studies show the typical response time in dialogue is around 200ms.
The requirements extend beyond speed. Voice agents must handle the messiness of natural conversation—interruptions, corrections, thinking pauses, and overlapping speech. A transcription API designed for recorded podcasts won't capture the dynamic nature of live interaction.
Key technical differences include:
- Real-time processing: Immediate transcription without buffering delays. The system must balance speed with accuracy, making decisions with limited future context.
- Intelligent endpointing: Understanding conversational pauses vs. completion. The system must distinguish between someone pausing to think and finishing their turn.
- Critical token accuracy: Perfect capture of business-critical information like emails and phone numbers. Errors on these tokens directly impact the user experience.
- Context carryover: The model should remember what's already been said and use it to disambiguate the current turn, not treat every utterance as if it arrived cold.
- Immutable transcripts: No revision cycles that force agents to backtrack. Once words are spoken and processed, they shouldn't change.
The choice of API directly impacts whether your voice agent feels helpful and human or robotic and frustrating. Users judge voice agents within seconds—slow responses, misunderstood commands, or awkward interruptions immediately erode trust. A survey of builders found that 95% of respondents have been frustrated with voice agents at some point.
Accuracy that matters: entities and short utterances
Generic word error rates tell you almost nothing about voice agent performance. What matters is accuracy on the specific information your agent has to capture and act on—and those tokens usually show up in short, high-stakes turns. When your voice agent mishears "john.smith@company.com" as "johnsmith@company.calm," you've lost a customer.
This is exactly where Universal-3.5 Pro Realtime is built to win. It adds two context features that target agent audio directly:
- Rolling conversation memory is on by default—the model keeps track of what's already been said, so the current turn is transcribed with the conversation as context.
- agent_context lets you pass the agent's question into the model. Across 20,000 real voice-agent files, passing agent context cut WER by 10.2%—with short-utterance errors down 13.7%, place entities down 15.5%, name entities down 9.4%, and fabrications down 18.3%. One team paired agent_context with prompting and watched their utterance error rate drop from 26% to 9% on production audio.
On top of context, voice_focus isolates the primary speaker (near-field for headsets and phones, far-field for rooms, kiosks, and drive-thrus), so background noise stops corrupting the tokens you care about. The practical takeaway: demand accuracy on business-critical tokens in your specific industry context, and test with audio that includes the short, entity-heavy turns your agent will actually field.
Latency and network resilience: demand sub-300ms and graceful failure
Humans respond within about 200ms in natural conversation—a finding supported by cross-linguistic research that found median response gaps between 0ms and 300ms—so anything over 300ms feels robotic and breaks conversational flow. Research on conversational dynamics shows that faster response times directly correlate with feelings of enjoyment and social connection between speakers. This is about end-to-end latency from speech input to actionable transcript, not just raw processing time.
The red flag is any API that quotes "processing time" without addressing end-to-end latency, or that revises transcripts after delivery. When a speech-to-text API rewrites what it already sent you, your agent has to backtrack mid-conversation. Universal-3.5 Pro Realtime returns immutable transcripts and its end-of-turn detection reads tonality, pacing, and rhythm to land around 300ms—so the agent knows the difference between a caller pausing to think and a caller finishing their turn. According to conversational analysis, nearly a quarter of speech segments are self-continuations after a pause, so this distinction is the difference between an agent that waits its turn and one that talks over people.
Network resilience matters just as much, and it's the part most demos skip. Real calls run over cellular and flaky Wi-Fi—packets drop, connections stall, sockets die. Test how a provider handles reconnection, whether it offers session resumption so a dropped connection doesn't restart the whole conversation, and what happens under packet loss. A model that's fast on a clean office network but falls over on a subway call isn't ready for production. Look for stable WebSocket handling, graceful reconnection, and session resumption baked into the API rather than left to your team.
Intelligent endpointing: move beyond basic silence detection
Basic Voice Activity Detection treats every pause like a conversation ending—a flawed approach. Picture someone saying "My email is... john.smith@company.com" with natural hesitation, and your agent cutting in with "How can I help you?" before they finish. Look for endpointing that combines configurable silence thresholds with model confidence, going beyond basic VAD to reduce false turn-endings. Basic VAD fires on any pause regardless of context; a smarter system waits until the model is confident the utterance is complete before closing the turn.
Test this immediately with natural speech patterns. Have someone provide information with realistic hesitation, interruptions, and clarifications. Learn more about these common voice agent challenges and how modern solutions address them.
Languages and code-switching: real callers don't stay in one language
If your users speak more than one language—or one language with heavy borrowing, like Hinglish or Spanglish—per-utterance language detection isn't enough. You need a model that handles switches inside a single sentence. Universal-3.5 Pro Realtime supports 18 languages (up from 6 on the previous generation): English, Spanish, French, German, Italian, Portuguese, Arabic, Danish, Dutch, Hebrew, Hindi, Japanese, Mandarin, Vietnamese, Finnish, Norwegian, Swedish, and Turkish, with mid-sentence code-switching built in. If you need to lock the model to a single language, the language_code parameter steers it there.
Top real-time speech-to-text API providers compared
The speech-to-text API landscape includes providers with different strengths, architectures, and ideal use cases. Understanding these differences helps you match capabilities to your specific requirements.
Voice agent-optimized providers
- AssemblyAI: Offers Universal-3.5 Pro Realtime, purpose-built for voice agents with context carryover, agent_context, intelligent end-of-turn detection, and market-leading accuracy on real agent conversations. The Voice Agent API provides a single WebSocket that replaces separate STT, LLM, and TTS providers, with flat-rate pricing ($4.50/hr).
- Deepgram: Speed-focused, with Flux positioned for conversational speech recognition; leans on turn detection.
- ElevenLabs: Strong on voice output; Scribe v2 covers streaming transcription with concurrency caps.
- Google Cloud Speech-to-Text: Broad language support with Chirp models; requires configuration tuning for voice agent use.
Realtime accuracy benchmark: how the models compare on agent conversations
We measured word error rate and entity error rate on real agent conversations using the open Pipecat STT benchmark. Lower is better on every metric. Universal-3.5 Pro Realtime leads on WER and on the entity categories agents care about most.
The gap on entity error rate is the story for voice agents. A model can look competitive on average WER and still be three times worse at getting a caller's name or phone number right—and those are precisely the tokens an agent has to act on. See the full benchmarks for methodology and complete data.
How the providers stack up for voice agents
Integration and implementation considerations
Technical implementation determines project success more than underlying model quality. Three areas require careful evaluation: orchestration framework compatibility, API design quality, and scaling considerations.
Orchestration framework compatibility
Custom WebSocket implementations often cost more in developer time than anticipated, which is why industry survey data shows a hybrid approach—combining vendor infrastructure with custom logic—is the most popular build strategy among teams. A recent industry report found that 45% of teams building voice agents cite integration difficulty as a top challenge that extends timelines and inflates costs. The initial connection setup is straightforward, but handling connection drops, managing state, and implementing proper error recovery quickly gets complex.
Pre-built integrations cut development time from weeks to days. AssemblyAI provides step-by-step documentation for major orchestration frameworks like LiveKit and Pipecat, with battle-tested code that handles edge cases your team hasn't hit yet. Consider framework compatibility early in your selection process.
API design quality: evaluate the developer experience
The quality of the developer experience directly impacts your implementation timeline and long-term maintenance costs. Well-designed APIs make complex tasks simple; poor APIs create ongoing frustration.
Green flags for good API design:
- Comprehensive error handling with clear error messages
- Consistent response formats across endpoints
- Robust SDKs in multiple programming languages
- Clear connection state management for streaming
- Graceful degradation when network conditions change
Red flags that indicate poor developer experience:
- Sparse or outdated documentation
- Limited SDK support forcing raw API calls
- Unclear pricing for production loads
- Complex authentication mechanisms
- Inconsistent behavior across different endpoints
For detailed technical guidance, review our streaming documentation.
Scaling considerations: plan for success scenarios
Production deployments expose limitations that aren't apparent during prototyping. Understanding scaling constraints prevents painful migrations later.
Verify actual concurrent connection limits, not marketing claims. Some providers throttle connections aggressively once you exceed free-tier limits, causing production failures during peak usage. Ask specific questions about concurrent WebSocket connections and what happens when you exceed them—AssemblyAI's streaming model runs with unlimited concurrency and no rate limits, so you're not renegotiating capacity every time you grow.
Geographic distribution matters for latency. Ensure low latency for your user base's locations, not just major US markets. A voice agent with 150ms latency in San Francisco but 800ms in Singapore will fail international expansion.
Cost scaling requires careful analysis. Session-based per-hour pricing offers more predictable costs than complex per-minute or per-token models with hidden fees. For implementation best practices, check our guide to getting started with real-time streaming transcription.
A real-time streaming integration in practice
The best real-time APIs act as invisible infrastructure, letting you focus on your product rather than voice plumbing. With AssemblyAI's Voice Agent API, you replace separate STT, LLM, and TTS providers with a single WebSocket connection that handles the entire conversational pipeline.
Here's a Python snippet showing how to connect and configure a voice agent session. The client sends a session.update message to define the agent's behavior—system prompt, voice, and available tools—which must follow a specific JSON schema.
import asyncio
import websockets
import json
import base64
# Define the WebSocket endpoint and your API key
API_KEY = "YOUR_ASSEMBLYAI_API_KEY"
URL = "wss://agents.assemblyai.com/v1/voice"
HEADERS = {"Authorization": f"Bearer {API_KEY}"}
# Define the tools the agent can use
TOOLS = [
{
"type": "function",
"name": "lookup_account",
"description": "Looks up account details by account number.",
"parameters": {
"type": "object",
"properties": {
"account_number": {
"type": "string",
"description": "The account number to look up."
}
},
"required": ["account_number"],
},
}
]
async def run():
async with websockets.connect(URL, extra_headers=HEADERS) as ws:
# Send session configuration
await ws.send(json.dumps({
"type": "session.update",
"session": {
"system_prompt": "You are a helpful customer support agent.",
"output": {"voice": "dawn"},
"tools": TOOLS,
}
}))
# Wait for the session.ready event before sending audio
ready_event = await ws.recv()
if json.loads(ready_event).get("type") == "session.ready":
print("Session is ready. Start sending audio.")
# Placeholder for audio streaming logic
# e.g., stream audio from a microphone
# for chunk in microphone_stream:
# await ws.send(json.dumps({
# "type": "input.audio",
# "audio": base64.b64encode(chunk).decode()
# }))
# Main event loop to process messages from the server
async for message in ws:
event = json.loads(message)
# Handle events like transcript.user, reply.audio, tool.call, etc.
print(event)
# To run this example:
# asyncio.run(run())
Because the API is built on Universal-3.5 Pro Realtime, you get market-leading speech accuracy, context carryover, intelligent turn detection, and interruption handling out of the box. The API is accessed via WebSocket events, so system prompt, tools, and conversation settings can all be updated mid-conversation without reconnecting. Most developers get a working agent running the same afternoon they start.
Pricing models and cost considerations
The price tag on an API is only one part of the total cost equation. Understanding different pricing models and hidden costs helps you budget accurately and avoid surprises at scale.
Common pricing models
- Per-minute/hour pricing: You pay for the amount of audio processed. Simple to understand and predict.
- Per-request or per-token pricing: Charges per call or per token regardless of audio length. Can be cost-effective for short utterances but unpredictable at scale.
- Tiered pricing: Volume discounts at certain usage thresholds. Beneficial for high-volume applications but usually requires commitment.
- Subscription models: Fixed monthly cost for a usage allowance. Predictable, but may include overage charges.
Most providers charge extra for advanced features. Diarization, custom vocabulary, entity detection, and real-time streaming often come with add-on fees that can significantly change your total cost at scale, and some charge separately for concurrency.
AssemblyAI takes a simpler approach. The streaming model is $0.45/hr base with unlimited concurrency and no rate limits, and the Voice Agent API is a flat $4.50/hr that covers speech understanding, LLM reasoning, and voice generation—no token math, no separate input/output charges. For current rates, see pricing.
Hidden and indirect costs
- Integration and development time: A poorly documented or complex API can cost weeks of engineering effort. Developer time often exceeds API usage fees early on.
- Maintenance overhead: Frequent API changes, poor reliability, or complex error handling create ongoing costs.
- Infrastructure requirements: Some solutions require extra infrastructure for audio preprocessing, result storage, or connection management.
- The cost of inaccuracy: As recent research shows, accuracy failures correlate directly with user frustration—leading to missed sales, compliance failures, or poor customer experiences that cost far more than the API itself.
Getting started with real-time speech-to-text
Moving from evaluation to implementation requires a structured approach.
Start with a focused proof of concept
Don't rely on generic demos or marketing materials. Build a proof of concept using your actual use case to validate both technical capability and business value. Your proof of concept should:
- Use real audio from your application domain
- Test with your actual latency requirements
- Include your critical business vocabulary
- Measure accuracy on your specific metrics—especially entities
- Evaluate the complete integration experience
Start small with one focused flow. Voice agents should begin with a single conversation path; meeting transcription should start with one team's calls. Our step-by-step voice agent tutorials walk through practical examples and best practices.
Prioritize based on constraints
- Timeline constraints: If you need to launch in 8 weeks, choose the solution with the best existing integrations and support, even if another option might be technically superior with more development time.
- Budget constraints: Consider total cost including development time, not just API pricing. A more expensive API with better documentation might be cheaper overall.
- Technical constraints: Your existing stack influences your options. If you're deeply invested in AWS, Amazon Transcribe might integrate more smoothly despite limitations.
- Compliance constraints: Healthcare and financial applications carry specific requirements. AssemblyAI is considered a business associate under HIPAA and offers a Business Associate Addendum (BAA) that's required under HIPAA to ensure AAI appropriately safeguards protected health information.
Plan for monitoring and optimization
Production deployment is the beginning, not the end. Essential monitoring includes:
- Accuracy metrics: Track WER and critical token accuracy over time
- Latency monitoring: Measure end-to-end response times, not just API latency
- Error rates: Monitor failed requests, timeouts, and retries
- User feedback: Collect qualitative feedback on transcription quality
- Cost tracking: Monitor usage patterns and cost per user or transaction
Build feedback loops into your application. When users correct transcriptions, capture those corrections to identify systematic errors. If certain audio conditions consistently cause problems, use voice_focus or contextual prompting before you reach for a different model.
Implementation checklist
- Latency: End-to-end response time meets requirements
- Accuracy: Acceptable performance on business-critical tokens and entities
- Reliability: Proper error handling, reconnection, and session resumption implemented
- Scalability: Tested at expected peak load and concurrency
- Monitoring: Metrics and alerting in place
- Compliance: Security and regulatory requirements met
- Documentation: Integration documented for team knowledge transfer
The gap between a voice agent that feels natural and one that feels broken usually comes down to two things the average WER never shows you: whether the model gets the entities right, and whether it knows when to talk. Score your shortlist on those first, test on your own audio, and the rest of the decision gets a lot easier.
Frequently asked questions about choosing an STT API for voice agents
How do I choose the best STT API for voice agents?
Score providers on six criteria: accuracy on short utterances and entities, latency and end-of-turn detection, language and code-switching coverage, streaming reliability and network resilience, pricing model, and diarization—then test your top two or three on your own audio. Weight entity accuracy and turn-taking most heavily, because that's where agents fail first. On real agent conversations, Universal-3.5 Pro Realtime leads with a 6.99% pooled word error rate and the lowest entity error rate of the major providers.
What accuracy matters most for voice agents?
Entity accuracy on short utterances matters most—names, phone numbers, emails, and account IDs—not average word error rate. A model can post a competitive overall WER and still be three times worse at capturing a phone number, which is exactly the token an agent has to act on. Test with audio that includes your real, entity-dense turns. Universal-3.5 Pro Realtime posts a 15.31% entity error rate on the agent benchmark versus 50.50% for Deepgram Flux and 39.70% for ElevenLabs Scribe v2, and its agent_context feature cut WER by 10.2% across 20,000 voice-agent files.
How low does STT latency need to be for voice agents?
Aim for sub-300ms end-to-end latency, from speech input to actionable transcript. Humans reply in about 200ms in natural conversation, so anything slower than 300ms feels robotic and causes users to talk over the agent. Universal-3.5 Pro Realtime returns immutable transcripts with end-of-turn detection that reads tonality and pacing to land around 300ms—and check network resilience too, since real calls drop packets and stall connections.
Which STT handles code-switching and multilingual calls?
You need a model with native mid-sentence code-switching, not per-utterance language detection, because real callers switch languages inside a single sentence. Universal-3.5 Pro Realtime supports 18 languages with mid-sentence code-switching built in (Hinglish, Spanglish, and more), and you can lock it to one language with the language_code parameter when you need to.
Should I use streaming or batch STT for voice agents?
Use streaming for voice agents—batch APIs return complete transcripts only after processing finishes, which breaks conversational flow. Streaming APIs process live audio over a persistent WebSocket and return transcripts in milliseconds, which is what a real-time agent needs. Reserve batch for pre-recorded work like podcasts, recorded meetings, and video where accuracy over full context matters more than speed.
What does STT for voice agents cost?
Pricing usually runs per hour or per minute of audio, often with add-on fees for diarization, custom vocabulary, or concurrency. AssemblyAI's streaming model is $0.45/hr base with unlimited concurrency, and the Voice Agent API is a flat $4.50/hr covering speech understanding, LLM reasoning, and voice generation—no token math or separate invoices. Watch for per-token pricing and concurrency commitments that make costs unpredictable at scale; see pricing for current rates.
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Related posts
