Real-time entity extraction from speech: Capturing emails, phone numbers, and addresses in live audio

A word error rate of 5% sounds excellent until the 5% lands on someone's email address.

That's the whole problem with real-time entity extraction. The words that matter most in a live conversation — the account number, the postcode, the phone number, the spelled-out surname — are the words a speech model is least likely to get right, because they carry no linguistic context to fall back on. There's no grammar that tells you the next character after "J-A-R-V" should be "I" rather than "E."

And unlike batch processing, you don't get a second pass. Real-time gives you one shot to get it right.

What is real-time entity extraction from audio?

It's the identification of structured information — emails, phone numbers, addresses, names, dates, organizations, amounts — while someone is still speaking, rather than after the conversation ends.

The distinction from batch matters practically, not just architecturally. If a caller gives you their email address and your system extracts it forty minutes later, you can't read it back to confirm it. You can't validate it against a database while they're still on the line. You can't route the call based on it. Everything that makes the data useful in the moment depends on having it in the moment.

How accurate is entity capture in live audio?

This is the number to evaluate on, and almost nobody publishes it. Aggregate word error rate averages the easy words in with the hard ones, which conceals exactly the failure you care about here.

Entity error rate doesn't. Here's how the current generation of streaming models compares on real agent conversations:

Source: AssemblyAI benchmarks.

Look at the spread between the WER row and the entity row. Deepgram Flux is a bit over twice our word error rate but more than three times our entity error rate — the gap widens precisely where it hurts. Places is the most dramatic column: 6.28% against ElevenLabs Scribe v2's 34.06%, which is the difference between an address field you can auto-fill and one you have to ask for twice.

Phone numbers at 3.55% is the row I'd point a contact center team at first, because phone numbers are where the zero-tolerance rule bites hardest.

How the streaming pipeline works

One model, one pass. Universal-3.5 Pro Realtime transcribes over a WebSocket at wss://streaming.assemblyai.com/v3/ws, and the entity-bearing text arrives already formatted — "at" rendered as @, spoken digits rendered as digits, "dot com" as .com.

Set format_turns=true and you get punctuated, formatted turns rather than raw tokens. There's no separate entity_detection flag on the streaming path the way there is on async; formatting and transcription happen together, which is the point. A two-stage pipeline that transcribes first and extracts second cascades every transcription error into the extraction step.

For richer typed extraction — 50+ entity types with labels — run speech understanding over the completed transcript afterward. Live capture and structured post-processing are complementary, not alternatives.

Capturing emails, phone numbers, and addresses

Three features do most of the work here, and two of them shipped recently.

Tell the model what was asked: agent_context

This is the biggest single lever, and it's almost embarrassingly simple: pass the agent's question along with the audio.

Consider "B as in boy, four, four, at gmail." Out of context, that's a mess of digits and letters. Given the context "What's the best email address to reach you at?", the model knows to expect an email and resolves the fragments accordingly.

How much that's worth is published, and usefully for this post it's broken out by entity type — the next section has the table. Short utterances and fabrications are the two failure modes that ruin entity capture. A one-word confirmation, a spelled-out account ID, a mumbled postcode — these are low-information audio events where a model without context is essentially guessing, and a guessing model doesn't return nothing. It returns something plausible and wrong. Context is what stops the guess.

Alongside it, Context Carryover keeps a short rolling memory of the conversation, on by default and included at no extra cost, so an entity mentioned two turns ago still informs how the current turn is transcribed.

"We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms."

— David Zhao, Co-founder at LiveKit

How much context is worth writing?

More detail keeps paying, and for entity capture specifically it pays far better than the headline WER number suggests. The streaming docs publish a benchmark across 20,000 real voice-agent calls comparing three levels of free-text prompt against sending none at all. Every figure is a relative reduction versus no prompt.

The three levels are short. Domain is two to five words and names the domain only: "Medical consultation call." Scenario is five to fifteen words on what the conversation is actually about: "Cardiology consultation about chest pain symptoms." Detailed is twenty to fifty words naming the people, products, and identifiers in play: "Cardiology consultation between Dr. Smith and an elderly patient regarding recurring chest pain, ECG results, and medication adjustment for hypertension."

Read down the entity rows rather than the top one. Word error rate improves by a fifth at best; name errors nearly halve. That gap is the whole argument of this post restated as a table — context buys you disproportionately more on exactly the tokens you can't afford to lose.

Two takeaways worth building around. Scenario context is the practical default, because it needs only the kind of information your application already has — a call type, a queue name, a form the caller is filling in — and it already cuts WER around 10% and name and place errors 16% and 21%. Detailed context is the upper bound, and it's within reach whenever you know who's calling: a CRM record turned into a sentence takes name errors down 49%. Two results run against intuition and are worth stating plainly: hallucinated and fabricated words go down as you add context rather than up, and turn detection is unaffected. Full figures and prompt-writing guidance are in the streaming prompting and keyterms documentation.

Give it your vocabulary: keyterm prompting

For the terms context can't infer — product SKUs, clinic names, an unusual surname that appears in 40% of your calls — keyterm prompting biases the model toward a list you supply.

On async, keyterm prompting is a +$0.05/hr add-on. On Universal-3.5 Pro Realtime it's included at no additional cost, which changes the calculus entirely — there's no reason not to use it on a streaming agent. Current pricing has the full table.

The limits differ by surface, and mixing them up is the usual cause of a rejected request:

• Async. Up to 1,000 words or phrases, with a maximum of six words per phrase. Treat that as a ceiling rather than a promise — actual capacity may be lower due to internal tokenization, each word in a multi-word phrase counts toward the limit, and capitalization and longer words consume more capacity.
• Streaming. A maximum of 100 keyterms per session, each string 50 characters or shorter. Go over 100 and the request errors; individual terms longer than 50 characters are ignored.

Prompting and keyterms are two levers, not a choice between them. The free-text prompt tells the model what the conversation is; keyterms_prompt tells it which exact strings to expect. On streaming, both can be updated mid-session through an UpdateConfiguration message without reconnecting, so you can load a caller's details the moment you know who they are — and sending an empty array clears them again.

Clean up the input: voice_focus

Entity capture degrades fastest in noise, because there's no linguistic redundancy to recover from. voice_focus isolates the primary speaker and suppresses background speech and noise. Use near-field for headsets and phone handsets, far-field for rooms, kiosks, and drive-thrus. It's a +$0.10/hr realtime add-on, and for a drive-thru or a busy floor it pays for itself in re-asks avoided.

If multiple people are speaking, live speaker diarization attributes each turn, and each Turn message carries a speaker_label, and a single SpeakerRevision correction arrives at the end of the session — useful for deciding whether to trust an entity that was captured during a moment of crosstalk.

Real-time entity extraction use cases

Call center and CRM automation

The workflow is caller ID lookup, context injection, live capture, direct CRM write. Order numbers, ticket references, and updated contact details flow into the system while the call is happening, which eliminates post-call data entry rather than speeding it up.

Feed the customer's known details in as keyterms at session start. If you already know their surname is Nowakowski, the model shouldn't have to discover that from a noisy phone line.

Meeting transcription and action items

Combine entity detection with speaker attribution and you get action items with owners: person, task, deadline. The pattern that trips systems up is indirect reference — someone says "send it to the address I gave you earlier," and the system needs to have retained the earlier entity. Rolling context handles some of this; your application layer should handle the rest by keeping an entity store per session.

Voice agents

This is the demanding case, because the agent has to act on the entity immediately. Four things to build in:

• Confirmation patterns. Read critical entities back. Always, for payment and contact details.
• Interruption handling. Callers correct themselves mid-entity — "it's 44... sorry, 45."
• Entity memory. Track what's been captured so far so the agent doesn't re-ask.
• Error recovery. A graceful "could you spell that?" beats a confident wrong value.

If you're building the whole pipeline, the Voice Agent API runs speech-to-text, the language model, and speech synthesis over a single WebSocket at a flat $4.50/hr, with per-session controls for contextual prompts and voice focus settable mid-conversation. There's a fuller walkthrough in our Voice Agent API introduction, and more on picking a provider in our guide to choosing an STT API for voice agents.

Managing latency in production systems

There's a real tension here. Endpoint too fast and you cut off a caller mid-postcode; endpoint too slow and every exchange feels sluggish. Entity-heavy speech is precisely where people pause — between digit groups, between letters when spelling — so the pause patterns that normally signal "I'm done" signal nothing of the kind.

Your application resolves it. When you know an entity is coming, because your agent just asked for a callback number, raise min_turn_silence mid-stream with an UpdateConfiguration message so those pauses do not end the turn, then re-send your mode preset afterward to restore its defaults. Changes take effect immediately, without reconnecting.

Underneath that, end-of-turn detection weighs whether what has been said so far reads as a completed thought, rather than endpointing on silence alone. That's a meaningfully different mechanism from a silence timer, and it's why "wait, no—" gets recognized as an incomplete turn rather than a finished one.

Three other latency notes. Streaming is billed on session duration — WebSocket open time — not audio duration, so don't hold sessions open idle. The max_accuracy mode is worth the trade in noisy or far-field environments where entity capture is failing. And streaming now accepts AAC via encoding=aac, roughly 6x lighter on network I/O than PCM, which matters on constrained mobile connections.

Validation still isn't optional

Better entity accuracy raises the floor. It doesn't remove the need for a floor.

Three layers, in order:

• The model extracts, primed with agent context and your keyterms.
• Your code validates format — email syntax, phone number length and country prefix, postcode patterns, addresses against a postal or maps API.
• The user confirms before anything is written permanently.

Watch the edge cases that defeat all three: repeated digits, where "1111" and "111" differ by an utterance length no human reliably judges either; single-character utterances; and any entity captured while two people were talking. Flag low-confidence entities for confirmation rather than treating extraction as binary. Model IDs, message shapes, and every parameter named here are in the documentation, with recent additions in the changelog.

What changed, and why it matters

The old advice for entity capture was to throw a keyterm list at the problem and validate hard afterward. That's still good practice, but it treats the model as a fixed thing you work around.

The shift is that the model can now be told what it's listening for — the question that was just asked, the conversation so far, the acoustic environment — and it uses that information the way a human listener does. A person taking down an email address over the phone doesn't hear better than the model. They just know what they asked. Giving the model the same thing is the cheapest accuracy improvement available on this problem, and it costs nothing extra on the realtime path.

More on the broader architecture on our voice agents solutions page.

Frequently asked questions

How do I use APIs to highlight named entities in speech transcripts?

For live audio, stream to the WebSocket endpoint with format_turns=true and the formatted entities arrive inside the turn text as the conversation happens. For typed, labelled entities across 50+ categories — person, organization, location, date, phone number, email, and more — run speech understanding over the completed transcript, which returns each entity with its type and character offsets so you can highlight it in your UI. Many production systems do both: live capture for immediate action, typed extraction for storage and search.

Which speech-to-text provider do production voice agents rely on?

Evaluate on entity error rate rather than aggregate word error rate, since entities are what voice agents act on. On the Pipecat open STT benchmark of real agent conversations, Universal-3.5 Pro Realtime records a 15.31% entity error rate against Deepgram Flux at 50.50%, ElevenLabs Scribe v2 at 39.70%, and Google Chirp3 at 21.51%, with phone numbers at 3.55%. Also weigh turn detection quality and whether contextual features like agent context are included or charged as add-ons.

Is it possible to transcribe multi-speaker conversations automatically?

Yes. Streaming diarization labels speakers live for up to 10 participants, then re-clusters and sends a single correction within roughly half a second of the stream ending. Each turn carries a speaker_label, and a single SpeakerRevision message corrects any changed turns at the end of the session, which is useful for deciding whether to trust an entity captured during crosstalk.

What is AssemblyAI's Voice Agent API and how does it work?

It's a single WebSocket that handles speech-to-text, the language model, and speech synthesis together, at a flat $4.50/hr covering all three. One connection replaces three providers, one bill, one set of logs, with end-to-end response times of roughly 1 to 1.3 seconds. It uses standard JSON with no SDK required, supports live configuration changes mid-conversation without reconnecting, and ships drop-in plugins for LiveKit and Pipecat.

How much does keyterm prompting cost for real-time entity extraction?

On Universal-3.5 Pro Realtime, keyterm prompting is included at no additional cost, along with agent context and rolling conversation memory. On the asynchronous API it's a +$0.05/hr add-on accepting up to 1,000 words or phrases, at a maximum of six words per phrase. Streaming sessions take a different ceiling — 100 keyterms per session, each 50 characters or shorter.

How do I stop a voice agent from cutting callers off while they spell an email address?

Raise min_turn_silence mid-stream with an UpdateConfiguration message as soon as your agent has asked for an email address or an account number, then re-send your mode preset to restore the faster default. This handles the pauses people naturally leave between digit groups or spelled letters without making every other turn feel sluggish. End-of-turn detection also weighs whether what has been said so far reads as a completed thought, rather than endpointing on silence duration alone.