What is speech understanding (audio intelligence)? Definition, features, examples

An audio intelligence tool, now usually called speech understanding, is an API or platform that turns recorded or live speech into structured data: transcripts, speaker labels, sentiment, entities, topics, summaries and PII redaction. Developers use it to build conversation intelligence, meeting notes, compliance monitoring and voice agents on top of what was said, not just the words.

One naming note before anything else. Apple Watch has a consumer feature called "Audio Intelligence". It shares the name and nothing else. This guide covers the developer category: APIs that analyze speech inside your own product.

A transcript is not an answer. It's a wall of text that happens to be shaped like a conversation, and if you've ever tried to build a product on top of raw transcripts you know the feeling: you have all the words and none of the meaning.

Speech understanding is the layer that closes that gap. It takes the output of transcription (words, timestamps, speakers) and produces structured fields you can actually query: who felt what, which entities were named, what the conversation was about, whether anything in it violates policy, and which sixty seconds a human should listen to.

I run Voice AI at AssemblyAI, and the thing I'd most like to correct about how this category gets explained is the implication that it's a nice add-on. For most products built on voice, it's the actual product. Transcription is the cost of entry; what you do with the transcript is the thing customers pay for.

So: what speech understanding is, what's in it, how each piece works, and where the real engineering difficulty sits.

Speech understanding, defined

Speech understanding is the set of AI capabilities that extract structured meaning from spoken audio (sentiment, entities, topics, key moments, content safety signals and redacted personal information) on top of a transcript.

Three properties distinguish it from transcription:

It produces schema, not prose. A transcript is a string. Speech understanding output is typed: a sentiment score attached to a speaker and a time range, an entity with a category and a character offset, a topic with a relevance weight. Typed output is queryable; prose isn't.

It's per-segment, not per-file. A single sentiment label for a 40-minute call is nearly useless. The signal is in the trajectory: where the conversation turned, and who turned it.

It's grounded in the audio, not just the text. Good implementations carry timestamps and speaker attribution all the way through, so every extracted fact points back at the moment it came from. That's what makes the output auditable rather than merely plausible.

Audio intelligence vs speech understanding vs voice intelligence

These terms overlap, and LLMs and search engines blur them. Here's how they line up.

In short: audio intelligence and speech understanding name the same developer capability set, voice intelligence is the business outcome, and speech analytics is one reporting layer built on top. When the analysis runs on sales and support calls, you'll also hear it called conversation intelligence.

The full feature set

Here's what's in AssemblyAI's Speech Understanding API, organized by what each feature is for rather than by how it's implemented.

Anything that doesn't fit a fixed schema (arbitrary questions, custom summaries, structured extraction against your own JSON shape) runs through LLM Gateway, a single API to OpenAI, Anthropic, Google and other providers, pointed at your transcripts. That's the escape hatch, and it's the right one: fixed features for the things you'll run on every file, a general model for the things that change with your product.

Two naming notes, because both come up in older integrations. Key Phrases is the current name for what used to be branded "Automatic Transcript Highlights"; the feature is live, the old label isn't. And Auto Chapters is genuinely deprecated. If you're still calling auto_chapters, the migration path is Summarization under Speech Understanding.

What are Voice AI Guardrails, and which work with pre-recorded speech-to-text?

Guardrails are the safety, compliance and quality controls that run alongside transcription rather than after it: the layer that keeps a voice product from saying, storing or surfacing something it shouldn't.

They divide into two groups by what they protect.

Content and compliance guardrails operate on what was said: content safety classification against a policy taxonomy, profanity detection and masking, and PII redaction across both the transcript and the audio file itself. The audio side is the one teams forget. A transcript with a card number masked is not much protection if the recording still reads it aloud. Redacting both is what actually shrinks the scope of a security review.

Quality and cost guardrails operate on the system: confidence signals you can threshold on, so low-certainty output routes to review rather than straight into a customer-facing summary, and controls that keep a runaway session from becoming a runaway bill. There's a fuller treatment in built-in protection for compliance, quality, and cost control.

For pre-recorded speech-to-text specifically, these run in the same async request as transcription: content safety, profanity filtering, PII redaction in text and audio, entity detection, topic detection, sentiment analysis and audio event tagging. You don't call a second service. You set the parameters on the transcription request, and the structured output comes back alongside the transcript, sharing its timestamps and speaker labels.

That shared-timestamp property is worth dwelling on. When a content safety flag, a sentiment dip and a speaker turn all reference the same time index, you can reconstruct exactly what happened. When they come from three vendors that each segmented the audio differently, you can't, and reconciling them becomes its own project.

For live audio, the streaming path at wss://streaming.assemblyai.com/v3/ws redacts PII in final turns before they reach your client, and everything else runs turn by turn. More on that below.

How does speaker separation work?

Speaker separation, or speaker diarization, answers "who spoke when," and it's the feature most likely to quietly ruin everything downstream when it's wrong.

Here's the mechanism in the current pre-recorded flagship. Rather than transcribing first and clustering speakers afterward as a separate post-process, Universal-3.5 Pro jointly produces the transcript and the speaker change points. The model knows where a turn ended because it's the same model that decided what the words were. That matters most in exactly the conditions where naive diarization collapses: short turns ("mm-hm," "right," "no wait"), rapid back-and-forth, and overlapped speech where two people talk at once.

Measure it with the right metric. Most diarization is evaluated with DER (diarization error rate), which measures how much audio time got assigned to the wrong speaker. That's a fine acoustic metric and a weaker product metric, because it doesn't care whether the words ended up attributed correctly. cpWER (concatenated minimum-permutation word error rate) measures the thing you actually consume: did each speaker's words land under that speaker? On the October 2026 pre-recorded set on AssemblyAI's benchmarks page, Universal-3.5 Pro has the lowest mean DER of the systems tested at 12.2%, and a mean cpWER of 26.7%, second to ElevenLabs Scribe v2 at 26.4%.

For live audio, streaming diarization labels speakers as the stream runs, for up to 10 speakers, and keeps refining its speaker model over the session. When the stream ends, it sends one SpeakerRevision message with any turns whose labels changed, which adds about 400 ms. Set speaker_labels_revision_interval_ms if you want those corrections mid-session too. You get usable labels in real time without being stuck with the early, under-informed guess. Streaming speaker diarization goes into the mechanics.

One configuration note that catches people: turning on speaker labels in streaming applies diarization-tuned turn defaults (min_turn_silence 640 ms, max_turn_silence 768 ms) so turns break cleanly at speaker changes. If your turn timing shifts the day you enable diarization, that's why.

Practical guidance that saves people weeks: if your audio is stereo with the participants already on separate channels (most call recordings are), use multichannel transcription instead of diarization. Channel separation is ground truth; diarization is inference. Don't infer something you were handed.

Can conversations be analyzed in real time?

Yes, and the architecture is genuinely different from batch analysis rather than just faster.

The streaming path uses Universal-3.6 Pro Realtime over the v3 WebSocket, returning partial and final transcripts within a few hundred milliseconds. PII redaction runs on final turns in the stream, and each finished turn can go to LLM Gateway for sentiment, intent or entity extraction while the conversation is still happening, early enough to change it.

Three streaming-specific capabilities are worth knowing about, because they don't exist in the batch world:

agent_context. You pass in what your agent just asked, and the model uses it to resolve exactly the utterances that are hardest without context: a spelled-out email address, a one-word confirmation, a mumbled account number. On Universal-3.6 Pro Realtime, passing agent context cut word error rate by 10.2% across 20,000 voice agent audio files, with fabrications down 18.3% and hallucinations down 17.2%. It's included in the base rate, so there's no reason not to use it.

Context Carryover. A short rolling conversation memory, on by default, so the model carries what was established earlier in the call into how it interprets what comes later. David Zhao, Co-founder at LiveKit, singled this out when the previous model, Universal-3.5 Pro, came to LiveKit Inference:

"We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms."

voice_focus. Isolates the primary speaker and suppresses background speech and noise, with near-field for headsets and phones and far-field for rooms, kiosks and drive-thrus. Note the hyphens: those are the only accepted values, and an underscored variant is rejected with a validation error that closes the session. Real-time analysis in a noisy environment is mostly a signal-isolation problem before it's a modeling problem.

There are also three turn-detection modes instead of a pile of low-level flags: min_latency, balanced (the default) and max_accuracy for noisy or far-field audio. Rather than simply waiting out a silence timer, the model evaluates whether what has been said is complete, and combines that judgment with the mode's silence thresholds (min_turn_silence 128/128/512 ms and max_turn_silence 640/1280/2560 ms across the three modes, with vad_threshold defaulting to 0.2). Entity-aware endpointing holds the turn open when it ends mid-entity, which is why reading out an account number digit by digit doesn't get chopped in half the way a pure silence timer chops it.

The design question for real-time isn't "can I get the transcript fast." It's "what do I do differently because I have it now?" Real-time conversation intelligence covers the patterns that earn their keep: live agent assist, in-call compliance prompts, and escalation detection before the customer asks for a supervisor. The product surface is on the streaming speech-to-text page.

How do you surface the significant moments in a recording?

Nobody listens to the whole file. The product job is to find the 90 seconds that matter and put the user there.

Three mechanisms do most of the work, and they're complementary rather than competing.

Key Phrases ranks the phrases in a transcript by significance and returns them with timestamps. This is the cheapest useful thing you can run on a recording, and it's often enough on its own for a jump-to interface. It's English-only.

Sentiment trajectory. Per-segment sentiment is a curve, not a number, and the steepest slope is usually the moment worth reviewing. A call that runs neutral for 18 minutes and drops hard at minute 19 has told you exactly where to look and, with speaker attribution attached, who caused it.

Entity clustering. When the same entity (a competitor name, a specific policy number, a product SKU) appears repeatedly in a short window, that window is about something. This is the least-used technique on the list and one of the most effective, because it finds significant moments without needing to know in advance what "significant" means for your domain.

For anything domain-specific ("find the moment the customer stated their reason for cancelling," "pull the section where the candidate described the system they built"), send the transcript with its timestamps through LLM Gateway and ask for time ranges back. The fixed features handle the general case cheaply; the general model handles the case that's specific to your product.

Metaview does this for recruiting conversations, and Shahriar Tajbakhsh, their Co-founder and CTO, described what changed the quality of the output:

"Since moving to AssemblyAI, we've seen a meaningful improvement in the confidence tail of our production transcripts….What stands out is not just the model quality, but the way [they] let us bring real meeting context into transcription, from calendar titles to organizations, domains, and participant names, so recruiting conversations come through with the nuance our customers depend on."

The phrase to notice is "confidence tail." Average transcript quality is rarely the constraint on a speech understanding product. The tail is: the small share of segments where the model was unsure, which is disproportionately where the names, numbers and decisions live.

That's what contextual prompting on Universal-3.5 Pro addresses. You prime the model with a domain or prior context (meeting agendas, participant names, product and competitor vocabulary, prior-visit clinical notes) before it transcribes. The Universal-3.5 Pro release post covers how to use it.

How do you identify key topics across hours of audio?

Single-file topic detection is straightforward. Topic analysis across a corpus (a quarter of support calls, a year of podcast episodes, an entire research study) is a different job, and it's where teams most often build something that doesn't survive contact with real data.

The approach that works:

Start with IAB topic detection as your stable spine. A standardized taxonomy gives you labels that mean the same thing in January and in June, which is the whole point of trend analysis. Custom taxonomies drift as whoever maintains them changes their mind.

Layer entity frequency on top. Topics tell you the category; entities tell you the specifics. "Billing" as a topic is not actionable. "Billing" plus a spike in mentions of a specific plan name after a pricing change is a root cause.

Use LLM Gateway for emergent themes, then promote the durable ones. Run open-ended theme extraction over a sample, see what recurs, and once a theme proves stable, encode it as a keyterm list or a classification rule so it's measured consistently instead of rediscovered every run.

Weight by relevance and duration, not by count. A topic mentioned once in 400 calls and a topic that occupies half of 40 calls both produce a count. Only one is a signal.

One accuracy note that has outsized impact at corpus scale: if the transcription layer inconsistently renders your product names, topic and entity aggregation fragments across spellings, and your trend lines become meaningless. Custom spelling and keyterm prompting fix this at the source. It's an unglamorous configuration step that determines whether corpus analysis works at all, though keep the keyterm list scoped, because an over-broad list can pull the model off words it was already getting right.

Jiminny builds conversation intelligence for revenue teams on top of this kind of pipeline, and Tom Lavery, their CEO and Founder, framed the partnership side of it:

"AssemblyAI has a real high-touch personal service. It's a great partnership and we're very collaborative and get to test new AI models early and work together. And AssemblyAI is really pushing boundaries, helping us create a well-rounded conversation intelligence platform."

Can it handle enterprise-scale processing?

This is the question that decides deployments, and it has four parts.

Capacity and rate limits. Worth being precise, because "unlimited" is the wrong word and the real mechanism is favorable. On streaming there's no cap on how many sessions you hold open at once. What's governed is how many new sessions you open per minute (200 to start on a paid plan), and that allowance grows by 10% every minute you use 80% or more of it, with no ceiling, so sustained load lifts your limit rather than throttling it. Go over the current allowance and new connections are refused with close code 1008 while existing calls continue untouched. The allowance also relaxes back toward your starting limit after quiet periods, so a cold spike is the case to plan for. On pre-recorded, the limit is jobs processing in parallel (200+ on paid), and anything beyond it is queued first-in, first-out rather than rejected. Higher starting limits are available on request. This matters more than throughput benchmarks suggest, because enterprise audio volume isn't smooth: it's Monday morning, the day after an outage, the hour a campaign lands.

Deployment shape. Three options, and organizations pick by constraint rather than preference. Voice AI Cloud is AssemblyAI-hosted and the default. EU data residency runs the same models through api.eu.assemblyai.com and streaming.eu.assemblyai.com at the same price, with data processed and stored in the EU. Self-hosted runs inside your own infrastructure, a customer AWS VPC for instance, for teams with contractual or regulatory limits on where audio can travel.

Cost predictability. Billing is per second with no minimums and no upfront commitments, which means volume modeling is arithmetic rather than negotiation. Current rates for models, add-ons and speech understanding features are on the pricing page.

Model stability. Pin a specific model when you need reproducible output across a long-running analysis, or omit the model parameter to auto-upgrade to the latest Universal Pro release. Both are legitimate; the mistake is not deciding, and discovering during a compliance review that your Q1 and Q3 numbers came from different models.

For regulated industries, the supporting pieces are PII redaction across audio and transcripts, SOC 2 Type 2 and, for healthcare workloads, Medical Mode: one domain: "medical-v1" parameter on Universal-3.5 Pro for pre-recorded audio or Universal-3.6 Pro Realtime for streaming, in English, Spanish, German and French, as a $0.15/hr add-on. AssemblyAI also offers a standard Business Associate Addendum (BAA) for customers processing PHI.

Where the difficulty actually is

If you take one thing from this, take this: speech understanding output is only as good as the transcript underneath it, and the failure mode isn't obvious.

A classifier reading a corrupted transcript doesn't return an error. It returns a confident, well-formatted, completely wrong label, and it does that at scale, on every file, in a JSON shape that looks exactly like the correct output. There's no exception to catch. The only way to find it is to measure the transcript separately from everything built on it, which almost nobody does until an incident forces it.

That's the argument for caring about the model underneath more than the feature list on top. The transcription layer determines your ceiling; the speech understanding layer determines how close you get to it. Teams that pick the second without examining the first spend the following year tuning prompts against a problem that lives one layer down.

The other thing worth saying, because it's changed recently: speech understanding used to be a batch category. You processed a recording, you got fields back, you built a dashboard. The interesting products now run the same extraction on a live stream and do something with the result inside the conversation, and that reframes the whole category from reporting to intervention. It's a much better place for this technology to be, and it's a much harder engineering problem, which is usually a good sign about where the value is.

Frequently Asked Questions (FAQs)

What is an audio intelligence tool?

An audio intelligence tool is an API or platform that turns recorded or live speech into structured data: transcripts, speaker labels, sentiment, entities, topics, summaries and PII redaction. Today the category is usually called speech understanding. Developers use it to build products such as conversation intelligence, meeting notes and compliance monitoring without writing their own analysis models.

Is audio intelligence the same as speech understanding?

Yes. Speech understanding is the current name for the capability set that used to be marketed as audio intelligence, and AssemblyAI's Speech Understanding API is where those features now live. The name changed; sentiment analysis, entity detection, topic detection, key phrases, content safety and PII redaction are still the core, now joined by speaker identification, translation and LLM Gateway for custom extraction.

What can audio intelligence detect in a recording?

It can detect who spoke when, the sentiment of each sentence, named entities such as people, organizations, account numbers and medications, IAB topics, key phrases, sensitive or policy-violating content, profanity, personal information to redact, non-speech audio events and the spoken language. Everything comes back with timestamps, so each result points to the moment in the recording it came from. Anything outside those fixed features, such as a custom summary or a field in your own JSON schema, can be extracted by running the transcript through an LLM.

How is audio intelligence different from speech analytics?

Audio intelligence, or speech understanding, is the extraction layer: it turns speech into typed fields like sentiment, entities and topics. Speech analytics is a reporting layer built on those fields, usually inside contact center software, that scores calls, tracks trends and powers QA dashboards. Most speech analytics products run on audio intelligence APIs underneath.

Can audio intelligence run in real time?

Yes. A streaming speech-to-text model such as Universal-3.6 Pro Realtime returns partial and final transcripts within a few hundred milliseconds, redacts PII in final turns, and labels speakers live. Each finished turn can then go to an LLM for sentiment, intent or entity extraction while the call is still going, which is how live agent assist and in-call compliance prompts work.

Is the Apple Watch "Audio Intelligence" feature the same thing?

No. Apple Watch's "Audio Intelligence" is a consumer feature that shares the name. Audio intelligence in the developer sense is a category of APIs, now usually called speech understanding, that developers call from their own products to analyze conversations, calls and recordings.