Inside dictation cleanup: How raw speech becomes finished text

Here’s a sentence the way a speech model receives it, with nothing taken out:

um, so, the patient — the patient is tired

And the same sentence after cleanup:

The patient is tired.

Both are correct. They answer different questions. The first is a record of what was said; the second is what the speaker meant to write down, and it’s the one that belongs in a chart note, a Jira ticket, or a text field a human is staring at.

Nobody wants to copy and paste their own ums.

The gap between those two outputs is where dictation products live, and it’s less mysterious than it looks. Two separate jobs happen between the moment someone speaks and the moment finished text appears. They run in a fixed order, they fail in completely different ways, and confusing them is why so much do-it-yourself dictation feels broken in a way teams can’t diagnose.

This page is the map — every stage, in order, including the one that can’t be fixed downstream no matter what you put in your cleanup prompt.

Stage 0 — Getting the words right in the first place

Recognition happens before cleanup. That’s not a technicality, it’s the argument: cleanup inherits whatever recognition got wrong. So the first controls to reach for are the ones that shape what gets heard, not how it reads.

Two ideas do jobs that sound similar and aren’t, and the distinction is worth memorizing before you write a line of code:

A prompt tells the speech model what it is about to hear — the setting and the vocabulary. A rewrite instruction tells a language model what to do with the transcript after it is written down.

Contextual prompting is the transcription-context control. You describe the setting and the terms the model should expect: a cardiology follow-up, a standup for a Kubernetes team, a support call about one product line. It’s a capability of the Universal-3.5 Pro line rather than a keyword list, and describing the domain generally buys you a whole vocabulary neighborhood instead of one term at a time.

Keyterms prompting is the sharper instrument. It takes terms you already know are coming — package names, drug names, the six people on the account — and biases transcription toward them. Our guide to keyterms prompting covers building that list without stuffing it.

Conversation context carries prior turns of dialogue, either side, so a follow-up utterance is heard against what came before it rather than in isolation.

One tradeoff nobody has published: explicit language selection is ignored when a custom prompt is set. The moment you use a prompt to nail domain jargon, you give up pinning the language on that request. Single-language products lose nothing. Multilingual ones are choosing between steering vocabulary and pinning language, and that should be a decision rather than a bug report — if the language matters more than the jargon, drop the prompt and lean on keyterms.

What disfluencies are, in six categories

A disfluency is any element of spontaneous speech that carries no content the speaker intended to write down — a hesitation sound, a repeated word, an abandoned phrase, a restart. Fluent speech is full of them; written text almost never is. That mismatch is the whole reason dictation cleanup exists.

They aren’t one thing, and the categories fail differently. Six worth naming:

Filled pauses are the non-lexical sounds people make while thinking: uh, um, er, ah. Zero content, safest to remove. “um, can we push the deploy to Thursday” becomes “Can we push the deploy to Thursday?”

Discourse markers are real words used as verbal padding — you know, I mean, I guess, plus leading so, well, right, and okay. Harder, because the same words carry meaning elsewhere. In “So, right, I mean the invoice went out Monday,” all three openers are padding. In “Right, so the invoice was wrong,” right is agreement, and dropping it changes what the sentence says.

False starts are an abandoned fragment followed by a restart: “we should ref-, we should refactor the auth layer first.” Cleaning one means dropping the abandoned attempt and keeping the completed one, which requires recognizing that both spans are the same attempt.

Stammered repeats are immediate, unintended duplication. “Send it to to the finance team” is one to. “It’s a a really tight deadline” is one a. The tell is adjacency — the repeat sits right next to itself with nothing in between.

Self-corrections are the hardest case, because some carry the point of the sentence. “Book the room for Thursday, sorry, make that Friday” has to come back as “Book the room for Friday” — the correction is the content, and a system that keeps both has produced a wrong meeting. Compare “the, uh, the north campus,” where nothing is being corrected at all; that’s a filled pause wrapped in a repeat. Telling those apart is the difference between a cleanup pass you’d trust with a calendar and one you wouldn’t.

Trailing abandonment is a phrase started and dropped, with the speaker moving on rather than restarting: “I’ll loop in legal if we need to — anyway, ship it.” There’s no completed version to fall back on, which makes it the category most likely to be over-trimmed. The safe behavior is to cut the broken-off fragment and leave the surviving clause intact.

Two of the six — self-corrections and trailing abandonment — are judgment calls. The other four are mechanical. Evaluate any cleanup system, ours or yours, on the two, because everyone gets the four right.

The parameter direction, stated carefully

On pre-recorded audio, filler words are removed by default, and setting disfluencies: true preserves them. The name reads like a switch you flip to get rid of them. It’s a switch you flip to keep them.

Worth being blunt about why we’re spelling that out: the playground’s own description of the setting stated the inverse of what the setting does, until it was corrected on August 19. If you built against it before then and your output has more ums than you expected, that’s why.

Stage 1 — Recognition

Short to describe, impossible to skip.

The transcript comes from Universal-3.5 Pro — the same flagship model behind our pre-recorded and streaming products, not a trimmed-down variant tuned for speed. That matters more than it sounds like it should. In most stacks, “make it faster” means “use the smaller model,” and the accuracy tax hides inside the latency win. Ours doesn’t work that way: the model doing dictation is the model doing everything else. Numbers live on the benchmarks page rather than getting minted here.

Coverage is 19 languages, with native code-switching across all of them — which matters for dictation more than it does for recorded audio, because people switch languages mid-sentence far more readily when they’re talking than when they’re writing.

For dictation specifically, this runs on the short-clip path: the Sync API is built around audio that’s already complete and a user waiting to see what it says, which is a different problem from a live stream and a different problem from a recording processed in the background.

Stage 2 — The rewrite

Now the second job, and the one teams underestimate.

A rewrite pass takes the transcript and returns the text the speaker meant. Wire it up yourself and you’ll send the transcript to a language model — through something like the LLM Gateway — with a prompt describing the transformation you want.

The smallest useful version removes filler sounds, false starts, and stammered repeats, and changes nothing else. Every surviving word keeps its spelling. Every punctuation mark stays as it was produced. No reordering, no tightening, no tone.

That conservatism is a design decision worth making deliberately rather than stumbling into. A rewrite pass allowed to improve your sentences will eventually improve one you didn’t want improved, and you won’t find out until a user does. Start with the smallest transformation that helps and widen it only when someone asks.

Past that floor, the instruction is where the product lives. Same audio, different instruction, different finished text: a chart note, a booking confirmation the customer reads, a JSON object shaped like the form fields you’re populating.

Three things builders learn the hard way, which you can learn here instead.

Describe only the transformation. The most common prompt bug is asking for too much at once. Tell the model what to change and nothing else — and then handle separately the things a prompt shouldn’t be trusted with, like making sure it returns only the rewritten text, leaves clean input alone, and writes down a question it hears rather than answering it. People dictate “wait, translate this into French” constantly, and it should come back written down rather than carried out.

Short instructions are faster. You’re paying for the prompt on every single utterance, in tokens and in time. That long rule-by-rule cleanup prompt is thousands of tokens, charged on a twelve-word sentence.

Decide what happens when the second hop fails. It will. The rewrite is a separate inference call across a separate connection, and it can time out while the transcript underneath is perfectly fine. Whatever your text field renders in that moment — the verbatim transcript, an error, a spinner — is a product decision, and it’s better made now than at 2am. The good version degrades to unpolished text rather than to a wait.

The failure mode that matters: a fluent wrong answer

Here’s the one that should shape how you build.

A developer dictates a sentence about an HTTP client: “I’ll swap the fetch calls over to axios.” The speech model doesn’t know axios — it’s a package name, not English — so it hears the closest thing it does know and writes down “axe eos.”

Run the rewrite over that. It removes nothing, because there’s nothing to remove. What it does do is treat two words as a proper noun in a technical sentence and capitalize them. The finished text reads: “I’ll swap the fetch calls over to Axe EOS.”

That output is worse than the raw one. It’s cleanly punctuated, confidently capitalized, and completely wrong — and because it looks finished, nobody reviewing it has a signal that something went sideways. The messy version at least looks like it needs checking.

A fluent wrong answer is worse than a messy right one.

So state it plainly: the rewrite polishes, it does not correct. It reformats the words it was handed. It cannot recover a word that was never heard, because that word isn’t in its input — there’s nothing in “axe eos” from which axios can be reconstructed. Adding “the user means the axios library” to your rewrite instruction does nothing, because the failure already happened one stage earlier.

The fix goes upstream, back to Stage 0. Put axios in your keyterms. Describe the setting in a prompt — “a developer discussing a JavaScript codebase” buys you the whole vocabulary neighborhood, not one term. That’s the difference between fixing a class of errors and patching one instance. It’s the same upstream work behind dictating prompts to coding agents, where package names are most of the sentence.

It’s also why aggregate word error rate is the wrong instrument here. Miss one word in forty and your WER looks fine; if that word was the drug name or the customer’s surname, the output is unusable. Entity accuracy tracks what users notice, and it’s worth understanding how speech-to-text accuracy is measured before picking a number to hold a vendor to.

When you should not clean up

Cleanup is what most products want. It isn’t what every product wants, and the exceptions aren’t edge cases.

Medical records sometimes need verbatim, because formatting can carry meaning — especially when the note is about to be read by another model, where a removed hesitation or a normalized phrase changes what gets extracted. A clinical dictation workflow feeding a summarizer is a different requirement from one feeding a human reader.

Physicians are also trained, in medical school, to speak their punctuation. “patient open parentheses Andrew close parentheses is tired” is a rendering instruction, not a filler pattern, and the expected output is patient (Andrew) is tired. Untraining that habit is a non-starter. Some teams go further and want zero model-generated punctuation at all, with deterministic conversion of spoken punctuation commands and nothing else — a legitimate spec, and incompatible with a rewrite pass by design. Legal and compliance work lands in the same place for the same reason: the record has to be the record.

When that’s the requirement, the answer is simple: don’t run a rewrite stage. Take the transcript.

Where the words themselves carry the clinical risk, the answer is still upstream. Medical Mode is the domain-specific version of Stage 0 — one parameter, "domain": "medical-v1", on Universal-3.5 Pro — and it reduces Missed Entity Rate on drugs, conditions, procedures, and clinical terms by roughly 20%. It covers English, Spanish, German, and French. Contextual prompting compounds it: feeding a patient’s prior-visit note cut missed medical terms by 31% in an internal healthcare test, even when the note came from an earlier visit.

For teams handling PHI: AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI.

None of that is the rewrite doing anything. It’s recognition getting the word right the first time, which is the only place the word can be gotten right.

What the frontier looks like

The interesting problem in dictation cleanup isn’t removing more. It’s knowing what not to remove.

That’s not philosophy. The prompt revision that improved false-start removal threefold on our eval set did it mostly by adding rules about which words to keep — the content words a naive filler-stripper eats, the right that means agreement rather than padding, the second half of a self-correction. Every gain came from teaching the pass to be more careful, not more aggressive.

Which suggests the ceiling on cleanup quality isn’t model capability. It’s specification. The teams shipping dictation features nobody complains about are the ones who wrote down, explicitly, what their product is never allowed to touch, then tested against that list instead of a word error rate. If you take one action after reading this, make it that list — everything else in dictation is downstream of it.

Earlier in the decision than that? What a dictation API is covers the shape of the request, and the features a dictation experience needs covers what ships alongside cleanup.

Frequently asked questions

What are disfluencies?

Disfluencies are the elements of spontaneous speech that carry no content the speaker intended to write down — filled pauses like um and uh, discourse markers like you know and I mean, false starts, stammered repeats, self-corrections, and phrases broken off mid-thought. They’re normal in fluent speech and almost absent from written text, which is why turning speech into finished text means deciding what to do with them. Filled pauses and stammered repeats are mechanical to remove; self-corrections and trailing abandonment take judgment, because some of them carry the meaning of the sentence.

How do I remove filler words from a transcript?

On pre-recorded audio, filler words are removed by default, and setting disfluencies: true preserves them instead — the parameter keeps fillers rather than stripping them. For short dictated clips where you want more than filler removal, run the transcript through a rewrite pass with an instruction describing the transformation. Keep the verbatim transcript alongside the rewritten version, so your product can show either one and never has to guess which it’s holding.

How does AI dictation cleanup work, turning spoken filler words into clean written text?

Cleanup runs after recognition, as a second pass over the transcript. The speech model produces a verbatim transcript, then a rewrite pass removes disfluencies and applies whatever transformation you asked for in a plain-English instruction. The rewrite only reformats the words it was given — it never re-listens to the audio, so it can polish text but cannot change which words were recognized.

Can dictation cleanup fix a word the speech model got wrong?

No. Cleanup reformats the transcript it receives, so a misheard term comes back cleanly formatted and still wrong — axios heard as “axe eos” becomes “Axe EOS,” which reads more authoritative than the raw version and is just as incorrect. A fluent wrong answer is worse than a messy right one, so the fix is steering recognition upstream with a prompt and keyterms, not adding rules to the rewrite instruction.

How do I improve accuracy for custom vocabulary and industry jargon in a dictation API?

Use a prompt to describe the setting and the kind of terms the model should expect, and keyterms to list the specific terms you know will appear. Describing the domain generally beats listing terms one by one, because it primes a whole vocabulary neighborhood. One caveat to plan around: explicit language selection is ignored when a custom prompt is set, so using a prompt for jargon costs you language pinning on that request.

When should you keep a verbatim transcript instead of cleaned-up text?

Keep verbatim when the exact words are the deliverable — medical records that feed downstream models, legal and compliance recordings, and any workflow where a removed hesitation could change what gets extracted. It also applies when users speak their punctuation, as clinicians are trained to do, since “open parentheses” is a rendering instruction rather than a filler pattern. In those cases, skip the rewrite stage entirely and take the transcript.