How to evaluate a dictation API: criteria, tests, and metrics

Most dictation API evaluations go wrong the same way. A team pulls a public benchmark, sorts vendors by word error rate, picks the lowest number, and only finds out three weeks into the integration that users keep editing the output before they hit send.

Users don’t judge dictation by its word error rate. They judge whether the text they get back is something they’d send. Did the “um” disappear? When they said “Tuesday, no, Wednesday,” did the output say Wednesday? Is the customer’s name spelled the way it is in the CRM? Did the text show up before they lost patience?

This guide is a method, not a leaderboard. It covers the five criteria that predict whether a dictation API will hold up in your product, the metrics to use for each one, and a 20-utterance test you can run in an afternoon on your own audio. If you’re looking for a vendor-by-vendor rundown with prices, that lives in our companion piece on the best dictation APIs for developers. Use that one to build a shortlist and this one to test it.

The five criteria, at a glance

The order matters. Cleanup and latency are what users feel on every single request. Jargon accuracy is what makes or breaks you in a specific domain. Integration and compliance decide whether you can ship at all.

Criterion 1: Cleanup quality

A speech-to-text API returns a transcript. A dictation API should return finished text. That’s the whole distinction, and it’s why you can’t evaluate the two the same way.

People don’t speak in clean sentences. They start over, correct themselves mid-thought, and fill pauses with “so” and “like.” A verbatim transcript captures all of that faithfully, which is exactly what you want for a meeting record and exactly what you don’t want in a text field. Our deep dive on how dictation cleanup works covers the mechanics. For evaluation, focus on four behaviors:

• Filler removal. “Um,” “uh,” “you know,” and false starts should be gone without the meaning changing.
• Self-correction resolution. “Send it to Mark, sorry, to Mike” should come back as “Send it to Mike.” This is the hardest one and the one users notice most.
• Punctuation and formatting. Sentences, capitalization, numbers, dates, and email addresses should look the way a person would type them.
• Restraint. Cleanup shouldn’t rewrite the user’s voice, add words they didn’t say, or “improve” their phrasing. Over-editing is a failure too. If your team relies on detailed instructions, score the smart and fast rewrite modes separately; they follow instructions differently and sit at different latencies.

The metric that captures all of this is edit rate: for each test utterance, would a user change the output before sending it? Score each one yes or no. It’s blunt, but it maps directly to the experience you’re shipping.

One more thing to check: can you see both versions? The AssemblyAI Dictation API returns the verbatim transcript in text and the cleaned version in llm_response in the same response, so you can show finished text by default and fall back to verbatim when you need to. If a vendor only returns one or the other, you lose either debuggability or the cleanup.

Criterion 2: Latency, measured the way users experience it

Dictation is a push-to-talk interaction. The user holds a key or taps a button, talks, lets go, and waits. Every millisecond after they let go is dead time.

So the number that matters is time to finished text: from the moment the user stops recording to the moment cleaned text is ready to insert. It’s not the model’s processing time, and it isn’t time to first partial. If cleanup runs as a second step, its time counts too.

Here’s how to measure it honestly:

• Measure from the client, in the region where your users are, over a real network. Server-side timing hides DNS, TLS, and upload time.
• Report p50 and p90, not an average. A tool that’s fast half the time and slow the rest feels slow.
• Use realistic clip lengths. Most dictation is a few seconds to a minute. Test across that range, and add one or two longer finished notes.
• Separate cold and warm requests. The first request on a new connection pays for the handshake. Pre-warming the connection before the user starts talking moves that cost off the critical path, and it’s worth testing both ways.

For reference, typical short clips on the AssemblyAI Dictation API come back in under one second, with cleanup included. Longer finished recordings take proportionally longer because nothing is transcribed until the upload completes. Don’t take that or any vendor’s number at face value, though. Run the test below and look at your own p90.

Streaming partials are a red herring here. They matter for live captions and voice agents, where text has to appear while someone is still talking. For push-to-talk dictation, nobody reads the partials. They read the final text.

Criterion 3: Jargon accuracy, not word accuracy

Here’s the uncomfortable truth about word error rate: it treats every word the same. Misspelling “the” costs exactly as much as misspelling your customer’s last name, a drug name, or an order number. In dictation, the second kind of error is the one that gets text deleted and retyped.

That’s why we argue that word error rate is broken as a standalone metric. For dictation, track entity accuracy instead: names, product terms, acronyms, numbers, emails, and addresses. Specifically:

• Seed your test set with the terms you care about. If your users dictate CRM notes, include account names. If they write code comments, include function names.
• Score entity hit rate. Of the seeded terms spoken, how many came back spelled exactly right?
• Test with and without steering. Most dictation APIs let you pass a vocabulary list or context. Measure the lift, not just the baseline.

On the AssemblyAI Dictation API, there are two steering inputs, and they do different jobs. keyterms_prompt takes the exact strings you expect to hear (up to 100 terms and 8,000 characters), and stt_prompt takes a plain-language description of the setting (up to 6,000 characters) so ambiguous audio resolves toward words that make sense there. The prompting and keyterms section of the Dictation API docs covers when to use each.

Context matters more than most teams expect. Metaview, which builds AI notes for recruiting conversations, put it this way:

“Since moving to AssemblyAI, we’ve seen a meaningful improvement in the confidence tail of our production transcripts....What stands out is not just the model quality, but the way [they] let us bring real meeting context into transcription, from calendar titles to organizations, domains, and participant names, so recruiting conversations come through with the nuance our customers depend on.”

— Shahriar Tajbakhsh, Co-founder and CTO, Metaview

Dictation works the same way. If your app already knows who the user is writing to and what they’re writing about, pass that in, then measure whether it helps.

Criterion 4: Integration effort

Update, October 2026: The Dictation API now accepts finished recordings up to 20 minutes (256 MB, including MP3, M4A, AAC, Ogg/Opus, FLAC, and WebM) on POST /v1/transcribe. The 120-second limit still applies when you upload while recording through /v1/transcribe/live or the Python SDK. A new rewrite_mode (fast or smart) gives you more control over formatting, and llm_instruction is no longer capped at 2,048 characters. Details in the Dictation API docs.

Count the moving parts between the microphone and the text box. A do-it-yourself dictation pipeline usually looks like this: record audio, send it to a transcription API, wait, send the transcript to an LLM with a cleanup prompt, wait again, handle failures from either one, then insert the text. That’s two vendors, two bills, two sets of timeouts, and a prompt you now maintain.

A dictation API collapses that into one call. The AssemblyAI Dictation API runs on Universal-3.5 Pro, and it’s a single HTTP POST with the audio and a small JSON config, and the response carries both the transcript and the finished text. You don’t need an SDK or a WebSocket.

When you score integration effort, look past the happy path:

• What happens when cleanup fails? You want the transcript back regardless. The AssemblyAI Dictation API gives the rewrite step a 5-second internal deadline; if it misses, llm_response comes back null with an llm_error value, and you still get the verbatim text. Your UI can fall back cleanly.
• What are the request limits? The AssemblyAI Dictation API accepts finished recordings up to 20 minutes per call over HTTP, including compressed formats such as MP3, or up to 120 seconds when you upload while recording, in WAV or raw 16-bit PCM only. Compressed audio on the upload-while-recording path is rejected with a 415 error, so check what your recorder produces.
• Can you customize cleanup? Default cleanup fits most apps. When it doesn’t (say you want bullet points or a specific tone), llm_instruction replaces the default behavior, and rewrite_mode lets you pick fast for latency or smart for detailed formatting instructions.
• Which languages do you need? The AssemblyAI Dictation API supports 32 languages. List the languages your users actually speak and test each one with native speakers.

Building it yourself is a real option, and sometimes the right one. We compare both paths in Dictation API vs speech-to-text plus an LLM, and the section near the end of this guide covers when it wins.

Criterion 5: Compliance and data flow

Dictation audio is often the most sensitive audio your product handles. People dictate emails to their lawyer, notes about patients, and messages to their partner. Your security team will ask where it goes, so get the answers before you’re deep into an integration.

Questions to put to every vendor:

• Is there an independent audit report, such as SOC 2 Type 2?
• Is customer audio used to train models, and can you opt out?
• How long are audio and transcripts retained?
• If cleanup runs on a third-party LLM, which one, and under what terms?
• Can the vendor sign a Business Associate Addendum (BAA) if you process protected health information?
• Is there regional data residency if you need it?

AssemblyAI maintains a SOC 2 Type 2 report covering security, availability, and confidentiality, audited annually by an independent firm. The AssemblyAI security page has the details. For healthcare teams, AssemblyAI is considered a business associate under HIPAA and offers a Business Associate Addendum (BAA); the BAA FAQ explains how it works.

Build your test set: 20 real utterances

You don’t need hundreds of hours of audio to make a good decision. You need 20 clips that look like what your users will actually say, recorded the way they’ll actually record them. Here’s the mix we recommend:

A few rules make the results trustworthy:

• Use real people and real devices. Laptop mics, phone mics, and headsets sound different. Text-to-speech audio flatters every vendor.
• Write down what the user meant to send before you run anything. That’s your reference text, and it’s what you score against.
• Run every vendor on the same 20 clips, from the same machine, in the same session.
• Score blind if you can. Strip vendor names from the outputs before someone judges edit rate.

Run the test and time it

The script below sends each clip in a folder to the Dictation API, measures client-side time to finished text, and prints p50 and p90. It reuses one HTTP session, so you can compare the first (cold) request against the rest (warm). Swap in your own key terms, and adapt the request for each vendor you’re testing.

import glob, json, statistics, time
import requests

API_URL = "https://dictation.assemblyai.com/v1/transcribe/live"
API_KEY = "<YOUR_API_KEY>"
CONFIG = {"keyterms_prompt": ["Priya Raman", "Zendesk", "SKU-4471"]}

session = requests.Session()  # reuse the connection across requests
results = []

for path in sorted(glob.glob("test_clips/*.wav")):
    with open(path, "rb") as f:
        audio = f.read()
    start = time.perf_counter()
    resp = session.post(
        API_URL,
        headers={"Authorization": API_KEY},
        files={
            "config": (None, json.dumps(CONFIG), "application/json"),
            "audio": (path, audio, "audio/wav"),
        },
        timeout=90,
    )
    elapsed_ms = (time.perf_counter() - start) * 1000
    body = resp.json()
    results.append(elapsed_ms)
    print(f"{path}: {elapsed_ms:.0f} ms")
    print("  verbatim:", body.get("text"))
    print("  finished:", body.get("llm_response") or f"(fallback, llm_error={body.get('llm_error')})")

warm = results[1:] or results
print(f"cold request: {results[0]:.0f} ms")
print(f"warm p50: {statistics.median(warm):.0f} ms")
print(f"warm p90: {statistics.quantiles(warm, n=10)[-1]:.0f} ms")

Paste the verbatim and finished output for each clip into a spreadsheet next to your reference text. Then fill in the scorecard.

Metrics that matter: WER vs entity accuracy vs finished-text quality

Different metrics answer different questions, and mixing them up is how evaluations end in the wrong place.

Notice the trap in the first row. If you score a dictation API’s finished output with WER against a verbatim reference, good cleanup looks like errors: every removed “um” counts as a deletion. Score the verbatim text with WER if you want to compare transcription quality, and score the finished output with edit rate. Our broader guide on how to evaluate speech recognition models goes deeper on running WER fairly, including text normalization.

Scoring template

Weight each criterion by how much it matters to your product, score each vendor from 1 to 5, and multiply. Here’s a starting point for a general-purpose dictation feature:

Adjust the weights to fit your product. A clinical notes product should weight entity accuracy and compliance much higher. A consumer messaging app might weight latency above everything. Price usually breaks ties rather than deciding the result, so compare it last, as cost per finished hour. The AssemblyAI Dictation API is $0.62 per hour with cleanup included; the Sync API is $0.30 per hour for the transcript alone, before you add your own LLM.

When Sync API plus your own LLM beats a dictation API

A bundled dictation API isn’t always the right call. Building the pipeline yourself makes sense when:

• Cleanup is your product. If your differentiation is a specific rewriting style, a fine-tuned model, or multi-step formatting, you’ll want full control of the LLM step.
• You already run an LLM in the request path. If the text goes straight into an agent or a summarizer anyway, a separate cleanup pass may be redundant.
• Your recordings run longer than 20 minutes, or need diarization. The Dictation API caps finished recordings at 20 minutes (120 seconds when uploading while recording). Long-form, multi-speaker recordings belong on an async speech-to-text pipeline.
• You need text while the user is still talking. Live captions and voice agents need streaming, not push-to-talk.

In AssemblyAI’s own dictation workshops, engineers have built the DIY version live with the Sync API for the transcript, connection pre-warming to hide network cost, and a small model on LLM Gateway for cleanup. It works well. The question your evaluation should answer is whether the control is worth the second call, the prompt maintenance, and the extra failure path. For most teams adding a dictation feature, it isn’t. For teams whose product is the dictation, it often is.

What a good evaluation leaves you with

When you finish, you should have more than a winner. You’ll have a 20-clip test set that reflects your users, a reference text for each clip, and a scorecard you can rerun in an hour. Keep all three.

Dictation models improve fast. Whatever you pick today, rerun the same test every quarter and whenever a vendor ships a new model. The teams that get dictation right treat evaluation as a habit, not a one-off procurement task, because the test set is what lets them notice when something better shows up.

If you want to see the product side of this, the Dictation API product page has the full feature list.

Frequently asked questions

How do I evaluate and choose a dictation API for my product?

Record 20 clips that match how your users really speak, write down what each speaker meant to send, and run every candidate API on the same clips. Score five criteria: cleanup quality (edit rate), latency (client-side p50 and p90 time to finished text), entity accuracy on your domain terms, integration effort, and compliance. Weight them for your product and compare price last, as cost per finished hour.

What are the key criteria for evaluating a speech recognition API?

For transcription in general, the core criteria are accuracy on audio like yours, latency, language coverage, customization such as keyterms or prompting, integration effort, security and compliance, and cost. For dictation specifically, add cleanup quality: whether the API returns finished text with fillers removed, self-corrections resolved, and formatting applied, rather than a verbatim transcript.

What is considered a good word error rate?

There’s no universal threshold, because WER depends heavily on the audio. The same model can score in the low single digits on clean read speech and much higher on noisy conversational audio. For example, Universal-3 Pro scores 1.52% WER on LibriSpeech Clean but 8.80% on Earnings21 earnings calls. Compare vendors on your own audio rather than against a fixed number, and for dictation, pair WER with entity accuracy and edit rate.

How should I measure dictation API latency?

Measure time to finished text from the client: start the timer when the user stops recording and stop it when cleaned text is ready to insert. Run it in your users’ region over a real network, report p50 and p90, and test cold and warm connections separately. Server-reported processing time leaves out network and upload time, so don’t rely on it.

How do I test custom vocabulary and jargon accuracy in a dictation API?

Seed your test clips with the names, product terms, and acronyms your users actually say, then measure entity hit rate: the share of those terms spelled exactly right. Run the test twice, once with no steering and once with the vendor’s vocabulary or context feature turned on, so you can see the lift. On the AssemblyAI Dictation API, those inputs are keyterms_prompt for exact terms and stt_prompt for context.

Is it worth paying for a dictation API with built-in LLM formatting, or should I combine a transcription API with an LLM myself?

For most teams adding dictation to an existing product, the bundle is worth it: one call, one bill, and a built-in fallback to the verbatim transcript if cleanup fails. Building your own pipeline makes sense when cleanup is your differentiation, when an LLM already sits in the request path, or when clips run past the bundled API’s length limit. Compare the two on cost per finished hour, including LLM tokens and engineering time.