How to evaluate TTS for voice agents

Your voice agent reads back a confirmation: "Your order number is W-three-seventeen…" The caller writes it down, hangs up, and calls back an hour later because the code doesn't work. The voice sounded great. It just said the wrong thing.

That's the gap in most TTS evaluations. Here's how to evaluate TTS for voice agents: test it in four layers, perceived quality, blind preference, correctness on names and numbers, and behavior under production conditions. For a voice agent, getting a confirmation code right matters more than a small gap in mean opinion score.

This guide walks through each layer, what it catches, what it misses, and how to build a round-trip entity accuracy test you can run on any provider.

Why voice agents need a different TTS evaluation

Most TTS evaluation was designed for read-aloud speech: audiobooks, narration, screen readers. A listener hears a clean paragraph through good headphones and rates how natural it sounds. That's a fair test for narration. It isn't the job a voice agent does.

The difference between conversational TTS and read-aloud TTS shows up in four places:

• Short turns. Agents say "Got it." and "Can I get the email on the account?" far more often than they read paragraphs, and short replies give the model little context for tone.
• Telephony audio. Most calls still travel as 8 kHz audio. Subtle quality differences you hear on a laptop can disappear, and consonant errors can get worse.
• Entities. Order numbers, prices, dates, phone numbers, emails, and names carry the actual business of the call.
• Interruptions. Callers barge in, and the agent has to stop cleanly and pick up again.

Generic quality scores don't measure any of these directly. So treat them as a starting filter, then test what your agent actually says, the way your callers will hear it.

What mean opinion score (MOS) tells you, and what it misses

Mean opinion score (MOS) is the average of listener ratings on a 1–5 scale, where 1 is bad and 5 is excellent. The method comes from telephony. ITU-T Recommendation P.800 defines it as an Absolute Category Rating test, and TTS research adopted it to score naturalness.

To run a TTS MOS test that means something:

• Use enough listeners. A handful of colleagues produces noisy averages. Recruit more raters than you think you need and report a confidence interval, not just the mean.
• Randomize and blind. Shuffle clips across systems so raters can't learn which voice is which.
• Include anchors. Mix in human recordings so you know where "real" lands on your scale.
• Use your sentences. Rate lines your agent will actually say.

Comparison MOS (CMOS) asks raters to compare two clips directly on a −3 to +3 scale. It's more sensitive than absolute MOS when two systems are close.

So what's a good MOS score for TTS? There isn't a portable number. Telecom teams often treat scores around 4 as toll quality, but TTS MOS from different studies, raters, and sentence sets can't be compared directly. Use MOS to compare systems inside one test you control.

What MOS misses matters more for agents. Raters judge how a clip sounds, not whether "$1,249.50" was read as "twelve forty-nine fifty." A voice can score high and still swap a digit. MOS also averages away rare failures, and it's rarely run on phone-quality audio.

Blind preference tests and TTS arenas

A blind preference test plays two clips of the same text from different systems and asks which one the listener prefers. Aggregate thousands of these pairwise votes and you get a ranking. That's how public TTS arenas and leaderboards work.

Pairwise tests are easier for listeners than absolute scores, and arenas collect far more votes than most teams can. They're a good screening tool. When you're narrowing a long list to a shortlist, a TTS leaderboard is a reasonable place to start.

But they're necessary, not sufficient, for a voice agent. Three gaps stand out:

• Generic prompts. Arena sentences are written to sound good, not to stress-test order IDs or street addresses.
• Wideband listening. Voters usually listen on headphones or laptop speakers, not through a phone line.
• Single clips. A vote judges one utterance. It doesn't capture voice drift across a ten-minute call or behavior after an interruption.

Run your own blind A/B on your own scripts, ideally after converting to phone audio. It's the same test with the conditions that matter to you.

Test what your agent actually says: names, numbers and IDs

This is the layer leaderboards skip, and it's where agents lose callers. The goal is simple: for every entity your agent speaks, check that the audio says exactly what the text said.

Build a test set from your scripts

Pull real templates from your agent's prompts and logs, then fill them with realistic, varied values. Cover every entity type you speak:

A few hundred sentences across these types is a solid start. Keep the set fixed so you can rerun it on every provider and every model update.

Round-trip entity accuracy: TTS → STT → compare

Round-trip entity accuracy is the share of entities that survive a trip from text to TTS audio and back to text through a strong speech-to-text model. Synthesize each test sentence, transcribe the audio, normalize both sides, and check whether the expected entity appears. Report a pass rate per entity type, not one blended score.

Why not just measure word error rate? Because word error rate is broken for this job. One wrong digit in a 20-word sentence is a 5% WER and a 100% failed call. The same lesson applies on the speech-to-text side, where entity accuracy predicts real-world outcomes better than WER.

Two rules keep the measurement honest. First, pin the speech-to-text model so the measuring instrument doesn't change between runs. Second, don't give it hints: no prompt and no keyterms, because those bias transcription toward the expected spelling and hide mispronunciations.

Here's a working version. It uses AssemblyAI's pre-recorded speech-to-text with Universal-3.5 Pro, which is built for entity accuracy on things like phone numbers, emails, addresses, and proper nouns, pinned to one model. The TTS call is a placeholder, so plug in whichever provider you're evaluating. Start with a CSV of test cases:

id,text,expected,entity_type
a1,"Your order number is W-317842.",W-317842,id
a2,"That comes to $1,249.50.","$1,249.50",money
a3,"You can reach us at (415) 555-0123.","(415) 555-0123",phone
a4,"Thanks, Siobhan. You're all set.",Siobhan,name

Then the round-trip script:

import csv
from collections import defaultdict
from pathlib import Path

import assemblyai as aai  # pip install assemblyai
from normalize import entity_present

aai.settings.api_key = "<YOUR_API_KEY>"

# Pin one model so the measuring instrument never changes.
# No prompt and no keyterms, so speech-to-text can't "help" the TTS by guessing the expected spelling.
config = aai.TranscriptionConfig(speech_models=["universal-3-5-pro"], language_code="en")
transcriber = aai.Transcriber(config=config)


def synthesize(text: str) -> bytes:
    """Placeholder: call the TTS provider you're evaluating and return WAV bytes."""
    raise NotImplementedError("Replace with your TTS provider's synthesis call")


def main() -> None:
    audio_dir = Path("audio")
    audio_dir.mkdir(exist_ok=True)
    results = defaultdict(lambda: [0, 0])  # entity_type -> [passed, total]

    with open("tests.csv", newline="") as f:  # columns: id, text, expected, entity_type
        for row in csv.DictReader(f):
            wav_path = audio_dir / f"{row['id']}.wav"
            if not wav_path.exists():
                wav_path.write_bytes(synthesize(row["text"]))

            transcript = transcriber.transcribe(str(wav_path))
            if transcript.status == aai.TranscriptStatus.error:
                raise RuntimeError(transcript.error)

            ok = entity_present(row["expected"], transcript.text, row["entity_type"])
            results[row["entity_type"]][0] += ok
            results[row["entity_type"]][1] += 1
            if not ok:
                print(f"FAIL {row['id']}: expected {row['expected']!r}, heard {transcript.text!r}")

    for kind, (passed, total) in sorted(results.items()):
        print(f"{kind:8s} {passed}/{total} passed")


if __name__ == "__main__":
    main()

The comparison is where most homegrown checks go wrong. Speech-to-text may write "$1,249.50" as digits or as "one thousand two hundred forty nine dollars and fifty cents," and a phone number as "415-555-0123" or "four one five…". In our testing, naive substring matching gave 3 false positives and 9 false negatives on 32 realistic transcripts. Normalize both sides first: number words to digits, money to one format, phone digits and IDs with or without hyphens, thousands separators removed. Then match on word boundaries so "$12,490.50" doesn't pass for "$1,249.50." Save this helper as normalize.py next to the script:

import re

DIGIT = {w: str(i) for i, w in enumerate("zero one two three four five six seven eight nine".split())} | {"oh": "0"}
SMALL = {w: i for i, w in enumerate("zero one two three four five six seven eight nine ten eleven twelve thirteen "
                                    "fourteen fifteen sixteen seventeen eighteen nineteen".split())}
TENS = {"twenty": 20, "thirty": 30, "forty": 40, "fifty": 50, "sixty": 60, "seventy": 70, "eighty": 80, "ninety": 90}
NUMWORDS = set(SMALL) | set(TENS) | {"hundred", "thousand", "oh"}


def _run_to_digits(run):
    if all(w in DIGIT for w in run):          # "four one five" -> "415" (digit-by-digit reading)
        return "".join(DIGIT[w] for w in run)
    total, cur = 0, 0                         # "one thousand two hundred forty nine" -> "1249"
    for w in run:
        if w in SMALL:
            cur += SMALL[w]
        elif w in TENS:
            cur += TENS[w]
        elif w == "hundred":
            cur *= 100
        elif w == "thousand":
            total, cur = total + cur * 1000, 0
    return str(total + cur)


def words_to_digits(t):
    out, run = [], []
    for w in t.split() + [""]:
        if w in NUMWORDS:
            run.append(w)
            continue
        if run:
            out.append(_run_to_digits(run))
            run = []
        if w:
            out.append(w)
    return " ".join(out)


def normalize(text):
    t = text.lower().replace("'", "")
    t = re.sub(r"(?<=\d),(?=\d{3}\b)", "", t)                 # 1,249 -> 1249
    t = re.sub(r"\$(\d+)(?:\.(\d{2}))?", lambda m: f" {m[1]} dollars {m[2] or '00'} cents ", t)
    t = re.sub(r"(\d+)\.(\d{2}) dollars", r"\1 dollars \2 cents", t)
    t = re.sub(r"(\d+)(st|nd|rd|th)\b", r"\1", t)              # 3rd -> 3
    t = re.sub(r"[^a-z0-9.]+", " ", t).replace(" . ", " ").strip(" .")
    t = re.sub(r"\.(?!\d)", " ", t)
    t = words_to_digits(t)
    t = re.sub(r"\b(\d+) dollars? (?:and )?(\d+) cents?\b", lambda m: f"{m[1]}.{int(m[2]):02d}", t)
    t = re.sub(r"\b(\d+) dollars?\b", r"\1.00", t)
    return re.sub(r"\s+", " ", t).strip()


def _alnum_runs(t):
    # join tokens of an ID or phone number read out in pieces: "w 3 1 7" / "415 555 0123"
    return re.sub(r"(?<=\d)\s(?=\d)|(?<=\b[a-z])\s(?=[a-z0-9])(?=\S*\d)", "", t)


def entity_present(expected, transcript, kind):
    e, t = normalize(expected), normalize(transcript)
    if kind in ("id", "phone"):
        e, t = e.replace(" ", "").replace(".", ""), _alnum_runs(t)
        prefix = "(?:1)?" if kind == "phone" else ""            # tolerate a leading country code
        return re.search(rf"(?<![a-z0-9]){prefix}{re.escape(e)}(?![a-z0-9])", t) is not None
    return re.search(rf"(?<![a-z0-9.]){re.escape(e)}(?![a-z0-9]|\.\d)", t) is not None

One more rule: listen to every failure. A round-trip test measures two systems at once, so some failures will be speech-to-text errors, not TTS errors. Our breakdown of why a voice agent mishears "8" as "H" covers the usual suspects. A quick listen tells you which side to blame.

Listen for the errors STT can't score

Some failures pass the round trip and still hurt the call. A strong speech-to-text model can recover a digit string read in one breathless run, but a caller writing it down can't. Add a short human review on a sample of entity clips:

• Grouping: are digits chunked the way people say them (3-3-4 for US phone numbers)?
• Pacing: is there a pause long enough to write each group down?
• Stress: does emphasis land on the right syllable of names and terms?
• Confidence: does the voice sound sure, or does it trail off on the hard part?

Score each clip pass or fail. Reviewers agree on a binary call far more often than on a 1–5 rating.

Test it like production: telephony audio, interruptions and concurrency

A clip that sounds right in a test harness can still fail on a live call. Before you commit, run these checks:

• Phone audio. Convert every clip to 8 kHz mu-law, the format most phone calls use, then rerun your MOS, A/B, and round-trip tests on the converted audio. Sibilants and fast digits are the first to suffer.
• Barge-in. Interrupt the agent mid-sentence. Playback should stop right away, the rest of the synthesis should cancel, and the next reply shouldn't carry leftover audio.
• Long calls. Generate a ten-minute conversation and listen for voice drift, volume jumps, and artifacts at the seams between streamed chunks.
• Concurrency. Run the full test set at your expected peak and compare P95 results to a single stream. Our guide to load testing a voice agent shows how to simulate realistic traffic.
• Time to first audio. Measure how long the caller waits for the first audible sound. Our TTS latency guide covers how to measure it.

Keep the test set in CI so you can catch regressions before your users do when a provider ships a new model. After launch, track the same entities in production observability.

A scorecard you can reuse

Here's the whole evaluation on one page. The pass bars are yours to set, but decide on the hard-fail rules before you look at results.

Why leaderboards miss entity accuracy

Leaderboards average. That's what makes them useful for ranking, and it's why they miss what matters to an agent. A wrong digit in one sentence out of fifty barely moves an average, but it ends that caller's trust in everything the agent said before.

Speech-to-text learned this lesson first. Teams that chose on WER alone found that speech recognition errors compound through the rest of the agent, and that the errors that matter cluster in names, numbers, and IDs. Our guide on how to evaluate speech recognition models makes the same case for the input side.

So use MOS and arenas to build a shortlist. Then decide with your own scripts, your own phone audio, and a round-trip test that counts every entity. When you're ready to compare providers side by side, our roundup of text-to-speech APIs is a good place to start the shortlist.

Frequently asked questions

How do you evaluate a text-to-speech model?

Evaluate a text-to-speech model in four layers: perceived quality (MOS), blind preference against alternatives, correctness on names, numbers, and IDs, and behavior under production conditions. For voice agents, weight correctness highest. Run every test on your own scripts and on phone-quality audio.

What is a good mean opinion score for TTS?

There's no universal good MOS for TTS, because scores depend on the raters, the sentences, and the listening setup. Telecom often treats about 4 out of 5 as toll quality, but TTS scores from different studies aren't comparable. Compare systems inside one blind test you run yourself, with human recordings as anchors.

Are TTS leaderboards and arenas enough to choose a TTS for a voice agent?

No, they're a screening tool. Public arenas rank voices on generic sentences heard through headphones, one clip at a time. They don't test order IDs, phone audio, interruptions, or long calls, so shortlist with them and decide with your own tests.

How do I test whether a TTS pronounces names, numbers, and order IDs correctly?

Use a round-trip entity accuracy test: synthesize sentences containing real entities, transcribe the audio with a pinned speech-to-text model and no keyterm hints, normalize both sides, and check each entity. Report pass rates per entity type and listen to every failure. Add a short human review for digit grouping and pacing.

What makes a TTS good for phone voice agents?

A good TTS for phone agents sounds natural in short turns, reads every name, number, and code exactly, and stays clear after conversion to 8 kHz phone audio. It also has to stop instantly when the caller interrupts and sound the same across a long call.

How do I test TTS quality at production concurrency?

Run your full test set, including the round-trip entity check, at your expected peak number of simultaneous streams. Compare P95 results and entity pass rates against a single stream. Any drop in entity accuracy or new audio artifacts under load is a hard fail.