How to choose a voice for your AI voice agent

Most teams pick their agent's voice the same way: open the voice library, play a dozen samples on good headphones, and choose the one that sounds best reading "Hello, how can I help you today?" Then the agent goes live on a phone line, reads a confirmation code to a caller in a noisy car, and the voice that won the demo turns out to be the wrong one.

Choosing a voice for an AI voice agent is a product decision, not a taste test. The right voice suits the caller, carries your brand, holds up on an 8 kHz phone line, and speaks your callers' language natively. This guide walks through each of those, then gives you a five-step process and a small script to test a shortlist the way your callers will actually hear it.

It assumes you've already picked a text-to-speech provider. If you haven't, start with our roundup of the top text-to-speech APIs. The examples use AssemblyAI-TTS-1, which offers thousands of voices across six languages.

Start from the caller and the brand, not the voice library

A voice library is a menu, and menus invite browsing. Before you open it, write down two things.

Who's calling, and in what state? A salon-booking line, a debt-collection line and a pizza-ordering line have different callers in different moods. Someone calling about a billing dispute wants calm and steady. Someone booking a table wants quick and friendly. Someone in a hurry wants the agent to get out of the way.

If you want a refresher on what makes one voice different from another (age, accent, timbre, style), what is text-to-speech covers it.

What does your brand sound like? If your support emails are warm and casual, a formal, measured voice will feel like a different company. If you sell to enterprises, a bubbly voice can undercut the conversation.

Turn those into a one-line persona before you listen to anything: "a calm, middle-aged voice with an even pace, for billing questions" is a brief you can shortlist against. "Something that sounds good" isn't.

One honest caveat: the voice is one part of the experience, and not always the biggest. Get the voice good enough for the line and the brand, then spend your effort on latency, the script and turn-taking, where a lot of the call is decided. Writing replies the way people talk on a call covers the script side, and it's worth as much attention as the voice.

Accent: match your callers or contrast on purpose

Accent is the first filter most teams should apply, and it's mostly a question of who's calling.

• Match your callers by default. A US insurance line serving US callers should usually sound American. An Australian utility should usually sound Australian. Matching reduces the mental work of listening, which matters more on a compressed phone line.
• Contrast on purpose, if at all. A British voice for a premium brand in the US can work as a deliberate brand choice. Make it a decision, not an accident of what sounded nice in the demo.
• Don't guess from the name. Each voice in the AssemblyAI catalog lists its language and accent. Filter on that, not on the voice's name.

The starter voices in the docs cover several English accents, plus Spanish, German, French and Portuguese:

These are starting points from the voices docs. The voice catalog in the dashboard holds thousands more, with descriptions, and a Copy voice ID button that gives you the voice_id to put in code.

Pace, pitch and energy for short turns

Voice agents speak in short turns: a question, a confirmation, a one-line answer. Qualities that are pleasant in a long narration can be tiring or slow in a conversation.

• Pace. For transactional lines (order status, appointment confirmations), a fairly fast natural pace keeps calls moving. For anything where the caller has to absorb details, such as amounts, dates or instructions, an even, steady pace is safer. The catalog describes each voice's pace, so you can shortlist on it.
• Pitch. Very high or very low voices can lose clarity on a phone line, where the audio is band-limited. A mid-range voice usually survives compression best. Test before you rule anything in or out.
• Energy. Energetic delivery suits sales and booking. Calm delivery suits support, billing and anything stressful. A voice that sounds upbeat reading "Your payment failed" sounds wrong.

Two per-voice behaviors are worth listening for specifically, because they vary from voice to voice:

1. Disfluencies. If your agent uses fillers like "uh" or false starts to sound natural, some voices render them convincingly and others sound clipped on the same text.
2. Inflection. Some voices lift the end of statements more than others, which can make a confirmation sound like a question.

Here's how starter voices map to common personas, based on the catalog's own descriptions:

Language: pick a voice recorded in the language you speak

AssemblyAI-TTS-1 speaks every voice in all six of its languages: English, Spanish, French, German, Italian and Portuguese. Set language and an English voice will read Spanish text. That's useful, but it isn't the best result.

Each voice was recorded in one language, and the docs are direct about it: a voice usually sounds best in its own language. For a Spanish-language agent, pick a voice recorded in Spanish, such as Constanza for Mexican Spanish, and set language to match the text you send. The API doesn't detect the language from the text, so set it explicitly.

Two defaults to know:

• If you leave language out, a catalog voice speaks its own language, and a built-in voice such as aai-jane speaks English.
• If you leave voice_id out, you get aai-jane, the default voice. Set the voice explicitly so a default never ships by accident.

Setting both looks like this in Python:

import os

import assemblyai as aai

config = aai.TTSConfig(voice_id="aai-constanza", language="spanish")

with aai.TTSSession(config, api_key=os.environ["ASSEMBLYAI_API_KEY"]) as session:
    with open("hola.pcm", "wb") as out:
        for chunk in session.speak("Hola, gracias por llamar. ¿En qué le puedo ayudar?"):
            out.write(chunk)  # raw 24 kHz 16-bit mono PCM

And in JavaScript:

import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

// A session for a live call: camelCase options.
const session = client.tts.session({ voiceId: "aai-kyla", language: "english" });
const begin = await session.connect();
console.log("speaking with", begin.configuration.voice_id);
await session.close();

// A single clip over HTTP: the API's own field names.
const clip = await client.tts.synthesize({
  text: "Thanks for calling. How can I help you today?",
  voice_id: "aai-kyla",
  container: "wav",
});

For agents that serve callers in more than one language, choose one voice per language rather than one voice for all of them. For the full session walkthrough, see how to stream text to speech in Python over a WebSocket.

Test the shortlist on a real phone line

This is the step most teams skip, and it's the one that changes decisions. Phone calls carry audio at 8 kHz, usually as G.711 μ-law in North America and Japan, or A-law in most other countries. That's a third of the 24 kHz audio you hear in a browser playground. Breathiness, high frequencies and some consonants get lost. A voice that sparkles on headphones can turn muddy on a call.

So test your shortlist in the format your callers will hear, with the lines your agent will actually say. This script renders the same lines with each shortlisted voice as 8 kHz μ-law WAV files you can play back, or better, play into a real call:

import os

import assemblyai as aai

SHORTLIST = ["aai-kyla", "aai-audrey", "aai-sara", "aai-hailey"]

# Use your agent's real lines, including the hard ones.
LINES = {
    "greeting": "Hi, um, thanks for calling. What can I help you with today?",
    "confirm": "So your card ending in 4-3-2-1 will be charged $1,248.50. Is that the right one?",
    "code": "Your confirmation code is B-7-Q-4-2. That's B as in bravo.",
    "bad_news": "I'm sorry, it looks like that appointment is no longer available.",
}

tts = aai.TextToSpeech(api_key=os.environ["ASSEMBLYAI_API_KEY"])

for voice_id in SHORTLIST:
    config = aai.TTSConfig(
        voice_id=voice_id,
        language="english",
        encoding=aai.TTSEncoding.pcm_mulaw,  # use pcm_alaw for A-law lines
        sample_rate=8000,
        container=aai.TTSContainer.wav,
    )
    for name, text in LINES.items():
        response = tts.synthesize(text, config)
        with open(f"{voice_id}_{name}.wav", "wb") as f:
            f.write(response.audio)
        print(voice_id, name, response.audio_duration_ms, "ms")

When you listen, score each voice on the things that matter on a call:

• Intelligibility of the details. Can you catch every digit of the code and the amount on the first listen?
• Tone on bad news. Does the voice sound appropriately serious, or cheerful?
• The opener. Does the greeting sound like a person who picked up the phone?
• Fatigue. Play all four lines back to back. Would you still like this voice at the end of a five-minute call?

For a full scoring method, including blind comparisons and entity-reading tests, see how to evaluate TTS for voice agents. In production, stream audio to the phone line in the same format with encoding="pcm_mulaw" and sample_rate=8000 on the session, so there's no conversion step on your side.

One more practical note: the built-in voices are always loaded, while other catalog voices load the first time a session uses them. Open the TTS session when the call starts, before your first reply is ready, so a less-used voice is ready by the time the agent speaks. TTS latency explained covers the rest of what affects the wait before the first audio.

Library voice or cloned voice?

Sometimes no library voice fits, usually because the brand already has a recognizable voice: a founder, a spokesperson, a well-known support lead. That's what voice cloning is for.

With AssemblyAI, you create a cloned voice in the dashboard from a short recording of one person speaking, ten to thirty seconds, and only with that speaker's consent. Your account also needs to accept the biometric data terms. A cloned voice is private to your account, speaks all six languages (best in its own), and can be used for 60 days after its consent was last given, unless you renew the consent.

The voice cloning docs cover recording tips. The short version: record somewhere quiet with little echo, at a natural pace.

Choose a library voice when:

• you need a voice live this week;
• no single person's voice is part of your brand;
• you want to avoid managing consent renewals.

Choose a cloned voice when:

• a specific, consenting person's voice is part of your brand;
• you've tested the clone on a phone line, as above, and it holds up as well as your best library option.

A five-step shortlist process

Put it together and choosing a voice takes an afternoon, not a sprint:

1. Write the persona. One line covering caller, mood and brand, before you open the library.
2. Filter the catalog. Accent and language first, then pace and energy. Aim for four to six voices.
3. Write a test script. Four or five of your agent's real lines, including a greeting, a number or code, and a piece of bad news.
4. Render at 8 kHz and listen. Use the script above, ideally played into a real call, and score intelligibility, tone and fatigue.
5. Ship one, and revisit. Pick the best, set its voice_id explicitly in code, and listen to real calls after launch. If callers keep asking the agent to repeat itself, the voice is the first thing to re-test.

Here's the checklist as a table you can copy into your planning doc:

The voice is the first thing a caller hears and the last thing most teams test properly. Test it the way it'll be heard, and it stops being a guess. If you're building the rest of the agent too, streaming speech-to-text and an LLM included, start with AssemblyAI for voice agents.

Frequently Asked Questions (FAQs)

How do I choose a voice for my AI phone agent?

Start from a one-line persona that describes the caller, their mood and your brand, then filter the voice library on accent, language, pace and energy. Render four to six shortlisted voices reading your agent's real lines at 8 kHz μ-law or A-law, the format phone calls use, and pick the one that keeps codes, amounts and tone clear.

What makes a good voice for a voice agent?

A good voice agent voice fits the caller's situation, matches the brand, stays intelligible on a compressed phone line, and suits short conversational turns. For most support and scheduling lines, that means a clear, mid-range voice with an even or fairly fast pace, recorded in the language the agent speaks.

Should my voice agent's accent match my callers?

Usually, yes. Matching your callers' accent makes the agent easier to understand, especially on 8 kHz phone audio. A contrasting accent can work as a deliberate brand choice, but decide it on purpose rather than picking whatever sounded best in a demo.

Can one voice speak several languages?

With AssemblyAI TTS, yes: AssemblyAI-TTS-1 speaks every voice in English, Spanish, French, German, Italian and Portuguese when you set language. A voice usually sounds best in the language it was recorded in, so for a multilingual agent, choose one voice per language.

Should I use a cloned voice for my voice agent?

Use a cloned voice when a specific, consenting person's voice is part of your brand. AssemblyAI creates a clone in the dashboard from ten to thirty seconds of speech, with the speaker's consent and your account's acceptance of the biometric data terms. Otherwise, a library voice is faster to ship and needs no consent renewals.

How many voices does AssemblyAI text-to-speech offer?

AssemblyAI's voice catalog holds thousands of voices across six languages, each labeled with its language, accent and a short description. You can browse and listen in the dashboard, copy a voice's ID, and pass it as voice_id in the Python SDK or voiceId on a JavaScript SDK session.