How to load test a voice agent before you launch

Your demo was one call, on a good microphone, in a quiet room, with someone who knew what to say.

Production is eight hundred of those at once, on telephony audio compressed to 8 kHz, with a caller reading a sixteen-digit policy number from the driver's seat of a moving car. Those are different systems. The second one is the only one that matters, and it's the one nobody tests.

Here's the uncomfortable part: a pipeline that averages 800 ms with one session open and spikes to three seconds at two thousand isn't a sub-second pipeline. It's a sub-second demo. The gap between those two numbers is where launch weeks go wrong.

This is a guide to closing that gap — what to load test, how to generate the load, what to measure, and what to do when the numbers come back bad.

What load testing a voice agent actually means

Load testing a voice agent means running many simultaneous conversations against your full pipeline — speech-to-text, LLM, text-to-speech, tool calls, and telephony — and measuring whether accuracy, latency, and turn-taking hold up at the concurrency you expect in production.

That's different from a functional test, which asks whether the agent works. Load testing asks whether it still works when four hundred people are talking to it.

Three things get conflated under the same heading, and they fail differently:

• Concurrency — how many sessions are open at the same time. Failures show up as queueing, rising P95 latency, and connection refusals.
• Throughput — how many new sessions you open per minute. Failures show up as rate-limit rejections during a spike, even when total open sessions is well within capacity.
• Duration — how long sessions stay open. Failures show up hours in: memory leaks in your orchestration layer, drifting turn detection, and sessions that never get closed.

Most teams test the first one and get surprised by the second. A callback campaign that dials four hundred numbers in ninety seconds is a throughput event, not a concurrency event, and the two hit different limits.

What breaks first, roughly in order: connection establishment under burst, LLM time-to-first-token under queueing, text-to-speech synthesis backpressure, and tool-call round trips against your own backend. Speech-to-text accuracy doesn't degrade under load the way the rest of the stack does — but the audio conditions you only encounter in production absolutely do change it, which is why the corpus matters more than the harness.

Build the corpus before you build the harness

The most common load-testing mistake isn't a bad script. It's testing on clean audio and concluding you're ready.

Pull fifty of your worst calls. Not fifty representative ones — the fifty your current system already fumbles. You want:

• Telephony compression. 8 kHz mulaw, not 16 kHz studio WAV. If your agent will live on a phone line, test on phone audio.
• Entity-heavy content. Confirmation codes, account numbers, email addresses spelled letter by letter, dosages, dates, dollar amounts. These are the tokens your workflow branches on.
• The accents in your actual user base, not a generic accent set.
• Real background noise. Cars, drive-thrus, open-plan offices, TVs.
• Interruptions and false starts. People saying "no wait, sorry, it's actually —" mid-sentence.

Then audit your ground truth before you trust a single number. Human transcribers systematically drop filler words, misspell proper nouns, clean up repetitions, and translate code-switched segments into English rather than transcribing what was said. When your model transcribes more accurately than the human label, that improvement shows up in your metrics as an error.

The practical rule from our evaluation guide: before reporting word error rate, manually audit at least twenty insertions and work out what share are genuine errors versus ground-truth omissions. If you see an unexpected insertion spike on a newer model, audit the labels before concluding the model got worse. There's a Truth File Corrector in the AssemblyAI dashboard that lets you listen back and fix human transcription errors by clicking through the differences.

And weight entity accuracy over word accuracy when you score. Word error rate averages across every token in the transcript, and most tokens are function words — get "the" wrong and nothing happens. Get an account number wrong and the agent looks up a different customer and reads back a different balance, with total confidence, because the LLM downstream has no idea its input was corrupted. We've made that argument at length in word error rate is broken, and it's the single most consequential thing to get right before you scale.

The four load profiles to run

Each profile is looking for a different failure. Run all four.

Steady state at projected peak. Hold your expected busiest-hour concurrency for ten minutes. You're looking for P95 latency that stays flat rather than climbing. If P50 is still falling at minute eight, you haven't reached steady state — extend it.

Linear ramp. Climb gradually until something degrades. This finds the knee in the curve, which is the number you actually plan capacity against. Ramping matters more than people expect: submitting a large spike upfront produces worse latency than a gradual climb, because the pipeline scales ahead of gradual traffic. For async workloads our bulk transcription and load test guide recommends 15-second windows starting at 25 requests per window and growing about 8–9% per window — rate_n = ceil(25 × 1.085ⁿ). The same shape works for streaming sessions.

Spike. 10× in sixty seconds. This is the outage-callback case, the Black Friday case, the "we just got mentioned on the news" case. It tests throughput limits, not concurrency limits, and it's the profile most teams skip.

Soak. Four hours at 70% of peak. This surfaces the slow failures — file handles, memory, sessions that never got terminated, and turn detection that drifts as your orchestration layer degrades.

What to measure, and what good looks like

Report percentiles, not averages. P50, P90, P95, P99, and max. Separate ramp-phase numbers from sustain-phase numbers — expect worse latency during the ramp, and benchmark against the sustain phase.

For reference on the accuracy side, on the Pipecat open speech-to-text benchmark of real agent conversations, Universal-3.5 Pro Realtime records a 6.99% pooled word error rate and a 15.31% entity error rate, against Deepgram Flux at 15.58% / 50.50%, ElevenLabs Scribe v2 at 9.76% / 39.70%, and Google Chirp3 at 9.04% / 21.51%. On phone numbers specifically it's 3.55% versus 10.41% for Deepgram Flux. Full figures for that benchmark are in the Universal-3.5 Pro Realtime launch post. Our benchmarks page runs a separate streaming evaluation against a different dataset and reports 5.53% word error rate, a 5.38% missed entity rate, and a 335 ms median time-to-complete-turn, so don't expect the two sets of numbers to line up. Published figures of either kind are a starting point, not an answer — run your own audio.

A harness you can actually run

Here's a concurrency harness against the v3 streaming API. It opens N sessions on a stagger, paces audio at real time, loops the file so sessions outlive the ramp, and records both finalization delay per turn and the peak concurrency it actually reached.

Three details trip people up, all worth knowing before you write your own. Audio chunks must be between 50 ms and 1000 ms — anything outside that closes the session with code 3007. You can't blast the file as fast as your loop will go either: sending audio faster than real time also closes the session with 3007, so pacing isn't politeness, it's a protocol requirement. And if your corpus is 8 kHz mulaw, as it should be, you have to say so. The default encoding is pcm_s16le, so an unlabelled 800-byte mulaw chunk gets read as 50 ms of 16-bit audio instead of 100 ms of mulaw — right at the 3007 floor — and transcribed as noise. Pass encoding="pcm_mulaw" and keep that audio headerless, because Python's wave module only reads uncompressed PCM and won't open a mulaw file at all.

import os
import time
import wave
import threading
import statistics
from dataclasses import dataclass, field

from assemblyai.streaming.v3 import (
    BeginEvent,
    Encoding,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    TerminationEvent,
    TurnEvent,
)

API_KEY = os.environ["ASSEMBLYAI_API_KEY"]

# Headerless 8 kHz mulaw (.ul / .raw) for telephony audio. For 16-bit PCM,
# point this at a .wav and set ENCODING to Encoding.pcm_s16le.
AUDIO_PATH = "worst_calls/policy_number_in_car.ul"
ENCODING = Encoding.pcm_mulaw
SAMPLE_RATE = 8000

CHUNK_MS = 100               # must stay between 50 and 1000
TARGET_SESSIONS = 200
NEW_SESSIONS_PER_MIN = 80    # keep headroom under your account's limit
HOLD_SECONDS = 300           # how long each session stays open

STAGGER = 60.0 / NEW_SESSIONS_PER_MIN
LAUNCH_WINDOW = TARGET_SESSIONS * STAGGER

# Sessions have to outlive the ramp or the early ones retire before the last
# one connects, and you silently test a fraction of TARGET_SESSIONS.
assert HOLD_SECONDS > LAUNCH_WINDOW, (
    f"HOLD_SECONDS ({HOLD_SECONDS}s) must exceed the {LAUNCH_WINDOW:.0f}s "
    f"launch window to reach {TARGET_SESSIONS} concurrent sessions"
)

_live = 0
_peak = 0
_live_lock = threading.Lock()


def _track(delta):
    """Count sessions that are actually open, so we can report real peak load."""
    global _live, _peak
    with _live_lock:
        _live += delta
        _peak = max(_peak, _live)


@dataclass
class SessionResult:
    index: int
    session_id: str = ""
    finalization_ms: list = field(default_factory=list)
    close_reason: str = ""
    error: str = ""


def load_audio_chunks():
    """Read the file once and cut it into exact CHUNK_MS pieces.

    A short trailing chunk would trip the 50 ms floor, so the remainder is
    dropped rather than sent.
    """
    if ENCODING is Encoding.pcm_mulaw:
        with open(AUDIO_PATH, "rb") as f:
            audio = f.read()
        bytes_per_frame = 1
    else:
        with wave.open(AUDIO_PATH, "rb") as wav:
            if wav.getnchannels() != 1 or wav.getsampwidth() != 2:
                raise ValueError("expected 16-bit mono PCM")
            if wav.getframerate() != SAMPLE_RATE:
                raise ValueError(f"file is {wav.getframerate()} Hz, SAMPLE_RATE is {SAMPLE_RATE}")
            audio = wav.readframes(wav.getnframes())
        bytes_per_frame = 2

    chunk_bytes = int(SAMPLE_RATE * CHUNK_MS / 1000) * bytes_per_frame
    chunks = [
        audio[i:i + chunk_bytes]
        for i in range(0, len(audio) - chunk_bytes + 1, chunk_bytes)
    ]
    if not chunks:
        raise ValueError("audio file is shorter than one chunk")
    return chunks


def run_session(result: SessionResult, chunks: list):
    started = None

    def on_begin(client: RealTimeTranscriber, event: BeginEvent):
        nonlocal started
        started = time.monotonic()
        result.session_id = event.id

    def on_turn(client: RealTimeTranscriber, event: TurnEvent):
        # Only finalized turns carry a usable measurement.
        if not event.end_of_turn or not event.words or started is None:
            return
        # words[].end is milliseconds into the stream. Because we pace audio at
        # real time, wall-clock elapsed and audio time are the same clock, so
        # the difference is the finalization delay for that turn — transcription
        # plus the endpointing silence the mode preset waits out.
        audio_time_ms = event.words[-1].end
        wall_ms = (time.monotonic() - started) * 1000
        result.finalization_ms.append(wall_ms - audio_time_ms)

    def on_terminated(client: RealTimeTranscriber, event: TerminationEvent):
        result.close_reason = f"{event.session_duration_seconds}s session"

    def on_error(client: RealTimeTranscriber, error: RealTimeError):
        result.error = str(error)

    client = RealTimeTranscriber(
        RealTimeTranscriberOptions(terminate_timeout=30.0),
        api_key=API_KEY,
    )
    client.on(RealTimeEvents.Begin, on_begin)
    client.on(RealTimeEvents.Turn, on_turn)
    client.on(RealTimeEvents.Termination, on_terminated)
    client.on(RealTimeEvents.Error, on_error)

    try:
        client.connect(
            RealTimeParameters(
                sample_rate=SAMPLE_RATE,
                encoding=ENCODING,        # default is pcm_s16le — say so for mulaw
                speech_model="universal-3-5-pro",
                mode="balanced",          # min_latency | balanced | max_accuracy
            )
        )
    except Exception as exc:
        # A refused connection is what a spike test is looking for, so record
        # it rather than letting the thread die silently.
        result.error = f"connect failed: {exc}"
        return

    _track(+1)
    try:
        deadline = time.monotonic() + HOLD_SECONDS
        next_send = time.monotonic()
        i = 0
        while time.monotonic() < deadline:
            # Loop the file so the session stays open for the whole hold window.
            client.stream(chunks[i % len(chunks)])
            i += 1
            # Pace at real time. Faster than real time closes the session.
            next_send += CHUNK_MS / 1000
            time.sleep(max(0.0, next_send - time.monotonic()))
    finally:
        _track(-1)
        # Terminating finalizes the open turn AND stops billing.
        client.disconnect(terminate=True)


def load_test():
    chunks = load_audio_chunks()
    results = [SessionResult(index=i) for i in range(TARGET_SESSIONS)]
    threads = []

    for result in results:
        t = threading.Thread(target=run_session, args=(result, chunks), daemon=True)
        t.start()
        threads.append(t)
        time.sleep(STAGGER)   # respect the new-sessions-per-minute limit

    for t in threads:
        t.join()

    delays = sorted(d for r in results for d in r.finalization_ms)
    errors = [r for r in results if r.error]

    def pct(p):
        if not delays:
            return float("nan")
        return delays[min(int(len(delays) * p), len(delays) - 1)]

    print(f"sessions opened:     {len(results)}")
    print(f"peak concurrent:     {_peak} (target {TARGET_SESSIONS})")
    print(f"errored sessions:    {len(errors)}")
    print(f"finalized turns:     {len(delays)}")
    if delays:
        print(f"finalization P50:    {statistics.median(delays):.0f} ms")
        print(f"finalization P95:    {pct(0.95):.0f} ms")
        print(f"finalization P99:    {pct(0.99):.0f} ms")
    else:
        print("finalization:        no finalized turns")
    for r in errors[:5]:
        print(f"  session {r.index}: {r.error}")


if __name__ == "__main__":
    load_test()

One thing in that script is worth internalizing beyond the script: HOLD_SECONDS has to exceed the launch window, or the test quietly lies to you. At 200 sessions and 80 new sessions per minute you spend 150 seconds opening them all — so if each session hangs up at the end of a 30-second call, the early ones are long gone before the last one connects and you never see more than about 40 open at once. Looping the audio to a deadline is what makes the target real, and printing peak concurrent is what proves it. A run that reports 200 sessions opened and 40 peak concurrent has tested a fifth of what you think it tested.

Swap mode to max_accuracy and re-run to see the accuracy-latency tradeoff on your own audio rather than in the abstract. balanced is the default and the right starting point for conversational agents.

Concurrency: what your provider actually gives you

This is the constraint that shows up last and hurts most, so get the answer in writing from every vendor before the pilot ends.

AssemblyAI's streaming API has no hard cap on total open sessions. The constraint is on new sessions per minute — 5 for free accounts, 100+ for paid — and it auto-scales: any time you're using 70% or more of your current limit, the ceiling rises 10% for the next minute. Max out for five straight minutes and you're at 146 new sessions per minute with 610 streams open, with no ceiling on where that goes. Below 50% utilization it scales back toward your starting limit, which is worth knowing before a spike test after a quiet week.

Exceed the limit and you get a WebSocket close with code 3009 and the message Unauthorized Connection: Too many concurrent sessions. (The rate limits page lists the same message under close code 1008, so handle both.) If you're seeing that unexpectedly, check that every session is being terminated — unclosed sessions keep counting against you. Full detail is in the streaming rate limits docs.

For async workloads the shape is different: 200 parallel jobs by default, plus an HTTP ceiling of 20,000 requests per 5 minutes across all endpoints, polling included. Blow through that and you get a 403.

The bill is part of the test

Here's the thing almost nobody load tests: the invoice.

Streaming and Voice Agent sessions are billed on how long the WebSocket stays open, not on how much audio you send. Idle time is billable. A streaming session you forget to close auto-closes after three hours — and bills for the full three hours. Improperly closed sessions are the single most common cause of surprise charges.

So a load test that leaks sessions doesn't just skew your latency numbers, it produces a bill that looks exactly like the 10× cost spike you were trying to rule out. Two habits fix it:

• Always send an explicit termination. For streaming that's {"type": "Terminate"}; the SDK's disconnect(terminate=True) does it for you.
• For the Voice Agent API, send session.end on any intentional disconnect. If you just close the socket, the server holds the session open for a 30-second resume window — and that window is billable. On a thousand-call test, that's over eight hours of billed time you didn't use.

While you're there, work out cost per completed task rather than cost per hour. An agent that gets 15% cheaper per hour while its task completion drops 20% has gotten more expensive, and the per-hour number will never show you that.

Three ways to generate the load

Agent-vs-agent simulation. A second AI drives a scripted or improvised caller. Fast to set up, scales trivially, and catches logic and tool-calling failures. It misses real acoustics almost entirely — synthesized speech is cleaner than any human on a phone, so it will make your speech-to-text look better than it is.

Replayed production audio. Your own recordings, streamed back through the pipeline at real time. Best signal by a distance, because the audio is exactly what you'll get in production. Requires a corpus and consent to reuse recordings.

SIP-level call generators. Real calls over real telephony. Closest to reality, most setup, and the only way to test carrier-side behavior and DTMF handling.

Most teams end up using two: replayed audio for accuracy, simulation for conversation logic and volume. If you'd rather not build the harness at all, BlueJay, Coval, and Hamming all specialize in this, and they plug into the same pipeline you're testing. Whichever route you take, our developer experience checklist covers what to judge while you're in there.

"We were searching for the best realtime ASR model for our voice agent pipeline in Fireflies. The new Universal 3.5 Pro speech model from Assembly is best so far in terms of accuracy, latency and language switching."

— Foysal Osmany, Software Engineer at Fireflies

Set the rollback plan before the first real call

Nobody writes this section, and it's the one you'll want at 2am.

Pin your model. If you omit speech_model, streaming sessions default to universal-3-5-pro and you float onto whatever the latest flagship becomes. That's the right default for most teams and the wrong one during a launch window. Pin it explicitly — and know what you'd pin it to if you had to move. The streaming API exposes model IDs, not dated snapshots, so your fallback isn't an earlier build of the same model, it's a different model: universal-streaming-english or universal-streaming-multilingual. Benchmark whichever one you'd fall back to on your own corpus now, not during an incident.

Know your reconnect behavior. Voice Agent sessions survive a dropped connection: capture session_id from session.ready, and on reconnect send session.resume as the first message within 30 seconds to pick up with context intact. Past that window you get session_not_found and start fresh.

import json

# On reconnect, resume before doing anything else.
if session_id:
    await ws.send(json.dumps({"type": "session.resume", "session_id": session_id}))
else:
    await ws.send(json.dumps({"type": "session.update", "session": {...}}))

async for raw in ws:
    event = json.loads(raw)
    if event["type"] == "session.ready":
        session_id = event["session_id"]          # save for the next reconnect
    elif event["type"] == "session.error" and event["code"] in (
        "session_not_found", "session_forbidden",
    ):
        session_id = None                          # expired — start clean

Know which errors are retryable. On the Voice Agent API, at_capacity, concurrency_exceeded, and internal_error are the only retryable codes. Everything else is fatal — retrying an invalid_config in a loop just burns your rate limit. Build the retry table into your client before launch, not after the first incident.

Decide your rollback trigger in advance. Pick the metric and the threshold now, while nobody's shouting. "P95 finalization delay above 800 ms for five consecutive minutes" is a rollback trigger. "It feels slow" is not.

A one-week plan

• Monday. Assemble the corpus. Fifty worst calls, telephony-encoded, ground truth audited.
• Tuesday. Pilot at 10% of target. Verify the harness measures what you think it measures — check that peak concurrent actually reaches your target — and that your metrics pipeline is writing durably.
• Wednesday. Steady state and ramp. Find the knee.
• Thursday. Spike and soak. Run the soak overnight.
• Friday morning. Rollback drill. Kill a model version mid-test and time how long recovery actually takes.
• Friday afternoon. Write up percentiles by profile, set the alert thresholds you'll launch with, and reconcile the bill against what you expected to spend.

If you're planning to exceed 200 requests per minute on async, running a large one-time batch, or testing in the EU region, email support first — AssemblyAI can pre-scale for the run and monitor it live, at no extra cost. And if you want to know where these stacks tend to hit their limits once they're live, we mapped that in the production ceiling.

The number that predicts production

Every vendor will quote you a median word error rate on clean audio. It is the least useful number in the entire evaluation, and it is the one on every landing page.

Ask instead for P95 entity error rate at your concurrency, on your audio, with a tool call in the middle of the turn. Watch which vendors can produce it and which change the subject back to WER. That single question sorts the field faster than a month of demos — and once you've built the harness above, you can answer it for yourself rather than taking anyone's word for it.

Your agent's ceiling isn't set by how good it is on a good day. It's set by what it does on the worst call of the week, and now you know how to find out what that is before your customers do.

Frequently asked questions

How can I test my AI agent?

Test in three layers. Functional tests confirm the agent does the right thing on a happy path. Evaluation runs your own recorded audio through the pipeline and scores accuracy, latency, and task completion. Load testing runs many concurrent sessions to confirm all of that holds at production volume. Most teams do the first, some do the second, and the third is where launch failures come from.

How much does a voice agent cost?

It depends on architecture. A do-it-yourself stack bills separately for speech-to-text, LLM tokens, and text-to-speech, which makes forecasting hard. AssemblyAI's Voice Agent API is a flat hourly rate covering all three through one WebSocket, billed per second on session duration — see the pricing page for current rates. Whichever you use, model cost per completed task rather than cost per hour, and remember that idle connection time is billable.

How many concurrent calls should I load test for?

Start at your projected busiest hour, then test 3× that. Peak traffic for voice agents is rarely smooth — a callback campaign or a service outage produces a burst several times your average peak. Also test throughput separately from concurrency: opening 400 sessions in 60 seconds hits a different limit than holding 400 sessions open. And instrument the concurrency you actually reached, rather than the number you configured — if sessions retire faster than you open them, the two aren't the same.

What's a good P95 latency for a production voice agent?

Under a second end-to-end is where conversation stops feeling like a bad phone connection. Work backward from there: finalization delay under 500 ms at P95, LLM time-to-first-token in the low hundreds, and text-to-speech time-to-first-byte under 300 ms. Measure P95 under load, not median in isolation — the median on one open session tells you nothing about production.

How do I benchmark voice agent providers on my own call data?

Record fifty representative calls including your hardest audio, audit the ground truth for human transcription errors, then run the same files through each provider with matched settings. Score entity error rate separately from word error rate, measure finalization delay per turn, and run a live side-by-side on real conversations too — turn-taking quality depends on the caller's device and environment in ways a file replay can't capture. The voice agent evaluation guide walks through the full methodology.

What's the rollback plan if a new model version underperforms in production?

Pin the model explicitly rather than floating on the latest, and pick your fallback ahead of time. The streaming API exposes model IDs rather than dated snapshots, so a rollback means moving to a different model — validate that one on your own corpus before you need it. Then decide your rollback trigger — a specific metric crossing a specific threshold for a specific duration — before you launch, and rehearse it. A rollback plan you've never executed is a hypothesis, not a plan.

‍