How to load test a voice agent before launch (plus a production-readiness checklist)

To load test a voice agent, replay your hardest real calls through the full pipeline at your projected peak concurrency, then measure entity accuracy, P95 latency, rejected connections and cost per completed task. On AssemblyAI's Voice Agent API and streaming API, sessions bill on connected time and capacity is governed per account. Streaming limits how many new sessions you open per minute and, on paid accounts, auto-scales that limit. The Voice Agent API limits concurrent sessions. So the test also has to prove that sessions close cleanly and that your ramp stays inside both limits.

Your demo was one call, on a good microphone, in a quiet room, with someone who knew what to say.

Production is eight hundred of those at once, on telephony audio compressed to 8 kHz, with a caller reading a sixteen-digit policy number from the driver's seat of a moving car. Those are different systems. The second one is the only one that matters, and it's the one nobody tests.

Here's the uncomfortable part: a pipeline that averages 800 ms with one session open and spikes to three seconds at two thousand isn't a sub-second pipeline. It's a sub-second demo. The gap between those two numbers is where launch weeks go wrong.

This is a guide to closing that gap — what to load test, how to generate the load, what to measure, and what to do when the numbers come back bad. It ends with a production-readiness checklist you can hand to whoever signs off on launch.

What load testing a voice agent actually means

Load testing a voice agent means running many simultaneous conversations against your full pipeline — speech-to-text, LLM, text-to-speech, tool calls, and telephony — and measuring whether accuracy, latency, and turn-taking hold up at the concurrency you expect in production.

That's different from a functional test, which asks whether the agent works. Load testing asks whether it still works when four hundred people are talking to it.

Three things get conflated under the same heading, and they fail differently:

• Concurrency — how many sessions are open at the same time. Failures show up as queueing, rising P95 latency, and connection refusals.
• Throughput — how many new sessions you open per minute. Failures show up as rate-limit rejections during a spike, even when total open sessions is well within capacity.
• Duration — how long sessions stay open. Failures show up hours in: memory leaks in your orchestration layer, drifting turn detection, and sessions that never get closed.

Most teams test the first one and get surprised by the second. A callback campaign that dials four hundred numbers in ninety seconds is a throughput event, not a concurrency event, and the two hit different limits.

What breaks first, roughly in order: connection establishment under burst, LLM time-to-first-token under queueing, text-to-speech synthesis backpressure, and tool-call round trips against your own backend. Speech-to-text accuracy doesn't degrade under load the way the rest of the stack does — but the audio conditions you only encounter in production absolutely do change it, which is why the corpus matters more than the harness.

Build the corpus before you build the harness

The most common load-testing mistake isn't a bad script. It's testing on clean audio and concluding you're ready.

Pull fifty of your worst calls. Not fifty representative ones — the fifty your current system already fumbles. You want:

• Telephony compression. 8 kHz mulaw, not 16 kHz studio WAV. If your agent will live on a phone line, test on phone audio.
• Entity-heavy content. Confirmation codes, account numbers, email addresses spelled letter by letter, dosages, dates, dollar amounts. These are the tokens your workflow branches on.
• The accents in your actual user base, not a generic accent set.
• Real background noise. Cars, drive-thrus, open-plan offices, TVs.
• Interruptions and false starts. People saying "no wait, sorry, it's actually —" mid-sentence.

Then audit your ground truth before you trust a single number. Human transcribers systematically drop filler words, misspell proper nouns, clean up repetitions, and translate code-switched segments into English rather than transcribing what was said. When your model transcribes more accurately than the human label, that improvement shows up in your metrics as an error.

The practical rule from our voice agent evaluation guide: before reporting word error rate, manually audit at least twenty insertions and work out what share are genuine errors versus ground-truth omissions. If you see an unexpected insertion spike on a newer model, audit the labels before concluding the model got worse. There's a Truth File Corrector in the AssemblyAI dashboard that lets you listen back and fix human transcription errors by clicking through the differences.

And weight entity accuracy over word accuracy when you score. Word error rate averages across every token in the transcript, and most tokens are function words — get "the" wrong and nothing happens. Get an account number wrong and the agent looks up a different customer and reads back a different balance, with total confidence, because the LLM downstream has no idea its input was corrupted. We've made that argument at length in word error rate is broken, and it's the single most consequential thing to get right before you scale.

The four load profiles to run

Each profile is looking for a different failure. Run all four.

Steady state at projected peak. Hold your expected busiest-hour concurrency for ten minutes. You're looking for P95 latency that stays flat rather than climbing. If P50 is still falling at minute eight, you haven't reached steady state — extend it.

Linear ramp. Climb gradually until something degrades. This finds the knee in the curve, which is the number you actually plan capacity against. Ramping matters more than people expect: submitting a large spike upfront produces worse latency than a gradual climb, because the pipeline scales ahead of gradual traffic. For async workloads our bulk transcription and load test guide recommends 15-second windows starting at 25 requests per window and growing about 8–9% per window — rate_n = ceil(25 × 1.085ⁿ). The same shape works for streaming sessions.

Spike. 10× in sixty seconds. This is the outage-callback case, the Black Friday case, the "we just got mentioned on the news" case. It tests throughput limits, not concurrency limits, and it's the profile most teams skip. Better still, replay the arrival pattern of your real busiest hour instead of a flat rate: real mornings arrive in clumps, and clumps are what trip a per-minute limit.

Soak. Four hours at 70% of peak. This surfaces the slow failures — file handles, memory, sessions that never got terminated, and turn detection that drifts as your orchestration layer degrades.

What to measure, and what good looks like

Report percentiles, not averages. P50, P90, P95, P99, and max. Separate ramp-phase numbers from sustain-phase numbers — expect worse latency during the ramp, and benchmark against the sustain phase.

Be careful with the LLM-style latency metrics. For speech-to-text, time to first token is easy to game and tells you little about when your agent can actually respond. Finalization delay is the speech-to-text number that gates the turn.

For reference on the accuracy side: on AssemblyAI's English voice-agent benchmark of 12,460 scripted voice-agent scenarios, Universal-3.6 Pro Realtime records a 5.19% word error rate and a 14.4% entity error rate, against Deepgram Flux EN at 13.50% / 30.1%, ElevenLabs Scribe v2 at 7.78% / 18.5%, and Deepgram Nova-3 at 8.64% / 26.1%. On phone numbers specifically it's 2.4% versus 11.5% for Deepgram Flux EN. The method is in the Universal-3.6 Pro Realtime research post, and every system's row is on the benchmarks page. On Pipecat's independent open STT benchmark of real agent conversations, it posts a 0.80% pooled semantic word error rate, with the final transcript a median 307 ms after the speaker stops. Pipecat notes that gaps of less than about a quarter of a point between the most accurate services aren't meaningful. Published figures of either kind are a starting point, not an answer — run your own audio.

A harness you can actually run

Here's a concurrency harness against AssemblyAI's v3 streaming speech-to-text API. It opens N sessions on a stagger, waits for the server to confirm each one, paces audio at real time, loops the file so sessions outlive the ramp, and records finalization delay per turn, the peak concurrency it actually reached, and every rejection with its code and text.

Three details trip people up, all worth knowing before you write your own. Audio chunks must be between 50 ms and 1000 ms — anything outside that closes the session with code 3007. You can't blast the file as fast as your loop will go either: sending audio faster than real time also closes the session with 3007, so pacing isn't politeness, it's a protocol requirement. And if your corpus is 8 kHz mulaw, as it should be, you have to say so. The default encoding is pcm_s16le, so an unlabelled 800-byte mulaw chunk gets read as 50 ms of 16-bit audio instead of 100 ms of mulaw — right at the 3007 floor — and transcribed as noise. Pass encoding="pcm_mulaw" and keep that audio headerless, because Python's wave module only reads uncompressed PCM and won't open a mulaw file at all.

import time
import wave
import threading
import statistics
from dataclasses import dataclass, field

from assemblyai.streaming.v3 import (
    BeginEvent,
    Encoding,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    TerminationEvent,
    TurnEvent,
)

API_KEY = ""

# Headerless 8 kHz mulaw (.ul / .raw) for telephony audio. For 16-bit PCM,
# point this at a .wav and set ENCODING to Encoding.pcm_s16le.
AUDIO_PATH = "worst_calls/policy_number_in_car.ul"
ENCODING = Encoding.pcm_mulaw
SAMPLE_RATE = 8000
SPEECH_MODEL = "universal-3-6-pro"  # pin it; omitting it tracks the newest flagship

CHUNK_MS = 100               # must stay between 50 and 1000
TARGET_SESSIONS = 200
NEW_SESSIONS_PER_MIN = 80    # keep headroom under your account's limit
HOLD_SECONDS = 300           # how long each session stays open
BEGIN_TIMEOUT = 10           # seconds to wait for the server's Begin event

STAGGER = 60.0 / NEW_SESSIONS_PER_MIN
LAUNCH_WINDOW = TARGET_SESSIONS * STAGGER

# Sessions have to outlive the ramp or the early ones retire before the last
# one connects, and you silently test a fraction of TARGET_SESSIONS.
assert HOLD_SECONDS > LAUNCH_WINDOW, (
    f"HOLD_SECONDS ({HOLD_SECONDS}s) must exceed the {LAUNCH_WINDOW:.0f}s "
    f"launch window to reach {TARGET_SESSIONS} concurrent sessions"
)

_live = 0
_peak = 0
_live_lock = threading.Lock()


def _track(delta):
    """Count sessions the server actually accepted, so peak load is real."""
    global _live, _peak
    with _live_lock:
        _live += delta
        _peak = max(_peak, _live)


@dataclass
class SessionResult:
    index: int
    session_id: str = ""
    began: bool = False
    finalization_ms: list = field(default_factory=list)
    error_code: object = None
    error: str = ""
    session_seconds: float = 0.0


def load_audio_chunks():
    """Read the file once and cut it into exact CHUNK_MS pieces.

    A short trailing chunk would trip the 50 ms floor, so the remainder is
    dropped rather than sent.
    """
    if ENCODING is Encoding.pcm_mulaw:
        with open(AUDIO_PATH, "rb") as f:
            audio = f.read()
        bytes_per_frame = 1
    else:
        with wave.open(AUDIO_PATH, "rb") as wav:
            if wav.getnchannels() != 1 or wav.getsampwidth() != 2:
                raise ValueError("expected 16-bit mono PCM")
            if wav.getframerate() != SAMPLE_RATE:
                raise ValueError(f"file is {wav.getframerate()} Hz, SAMPLE_RATE is {SAMPLE_RATE}")
            audio = wav.readframes(wav.getnframes())
        bytes_per_frame = 2

    chunk_bytes = int(SAMPLE_RATE * CHUNK_MS / 1000) * bytes_per_frame
    chunks = [
        audio[i:i + chunk_bytes]
        for i in range(0, len(audio) - chunk_bytes + 1, chunk_bytes)
    ]
    if not chunks:
        raise ValueError("audio file is shorter than one chunk")
    return chunks


def run_session(result: SessionResult, chunks: list):
    begun = threading.Event()
    failed = threading.Event()
    stream_start = None

    def on_begin(client: RealTimeTranscriber, event: BeginEvent):
        result.session_id = event.id
        result.began = True
        _track(+1)            # only now is the session really open
        begun.set()

    def on_turn(client: RealTimeTranscriber, event: TurnEvent):
        # Only finalized turns carry a usable measurement.
        if not event.end_of_turn or not event.words or stream_start is None:
            return
        # words[].end is milliseconds of audio since the first chunk. Audio is
        # paced at real time from stream_start, so wall-clock elapsed minus
        # audio time is the finalization delay for the turn: transcription
        # plus the endpointing silence the mode waits out.
        audio_time_ms = event.words[-1].end
        wall_ms = (time.monotonic() - stream_start) * 1000
        result.finalization_ms.append(wall_ms - audio_time_ms)

    def on_terminated(client: RealTimeTranscriber, event: TerminationEvent):
        result.session_seconds = event.session_duration_seconds or 0.0

    def on_error(client: RealTimeTranscriber, error: RealTimeError):
        # connect() never raises on a rejected session: rejections arrive
        # here. Record the code and text; a load test counts them, it
        # doesn't retry them.
        result.error_code = error.code
        result.error = str(error)
        failed.set()
        begun.set()           # unblock the wait below

    client = RealTimeTranscriber(
        RealTimeTranscriberOptions(terminate_timeout=30.0),
        api_key=API_KEY,
    )
    client.on(RealTimeEvents.Begin, on_begin)
    client.on(RealTimeEvents.Turn, on_turn)
    client.on(RealTimeEvents.Termination, on_terminated)
    client.on(RealTimeEvents.Error, on_error)

    client.connect(
        RealTimeParameters(
            sample_rate=SAMPLE_RATE,
            encoding=ENCODING,        # default is pcm_s16le, so say so for mulaw
            speech_model=SPEECH_MODEL,
            mode="balanced",          # min_latency | balanced | max_accuracy
        )
    )

    # Don't send audio until the server says the session exists.
    if not begun.wait(BEGIN_TIMEOUT) or failed.is_set():
        if not result.error:
            result.error = f"no Begin within {BEGIN_TIMEOUT}s"
        client.disconnect(terminate=True)
        return

    try:
        stream_start = time.monotonic()
        deadline = stream_start + HOLD_SECONDS
        i = 0
        while time.monotonic() < deadline and not failed.is_set():
            # Sleep, then send: chunk i is only sent once its audio has
            # "happened", so the stream can never run faster than real time.
            next_send = stream_start + (i + 1) * CHUNK_MS / 1000
            time.sleep(max(0.0, next_send - time.monotonic()))
            # Loop the file so the session stays open for the whole hold window.
            client.stream(chunks[i % len(chunks)])
            i += 1
    finally:
        _track(-1)
        # Terminating finalizes the open turn AND stops billing.
        client.disconnect(terminate=True)


def load_test():
    chunks = load_audio_chunks()
    results = [SessionResult(index=i) for i in range(TARGET_SESSIONS)]
    threads = []

    for result in results:
        t = threading.Thread(target=run_session, args=(result, chunks), daemon=True)
        t.start()
        threads.append(t)
        time.sleep(STAGGER)   # respect the new-sessions-per-minute limit

    for t in threads:
        t.join()

    delays = sorted(d for r in results for d in r.finalization_ms)
    accepted = [r for r in results if r.began]
    rejected = [r for r in results if not r.began]
    rate_limited = [r for r in results if "Too many concurrent sessions" in r.error]
    errored_mid_call = [r for r in accepted if r.error]

    def pct(p):
        return delays[min(int(len(delays) * p), len(delays) - 1)]

    print(f"sessions attempted:  {len(results)}")
    print(f"sessions accepted:   {len(accepted)}")
    print(f"peak concurrent:     {_peak} (target {TARGET_SESSIONS})")
    print(f"rejected at connect: {len(rejected)} ({len(rate_limited)} rate-limited)")
    print(f"errored mid-call:    {len(errored_mid_call)}")
    print(f"finalized turns:     {len(delays)}")
    if delays:
        print(f"finalization P50:    {statistics.median(delays):.0f} ms")
        print(f"finalization P95:    {pct(0.95):.0f} ms")
        print(f"finalization P99:    {pct(0.99):.0f} ms")
    for r in (rejected + errored_mid_call)[:5]:
        print(f"  session {r.index}: code={r.error_code} {r.error}")


if __name__ == "__main__":
    load_test()

Two things in that script matter more than the rest.

A session isn't open until the server says it is. connect() doesn't raise when the server refuses a session; the refusal arrives on the Error handler. A harness that counts a session as live the moment connect() returns will happily report "peak 200" while a third of those sessions were rejected. So the script counts a session only after the Begin event, and doesn't send a byte of audio until then. That also keeps the latency math honest: start streaming before Begin and your audio clock runs ahead of the server's, and finalization delay comes out negative.

HOLD_SECONDS has to exceed the launch window, or the test quietly lies to you. At 200 sessions and 80 new sessions per minute you spend 150 seconds opening them all — so if each session hangs up at the end of a 30-second call, the early ones are long gone before the last one connects and you never see more than about 40 open at once. Looping the audio to a deadline is what makes the target real, and printing peak concurrent is what proves it. A run that reports 200 sessions attempted and 40 peak concurrent has tested a fifth of what you think it tested.

The script records rejections instead of retrying them, because in a load test a rejection is the finding. Your production client should do the opposite and back off and retry, but only on the right error (more on that below). Pin speech_model the way the script does, and swap mode to max_accuracy and re-run to see the accuracy-latency tradeoff on your own audio rather than in the abstract. balanced is the default and the right starting point for conversational agents.

Concurrency: what your provider actually gives you

This is the constraint that shows up last and hurts most, so get the answer in writing from every vendor before the pilot ends.

AssemblyAI's streaming API has no hard cap on total open sessions. The constraint is on new sessions per minute — 5 for free accounts, 100+ for paid — and on paid accounts it auto-scales. Free accounts stay at 5 concurrent sessions. Any minute you use 80% or more of your current limit, the next minute's limit rises 10%. Between 60% and 79% it holds. Below 60% it eases down 5% a minute, never below your account's starting limit. Max out for five straight minutes and you're at 146 new sessions per minute with 610 streams open, and it keeps climbing while you stay above the threshold. The flip side is worth knowing before a spike test after a quiet week: the limit resets to your starting limit after 15 minutes with no new sessions, even with plenty of sessions open.

Exceed the limit and the server sends an Error message, "Unauthorized Connection: Too many concurrent sessions", and closes the socket with code 1008. Despite the wording, it counts session starts per minute, not open sessions. Don't retry on the code alone: 1008 also covers a missing or invalid API key, an expired temporary token, a disabled key and insufficient funds, none of which a retry fixes. Read the text of the Error message (the close reason just says to see it), and back off and retry in the next minute only on "Too many concurrent sessions". Both are spelled out in the streaming rate limits and session errors docs. If you're seeing rejections unexpectedly, check that every session is being terminated.

The Voice Agent API is shaped differently. There are no concurrency fees or per-agent subscriptions, but your account does have a concurrent-session limit, and each simulated or real call is one session against it. Past the limit, new sessions are refused with a concurrency_exceeded session error. Treat that, at_capacity and internal_error as back-off-and-retry, reconnect once if the server drops you with a 1011 internal error (that close can arrive with no error event, so catch the close itself), and treat validation errors such as invalid_config as fatal: retrying them in a loop just burns attempts. Here's one Voice Agent session written for a load test, using a stored agent so every call is identical.

import asyncio
import base64
import json

import websockets

API_KEY = ""
AGENT_ID = ""          # a stored agent, so every session is identical
URL = "wss://agents.assemblyai.com/v1/ws"

CHUNK_MS = 100
SAMPLE_RATE = 24000                   # PCM16 mono at 24 kHz in and out
CHUNK_BYTES = SAMPLE_RATE * 2 * CHUNK_MS // 1000

# Capacity and internal errors: back off and try again. Everything else is a
# bug in the request (invalid_config, invalid_audio, agent_not_found...) and
# retrying just burns attempts.
RETRYABLE = {"at_capacity", "concurrency_exceeded", "internal_error"}


async def run_call(pcm: bytes, result: dict):
    headers = {"Authorization": f"Bearer {API_KEY}"}
    async with websockets.connect(URL, additional_headers=headers) as ws:
        await ws.send(json.dumps({"type": "session.update", "session": {"agent_id": AGENT_ID}}))

        async def stream_audio():
            loop = asyncio.get_running_loop()
            start = loop.time()
            for i in range(0, len(pcm) - CHUNK_BYTES + 1, CHUNK_BYTES):
                # Real-time pacing: faster than real time is audio_rate_violation.
                await asyncio.sleep(max(0.0, start + (i // CHUNK_BYTES + 1) * CHUNK_MS / 1000 - loop.time()))
                chunk = base64.b64encode(pcm[i:i + CHUNK_BYTES]).decode()
                await ws.send(json.dumps({"type": "input.audio", "audio": chunk}))
            # End explicitly. Just closing the socket keeps the session
            # resumable, and billable, for another 30 seconds.
            await ws.send(json.dumps({"type": "session.end"}))

        sender = None
        try:
            async for raw in ws:
                event = json.loads(raw)
                kind = event.get("type")
                if kind == "session.ready":
                    result["session_id"] = event["session_id"]
                    sender = asyncio.create_task(stream_audio())   # audio only after ready
                elif kind in ("session.error", "error"):
                    code = event.get("code")
                    result["error"] = code
                    result["retryable"] = code in RETRYABLE
                    break
                elif kind == "session.ended":
                    result["seconds"] = event.get("session_duration_seconds")
                    break
        except websockets.ConnectionClosed as e:
            # A 1008 rate-limit or 1011 internal-error close can arrive with no
            # error event, so record the close code. Only 1011 is worth a retry.
            close_code = e.rcvd.code if e.rcvd else 1006
            result["error"] = f"close_{close_code}"
            result["retryable"] = close_code == 1011
        finally:
            if sender:
                sender.cancel()


async def load_test(pcm: bytes, calls: int, per_second: float):
    results = [{} for _ in range(calls)]
    tasks = []
    for r in results:
        tasks.append(asyncio.create_task(run_call(pcm, r)))
        await asyncio.sleep(1 / per_second)     # ramp, don't burst
    await asyncio.gather(*tasks, return_exceptions=True)
    refused = [r for r in results if r.get("error") == "concurrency_exceeded"]
    print(f"calls: {calls}  completed: {sum('seconds' in r for r in results)}  "
          f"refused at concurrency limit: {len(refused)}")

It's a fragment rather than a script: call it with asyncio.run(load_test(pcm, calls=..., per_second=...)), passing your own 24 kHz 16-bit mono PCM audio. Note the order: audio goes out only after session.ready, and the call ends with session.end rather than a dropped socket. The next section explains why that last line is worth money.

For async workloads the shape is different again: 200 parallel jobs by default, plus an HTTP ceiling of 20,000 requests per 5 minutes across all endpoints, polling included. Blow through that and you get a 403. Use webhooks instead of tight polling loops.

The bill is part of the test

Here's the thing almost nobody load tests: the invoice.

Streaming and Voice Agent sessions are billed on how long the WebSocket stays open, not on how much audio you send. Idle time is billable. A streaming session you forget to close auto-closes after three hours — and bills for the full three hours. Improperly closed sessions are the single most common cause of surprise charges.

So a load test that leaks sessions doesn't just skew your latency numbers, it produces a bill that looks exactly like the 10× cost spike you were trying to rule out. Two habits fix it:

• Always send an explicit termination. For streaming that's {"type": "Terminate"}; the SDK's disconnect(terminate=True) does it for you.
• For the Voice Agent API, send session.end on any intentional disconnect. If you just close the socket, the server holds the session open for a 30-second resume window — and that window is billable. On a thousand-call test, that's over eight hours of billed time you didn't use.

While you're there, work out cost per completed task rather than cost per hour. An agent that gets 15% cheaper per hour while its task completion drops 20% has gotten more expensive, and the per-hour number will never show you that. On the Voice Agent API the per-hour side is simple multiplication: connected time × $4.50/hr ($0.075/min), billed per second, with speech-to-text, the LLM and voices included. Current rates for every product are on the pricing page.

Three ways to generate the load

Agent-vs-agent simulation. A second AI drives a scripted or improvised caller. Fast to set up, scales trivially, and catches logic and tool-calling failures. It misses real acoustics almost entirely — synthesized speech is cleaner than any human on a phone, so it will make your speech-to-text look better than it is.

Replayed production audio. Your own recordings, streamed back through the pipeline at real time. Best signal by a distance, because the audio is exactly what you'll get in production. Requires a corpus and consent to reuse recordings.

SIP-level call generators. Real calls over real telephony. Closest to reality, most setup, and the only way to test carrier-side behavior and DTMF handling.

Most teams end up using two: replayed audio for accuracy, simulation for conversation logic and volume. If you'd rather not build the harness at all, Bluejay, Coval, Hamming, and Cekura all specialize in this, and they plug into the same pipeline you're testing. If you're on the Voice Agent API, the docs walk through running Bluejay simulations and Cekura simulations against a stored agent. Whichever tool drives the calls, each one is a real session on your account, so size the run against your concurrency limit.

"We were searching for the best realtime ASR model for our voice agent pipeline in Fireflies. The new Universal 3.5 Pro speech model from Assembly is best so far in terms of accuracy, latency and language switching."

— Foysal Osmany, Software Engineer at Fireflies

Set the rollback plan before the first real call

Do the load-test half of rollback now. Pin your model: if you omit speech_model, streaming sessions default to universal-3-6-pro and float onto whatever the next flagship becomes. The streaming API exposes model IDs, not dated snapshots, so your fallback is a different model — universal-3-5-pro (still supported), universal-streaming-english, or universal-streaming-multilingual — and you should benchmark it on your corpus this week. Know your reconnect behavior: a dropped Voice Agent session can be resumed with session.resume and its session_id within 30 seconds; after that you start fresh. For ramp stages, numeric rollback thresholds and on-call ownership, see the voice agent launch runbook.

A one-week plan

• Monday. Assemble the corpus. Fifty worst calls, telephony-encoded, ground truth audited.
• Tuesday. Pilot at 10% of target. Verify the harness measures what you think it measures — check that peak concurrent actually reaches your target and that rejections are being recorded — and that your metrics pipeline is writing durably.
• Wednesday. Steady state and ramp. Find the knee.
• Thursday. Spike and soak, with the spike shaped like your real peak hour. Run the soak overnight.
• Friday morning. Rollback drill. Kill a model version mid-test and time how long recovery actually takes.
• Friday afternoon. Write up percentiles by profile, set the alert thresholds you'll launch with, and reconcile the bill against what you expected to spend.

If you're planning to exceed 200 requests per minute on async, running a large one-time batch, or testing in the EU region, email support first — AssemblyAI can pre-scale for the run and monitor it live, at no extra cost. The same goes for a big streaming or Voice Agent test: tell us the ramp before you run it.

The number that predicts production

Every vendor will quote you a median word error rate on clean audio. It is the least useful number in the entire evaluation, and it is the one on every landing page.

Ask instead for P95 entity error rate at your concurrency, on your audio, with a tool call in the middle of the turn. Watch which vendors can produce it and which change the subject back to WER. That single question sorts the field faster than a month of demos — and once you've built the harness above, you can answer it for yourself rather than taking anyone's word for it.

Your agent's ceiling isn't set by how good it is on a good day. It's set by what it does on the worst call of the week, and now you know how to find out what that is before your customers do.

Voice agent production-readiness checklist

Load testing is one of four things a voice agent needs before launch. Here's the whole list, one line per item, with links to the posts that go deep on each.

Load and concurrency

• A corpus of your 50 worst calls on 8 kHz telephony audio, ground truth audited.
• All four profiles run: steady state, ramp, spike, and a four-hour soak.
• Peak concurrency actually reached is measured, not the number you configured.
• The spike replays real peak-hour arrival patterns, not a flat rate.
• Every session ends explicitly: Terminate on streaming, session.end on the Voice Agent API.
• Support has been told about any large test so it can pre-scale before you run it.

Monitoring

• Per-hop P95 latency: speech-to-text finalization, LLM first token, voice first audio, tool calls.
• Caller repeat rate, as the early warning that the agent isn't understanding.
• Barge-in and dead-air rates, so silent failures emit a signal.
• Close codes and error text logged per session, not just counted.
• Cost per completed task, with an alert on drift.
• Depth: what to instrument once the agent is live, and how to catch regressions automatically from your call logs.

Benchmarking on your own data

• Ground truth audited before you score anything.
• Entity error rate scored separately from word error rate.
• Matched settings across vendors: same audio, same encoding, same endpointing mode.
• A live side-by-side on real conversations for turn-taking, which file replay can't capture.
• Depth: the evaluation guide linked above for methodology, and why voice AI evals are hard for picking metrics that match your product goal.

Contract and vendor questions

• What exactly is rate limited, and what's the unit: new sessions per minute, concurrent sessions, or requests?
• Can I pin a model version, and how much deprecation notice do I get?
• Is billing per second, and are idle and resume-window time billable?
• Are there concurrency fees, per-agent fees or minimums?
• Where can my data be processed, and is a Business Associate Addendum (BAA) available if I handle PHI?
• What's the support response target for a production-down issue? (The launch runbook covers the on-call questions to ask.)

Choosing a provider? Start with the best voice agent APIs compared, and check how flat-rate and per-minute pricing models compare against your projected call volume before you sign.

Frequently asked questions

How do I load test a voice agent before launching it to production call volume?

To load test a voice agent on AssemblyAI's Voice Agent API or streaming API, replay your 50 hardest real calls at real-time pace across many concurrent sessions, and ramp up to at least your projected peak hour. Measure finalization delay and end-to-end turn latency at P95, entity error rate, rejected and dropped sessions, and cost per completed task. Run four profiles (steady state, ramp, spike and soak), and end every session explicitly so leaked sessions don't distort the bill.

What questions should I ask before signing a contract for a voice agent API provider?

Ask what exactly is rate limited and in what unit, whether you can pin a model version and how much deprecation notice you get, and whether billing is per second with idle or resume time billable. Then ask about concurrency fees, per-agent fees or minimums, where data can be processed, whether a Business Associate Addendum (BAA) is available, and the support response target for production-down issues. AssemblyAI's Voice Agent API, for example, bills $4.50/hr per second of connected time with no concurrency fees or per-agent subscriptions, and refuses sessions past your account's concurrent-session limit with a concurrency_exceeded error.

How do I benchmark voice agent providers on my own call data before committing?

Record fifty representative calls including your hardest audio, audit the ground truth for human transcription errors, then run the same files through each provider with matched settings. Score entity error rate separately from word error rate, measure finalization delay per turn, and run a live side-by-side on real conversations, because turn-taking depends on the caller's device and environment in ways a file replay can't capture. AssemblyAI's voice agent evaluation guide walks through the full methodology.

How do I monitor latency and call quality for a live voice agent in production?

Track per-hop P95 latency and outcome signals such as caller repeat rate, dead-air rate and cost per completed task, segmented by language, intent and channel so cohort-specific failures don't hide in the average. On the Voice Agent API, store each session_id so you can pull that call's recording and transcript when a metric moves. The full metric set is in what to instrument once the agent is live.

How many concurrent calls should I load test for?

Start at your projected busiest hour, then test 3× that. Peak traffic for voice agents is rarely smooth — a callback campaign or a service outage produces a burst several times your average peak. Also test throughput separately from concurrency: opening 400 sessions in 60 seconds hits a different limit than holding 400 sessions open. And instrument the concurrency you actually reached, counted from the server's confirmation rather than the number you configured.

What's the rollback plan if a new voice agent model version underperforms?

On AssemblyAI streaming, pin speech_model explicitly and validate your fallback model on your own corpus before launch, then decide the metric, threshold and duration that trigger a rollback. Ramp stages, rollback thresholds and on-call ownership are covered in the voice agent launch runbook.