Voice AI observability: What to instrument once the agent is live

The agent looked up the wrong account.

Nothing errored. No alert fired. Latency was fine, the tool call returned in 180 ms, the LLM did exactly what it was told. The caller said their prescription number was RX-7704132, the model heard "RX-770 for 132," and the agent confidently pulled a record that didn't exist, apologized, and escalated a call that should have taken forty seconds.

Every dashboard you have says that call was healthy.

That's the distance between "it works" and "we'd know if it didn't," and closing it is what voice AI observability is for.

What voice AI observability means

Voice AI observability is the practice of instrumenting a live voice agent so you can explain any conversation after the fact — what the caller said, what the model heard, what the agent decided, how long each step took, and where it went wrong.

It splits into three layers, and most teams instrument exactly one of them:

• Transport. Packet loss, jitter, codec, connection drops. Standard telephony monitoring. Almost everyone has this.
• Pipeline. Per-hop timing and behavior across speech-to-text, LLM, and text-to-speech, plus turn detection and tool calls. Some teams have parts of it.
• Conversation outcome. Did the caller get what they came for, did they have to repeat themselves, did they ask for a human. Almost nobody has this, and it's the layer that tells you whether the product works.

The tell is simple. If a stakeholder asks "why did that call go badly?" and your answer requires listening to the recording, you have monitoring, not observability.

The four signal families

Everything worth instrumenting falls into one of four groups. Emit all four per turn, tagged with a session ID, and you can answer almost any question after the fact. If you're still deciding which capabilities matter enough to measure, our breakdown of the voice agent features that actually matter in production is the companion piece to this list.

Latency

Per-hop, not just end-to-end. An 1,800 ms turn is a different problem depending on whether it's 1,400 ms of speech-to-text finalization or 1,400 ms of your own CRM lookup.

Track time to complete transcript (the gap between the caller finishing a word and the final transcript landing — this is what gates your LLM), LLM time-to-first-token, text-to-speech time-to-first-byte, and end-of-turn decision time. Report P50, P95 and P99. Median latency on a quiet Tuesday is not a metric, it's a comfort.

One caveat worth knowing: time-to-first-token is a weaker signal for speech-to-text than it looks, because some providers emit tokens before anything has actually been said, which games the number. Prefer emission latency if you consume partials, or time to complete transcript if you act on finalized turns.

Accuracy

The hard one, because accuracy on live traffic has no ground truth. Three proxies get you most of the way:

• Word-level confidence. Streaming Turn messages carry a words[] array with per-word confidence, and the Voice Agent timeline carries user_confidence per turn. A rising share of low-confidence tokens is drift you can see before customers report it.
• Entity capture rate. Instrument the tokens your workflow actually branches on — account numbers, confirmation codes, emails. Compare captured-on-first-try against had-to-re-ask. This is also why word error rate is broken as a production metric: it averages across every token, and the ones that matter are a rounding error in the average.
• Caller repeat rate. More on this below. It's the best one and it's nearly free.

Conversation

Barge-in rate, escalation rate, task completion rate, dead-air events, and average turns to resolution. These are product metrics that happen to live in your telemetry. A rise in turns-to-resolution with flat latency and flat error rates almost always means the agent stopped understanding people.

Cost

Cost per completed task, not cost per hour. An agent that gets 15% cheaper per hour while its completion rate drops 20% has gotten more expensive, and the per-hour number will never show you that.

Two cost mechanics are easy to miss on voice specifically. Streaming and Voice Agent sessions bill on WebSocket-open duration rather than audio sent, so idle time on a connection costs money. And if you close the socket without sending session.end, the server holds a 30-second resume window that is also billable — at scale, that's real money leaking out of a metric nobody watches.

Why the transcript is the log

Here's the position, and it's an architectural one rather than a preference.

A voice agent is a chain, and the LLM has no way to know its input was wrong. It receives text and treats it as ground truth. When the transcription turns "RX-7704132" into "RX-770 for 132," the model doesn't hesitate — it acts, confidently, on a corrupted input. The failure is invisible in your logs unless you stored the transcript, and it's indistinguishable from a model-quality problem unless you go listen to the audio.

Which means any architecture where you cannot inspect the text the model actually heard fails observability by design. Black-box audio-in, audio-out systems are exactly that architecture. That's the strongest practical argument for a cascading pipeline over a single speech-to-speech model — we compared all three approaches in voice agent architectures explained — and it has nothing to do with accuracy benchmarks: it's that when something goes wrong at 2am, one of them can be debugged and the other can only be listened to.

It's also why context matters more than raw accuracy. Universal-3.5 Pro Realtime keeps a rolling conversation memory — Context Carryover, on by default — so a short reply resolves against what was just asked rather than in a vacuum. Across a benchmark of 20,000 voice agent audio files, passing the agent's own spoken reply in as context cut word error rate by 10.2%, with fabrications down 18.3% and hallucinations down 17.2%. The same model powers streaming speech-to-text if you'd rather bring your own LLM and voice.

"We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms."

— David Zhao, Co-founder at LiveKit

You probably already have the dataset

This is the part most teams don't realize.

If you're on the AssemblyAI Voice Agent API, every call is stored as a session, and every session carries a timeline artifact: the whole conversation, turn by turn, as JSON. It already contains most of what you'd otherwise build a pipeline to collect.

Each turn gives you user_transcript and agent_text, a user_confidence score, time_to_first_audio_ms, a status of completed or interrupted, a trigger telling you whether the turn came from the greeting or from user speech, and — only on turns where a tool actually fired — a tool_calls array with dispatched_at_ms, result_received_at_ms, duration_ms, and is_error.

Barge-in rate is status == "interrupted". Tool latency is duration_ms. Agent responsiveness is time_to_first_audio_ms. Low-confidence rate is user_confidence. Four of the metrics above, sitting in an artifact you're already generating.

Here's a script that pulls a page of sessions and computes them:

import os
import re

import requests

BASE = "https://agents.assemblyai.com"
HEADERS = {"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}


def all_sessions(**filters):
    """Page through every session, newest first."""
    cursor = None
    while True:
        resp = requests.get(
            f"{BASE}/v1/sessions",
            headers=HEADERS,
            params={"limit": 200, "cursor": cursor, **filters},
        )
        resp.raise_for_status()
        page = resp.json()
        yield from page["sessions"]
        cursor = page["response_metadata"]["next_cursor"]
        if not page["has_more"] or not cursor:
            break


def timeline_for(session_id):
    """Fetch a session and download its timeline artifact."""
    session = requests.get(f"{BASE}/v1/sessions/{session_id}", headers=HEADERS).json()
    # Artifact URLs are pre-signed and short-lived — fetch fresh, don't cache.
    url = next((a["url"] for a in session["artifacts"] if a["type"] == "timeline"), None)
    if url is None:
        return None  # session still active, or no recording
    return requests.get(url).json()  # pre-signed: no auth header


def normalize(text):
    """Fold a transcript to a comparable form.

    Punctuation becomes a space rather than being deleted, so "RX-7704132" and
    "RX 7704132" fold together instead of differing by one. Apostrophes are
    dropped first, so "it's" matches "its" rather than becoming "it s".
    """
    text = (text or "").lower().replace("'", "").replace("’", "")
    return re.sub(r"[^a-z0-9]+", " ", text).strip()


def score(agent_id, max_sessions=200):
    ttfa, tool_ms = [], []
    turns = user_turns = interrupted = low_conf = repeats = tool_errors = 0
    tool_calls_seen = 0
    sessions_seen = 0

    for session in all_sessions(agent_id=agent_id, status="completed"):
        if sessions_seen >= max_sessions:
            break
        timeline = timeline_for(session["id"])
        if not timeline:
            continue
        sessions_seen += 1

        previous_user = None
        for turn in timeline.get("turns", []):      # `turns` is omitted when empty
            turns += 1

            # Agent-initiated turns (the greeting) can never carry
            # user_confidence, and their time_to_first_audio_ms is time to
            # greeting rather than a response time. Counting them would dilute
            # every caller-side rate below, so gate on trigger.
            is_caller_turn = turn.get("trigger") == "user_speech"

            if turn.get("status") == "interrupted":
                interrupted += 1

            if is_caller_turn:
                user_turns += 1

                if turn.get("time_to_first_audio_ms") is not None:
                    ttfa.append(turn["time_to_first_audio_ms"])

                confidence = turn.get("user_confidence")
                if confidence is not None and confidence < 0.75:
                    low_conf += 1

                # Caller repeat: the same thing said twice in a row.
                current_user = normalize(turn.get("user_transcript"))
                if current_user and current_user == previous_user:
                    repeats += 1
                if current_user:
                    previous_user = current_user

            # Tools can fire on any turn, so this stays outside the gate.
            for call in turn.get("tool_calls", []):  # omitted when no tool fired
                tool_calls_seen += 1
                if call.get("duration_ms") is not None:
                    tool_ms.append(call["duration_ms"])
                if call.get("is_error") or call.get("timed_out"):
                    tool_errors += 1

    def pct(values, p):
        if not values:
            return float("nan")
        values = sorted(values)
        return values[min(int(len(values) * p), len(values) - 1)]

    def rate(numerator, denominator):
        # nan rather than 0% when there's nothing to divide by — "no data" and
        # "healthy" should never print the same.
        return float("nan") if not denominator else numerator / denominator

    print(f"sessions                  {sessions_seen}")
    print(f"turns                     {turns} ({user_turns} from the caller)")
    if not turns:
        print("no turns in sample — nothing to score")
        return
    print(f"barge-in rate             {rate(interrupted, turns):.1%}")
    print(f"caller repeat rate        {rate(repeats, user_turns):.1%}")
    print(f"low-confidence turn rate  {rate(low_conf, user_turns):.1%}")
    print(f"time to first audio  P50  {pct(ttfa, 0.50):.0f} ms")
    print(f"time to first audio  P95  {pct(ttfa, 0.95):.0f} ms")
    print(f"tool round trip      P95  {pct(tool_ms, 0.95):.0f} ms")
    print(f"tool error rate           {rate(tool_errors, tool_calls_seen):.1%}")


if __name__ == "__main__":
    score(os.environ["AGENT_ID"])

Run that against yesterday's traffic and you have a baseline. Run it nightly and you have drift detection.

One detail in there is worth stealing even if you write your own: the caller-side rates divide by caller turns, not by all turns. Every session opens with an agent-initiated greeting that structurally cannot carry a confidence score, so counting it in the denominator quietly deflates your low-confidence and repeat rates on every single call. That's what trigger is for.

The metric almost nobody instruments

Caller repeat rate — how often a caller says substantially the same thing twice in a row.

It's the closest thing to a free accuracy signal on live traffic. You don't need ground truth, you don't need a human reviewer, and you don't need to change your pipeline. You need the transcript you already have and about ten lines of string comparison, which is the repeats counter above.

And it's a better proxy for speech failure than anything else available in production, because it measures the thing the caller actually experienced. A model can post a beautiful word error rate on your test set and still make people repeat their email address twice on every call. Repeat rate catches that. Word error rate on a benchmark corpus does not.

Normalize before you compare, and normalize carefully — this is where a naive version silently under-reports. Deleting punctuation instead of replacing it with a space makes "RX-7704132" fold to rx7704132 while "RX 7704132" folds to rx 7704132, so the two never match. That's exactly the shape of the entities repeat rate exists to catch: codes, phone numbers, spelled-out emails. The normalize above replaces punctuation with a space and drops apostrophes first, which handles it.

Two refinements once that's running. First, fuzzy-match rather than exact-match — "four one five five five five" and "it's four one five, five five five" are the same repeat even after normalization, and only an edit-distance threshold catches it. Second, segment by what the agent asked immediately before. A repeat rate of 3% overall might be 1% on yes/no confirmations and 19% on the turn where you ask for an email address, and that's a fixable, specific problem rather than a vague accuracy complaint.

When you find that hot spot, the fix is usually not a different model. It's keyterms prompting or a transcription prompt that tells the model what's coming, which you can update mid-session without reconnecting.

Instrumenting each hop

A few practical notes on emitting spans, drawn from the places teams usually get it wrong.

Tag everything with the session ID at connect time. On streaming, that's id from the Begin message. On the Voice Agent API, it's session_id from session.ready. Without it, correlating a per-hop span to a conversation is guesswork.

Tag the mode and the model version too. Universal-3.5 Pro Realtime runs in min_latency, balanced, or max_accuracy, and they have materially different latency profiles. Comparing P95 across modes without labelling them produces a dashboard that says something changed when nothing did.

Instrument the seams, not just the services. The two spans teams forget are the handoff from speech-to-text output to LLM input, and the tool-call round trip against their own backend. Both are usually your code, which is exactly why nobody thinks to instrument them, and both are common sources of latency that get blamed on a vendor.

Log close codes, don't just count disconnects. Streaming sessions close with a specific code and reason. 3008 means the session hit the three-hour limit — and you were billed for all of it. 3009 means you exceeded the new-session rate limit. 3007 means audio chunks outside the 50–1000 ms window, or audio sent faster than real time. 1008 covers authorization and account issues, including insufficient balance. These are four completely different incidents that look identical if all you record is "session closed." The full table is in the session errors and closures docs.

Read the Error frame, not the close reason. WebSocket close reasons cap at 123 bytes, so long errors get truncated to "See Error message for details" with the real text delivered in a separate Error frame just before the socket closes. Teams that only log the close reason lose the actual error.

Alerts that should page someone

Voice agents mostly fail slowly. Alert on drift, not spikes.

Treat these as starting points and tune against two weeks of your own baseline. An alert that fires every day teaches people to ignore alerts.

Closing the loop

Every bad production call is a test case you didn't have to invent.

Once you can identify the worst conversations — highest repeat rate, most turns to resolution, escalated — pull their audio and add them to the corpus you regression-test against. The recording is stereo with the caller on the left channel and the agent on the right, so it splits cleanly for per-speaker analysis. That closes the loop from production back into pre-launch load testing, and it's how a voice agent gets better over time instead of just older.

This is also where the eval platforms earn their place. LiveKit ships agent observability inside LiveKit Cloud; Hamming, Coval, and Langfuse do simulation, scoring, and regression suites. They sit above the speech layer and they're complementary to it — they'll tell you the agent handled a scenario badly, and the transcript tells you whether it handled it badly because it misheard.

The cheapest instrumentation is the one you already paid for

Most voice observability budget goes toward watching the agent talk — response latency, synthesis quality, dashboards of how quickly the bot replied.

The higher-yield instrumentation watches whether the caller had to repeat themselves. It costs nothing beyond the transcript you're already generating, it needs no ground truth, and it correlates with customer frustration better than any latency percentile on your wall.

Start there. Get the repeat rate for yesterday, segment it by the question the agent asked, and you'll find your worst turn inside an afternoon.

Frequently asked questions

What is voice AI observability?

Voice AI observability is instrumenting a live voice agent so any conversation can be explained after the fact — what the caller said, what the model heard, what the agent decided, how long each hop took, and where it failed. It spans three layers: transport (packet loss, jitter, drops), pipeline (per-hop timing across speech-to-text, LLM, and text-to-speech), and conversation outcome (task completion, escalation, repeats). The third layer is the one that tells you whether the product works.

What metrics should I track for a voice agent in production?

Four families. Latency: time to complete transcript, LLM time-to-first-token, text-to-speech time-to-first-byte, end-of-turn decision time, all at P95. Accuracy: word-level confidence, entity capture rate, caller repeat rate. Conversation: barge-in rate, escalation rate, task completion, turns to resolution. Cost: cost per completed task, not per hour.

How do I monitor latency and call quality for a live voice agent?

Emit a span per hop tagged with the session ID, the model version, and the streaming mode, then report percentiles rather than averages. On the AssemblyAI Voice Agent API, each session's timeline artifact already carries time_to_first_audio_ms per turn and duration_ms on every tool call, so you can compute a latency baseline from stored sessions before you build any live instrumentation. Filter to turns whose trigger is user_speech when you do — the greeting's time_to_first_audio_ms measures time to greeting, not responsiveness.

How do I detect when my voice agent's accuracy is degrading?

Live traffic has no ground truth, so use proxies: the share of turns with low word-level confidence, the entity capture rate on the tokens your workflow branches on, and caller repeat rate. Alert on week-over-week drift rather than single-call spikes — accuracy regressions in voice show up as a slow climb, not a step change.

Do I need an observability platform, or can I build this myself?

Both work, and they solve different problems. If your agent is on the Voice Agent API, the session timeline gives you barge-in rate, per-turn latency, tool timings, and confidence scores from stored data — enough for a nightly job and a dashboard. Dedicated platforms add scenario simulation, automated scoring, and regression suites on top. Start with the data you already have, then buy the layer above it if you need it.

What's the difference between voice agent testing and voice agent observability?

Testing happens before launch and asks whether the agent can handle a scenario — you control the inputs, you have ground truth, and you're looking for a pass or fail. Observability happens after launch and asks what actually happened on real calls, where you control nothing and have no ground truth. They feed each other: observability finds the failures, testing turns them into regressions you never ship again. See how to load test a voice agent for the pre-launch half.

‍