Why AssemblyAI’s speech APIs are built for coding agents

AssemblyAI’s speech APIs are built to be read by coding agents as much as by people. The docs publish a machine-readable index at llms.txt plus an integration prompt written for AI coding tools, the streaming protocol is plain JSON over a WebSocket with a handful of message types, the SDKs are typed, and the errors come back as documented close codes with exact text. That combination is what lets Claude Code, Cursor or Copilot scaffold a working voice agent on Universal-3.6 Pro Realtime on the first run instead of the fifth.

Below: what a coding agent is if you haven’t used one, why coding agents get speech APIs wrong, the four properties that make this one hard to get wrong, a working Python loop against the raw protocol, the prompts that scaffold a LiveKit, Pipecat or hand-rolled agent, and the benchmark data behind the speech layer.

What is a coding agent, and do I need to know how to code?

A coding agent is an LLM that has been given a terminal, a filesystem and your repository. You describe what you want in plain English; it reads the existing code, writes new files, runs them, reads the errors and fixes them. The loop is the point. A chat assistant hands you a snippet and wishes you luck. A coding agent runs the snippet, sees the stack trace and iterates until the thing works.

You don’t need to be able to write the code yourself. You do need to be able to read enough of it to tell when the agent has gone somewhere strange, and you need to be able to say precisely what you want. In practice the skill that matters is specification, not syntax: “when the caller gives an order number, look it up and read back the ship date” is a better instruction than anything you’d get out of a dropdown.

The four coding agents developers are actually using

• Claude Code — terminal-native. You run it inside your project directory and talk to it there. Strong at multi-file changes and at running what it writes, which matters for a WebSocket integration where the only real test is connecting.
• Cursor — a full editor built around the agent. Good when you want to watch diffs land in a familiar IDE and intervene by hand mid-task.
• GitHub Copilot — the most widely installed of the four, and increasingly agentic rather than just autocomplete. Usually the path of least resistance if your team already lives in GitHub.
• Windsurf — editor-based like Cursor, with a stronger emphasis on the agent keeping its own map of the codebase across a long task.

They differ in ergonomics more than capability. For the integration described here, any of the four will do the job. What determines whether the result works on the first run isn’t which agent you picked. It’s whether the agent is reading current documentation or guessing from training data.

Visual builder or coding agent?

Visual builders are faster for the first hour and worse for the next six months.

A drag-and-drop voice agent builder gets you to a working demo quickly because it has already made every decision for you: which speech model, which LLM, which voice, how turns are detected, what happens on a barge-in. That’s exactly why it stops being useful. The moment you need something the builder didn’t anticipate — a custom tool call against your internal API, different behavior for a drive-thru than for a headset, a retry policy your ops team will sign off on — you’re either filing a feature request or rebuilding the thing in code.

There’s also a portability problem. A flow built in a vendor’s canvas lives in that vendor’s canvas. A flow built as code lives in your repository, gets code review, gets tested in CI, and gets deployed by the same pipeline as everything else you run.

A coding agent collapses the gap. You get the speed of describing what you want in English and the durability of owning actual source. The tradeoff is that you have to be able to read the output, which is a real cost and a smaller one than being stuck inside someone else’s flowchart.

Why coding agents get speech APIs wrong

Coding agents fail in a specific way. They don’t usually write nonsense. They write plausible code against an API surface they’ve half-memorized: a field that sounds right, a parameter that existed two versions ago, an auth header copied from a different vendor.

Speech APIs are especially exposed to this. The category has years of outdated endpoints in public repos, every vendor names its messages differently, and the cost of a wrong guess is rarely a clean error. A misspelled parameter on a streaming connection often gets silently ignored, so the code runs, the transcript arrives, and the agent declares victory on a configuration you never asked for.

So the question to ask of any speech API you plan to build on with a coding agent is simple: how hard is it to get wrong? Four properties answer it.

1. Docs written for agents to read: llms.txt and the integration prompt

The single highest-leverage thing you can do is stop the agent guessing. AssemblyAI publishes its documentation index at https://www.assemblyai.com/docs/llms.txt. It’s a Markdown file: a title, a one-paragraph summary of the platform, then a Core machine-readable resources block that points at three things:

• llms-full.txt — the complete generated Markdown content of every docs page and API spec in one file. The docs note you can narrow it with ?lang=python or ?lang=typescript, or add ?excludeSpec=true to skip the API reference.
• agent-instructions.md — the integration prompt, described as rules of the road for AI coding agents building AssemblyAI integrations.
• llms.txt itself — the structured index.

After that, the index is organized into sections that mirror the docs: Overview, Pre-recorded STT, Real-time STT, Sync STT, Dictation API, Speech Understanding, Guardrails, LLM Gateway, FAQ, additional pages and the API specs. Each entry is a link with a one-line description, so an agent can find the turn detection page or the WebSocket spec without crawling the site. The Real-time STT section lists the voice agent integrations for LiveKit, Pipecat, Vapi and Bolna alongside the protocol reference, the rate limits page and the common session errors page.

The Build with AI agents page in the docs lays out four ways to wire this in: pin the integration prompt’s URL in your agent’s project instructions (the recommended default), paste the prompt directly, connect the docs MCP server, or install the Claude Code skill. Pinning is one line in AGENTS.md, CLAUDE.md, .cursorrules or .github/copilot-instructions.md:

Before writing AssemblyAI code, read https://www.assemblyai.com/docs/agent-instructions.md

If you’d rather have the agent query the docs on demand, the same page documents the MCP server for Claude Code:

claude mcp add assemblyai-docs --transport http https://assemblyai.com/docs/mcp

Whichever route you take, add a rule of your own to the project instructions: use the exact field and message names in the docs, and ask rather than infer when something isn’t there. That one sentence prevents most of the bugs in the rest of this post.

2. A protocol small enough to hold in context

The streaming WebSocket API is JSON control messages and binary audio over one connection. Configuration goes in the query string at connect time; everything else is a short list of typed messages. Here’s the whole contract a coding agent needs to start:

Endpoint   wss://streaming.assemblyai.com/v3/ws
           (US: streaming.us.assemblyai.com, EU: streaming.eu.assemblyai.com)
Auth       Authorization: <API_KEY>          (server side, no Bearer prefix)
           ?token=<TEMP_TOKEN>                (browsers: temporary token only)
Query      speech_model=universal-3-6-pro     (the default if omitted)
           sample_rate=16000
           encoding=pcm_s16le                 (or pcm_mulaw, opus, ogg_opus, aac)

Client -> server
  binary audio frames                         (50-1000 ms each, real-time pace)
  {"type": "UpdateConfiguration", ...}        (keyterms_prompt, agent_context, ...)
  {"type": "ForceEndpoint"}
  {"type": "KeepAlive"}                       (only with inactivity_timeout)
  {"type": "Terminate"}

Server -> client
  Begin          id, expires_at, configuration
  SpeechStarted  timestamp, confidence
  Turn           turn_order, end_of_turn, transcript, utterance, words, ...
  Termination    audio_duration_seconds, session_duration_seconds

Optional messages sit on top of that: SpeakerRevision when diarization is on, Heartbeat when you set session_heartbeat, and LLMGatewayResponse when you attach an LLM Gateway config to each finalized turn. None of them are needed for a first working agent.

The auth line is the one coding agents trip on most. Almost every API they’ve seen uses Authorization: Bearer. AssemblyAI’s streaming endpoint takes the raw key with no prefix, and browsers can’t set WebSocket headers at all, so client-side code connects with a short-lived token generated on your server. The spec is explicit that the permanent key never goes in a URL or in client-side code. Put that in your project instructions and the agent stops guessing.

The message your agent acts on is Turn. A finalized one looks like this (trimmed):

{
  "type": "Turn",
  "turn_order": 3,
  "turn_is_formatted": true,
  "end_of_turn": true,
  "end_of_turn_confidence": 0.91,
  "transcript": "My order number is 41820933.",
  "utterance": "My order number is 41820933.",
  "words": [
    {"text": "My", "start": 1120, "end": 1240, "confidence": 0.99, "word_is_final": true}
  ]
}

transcript carries all finalized words in the turn so far; utterance is populated only on end_of_turn messages. A voice agent hands the end-of-turn text to its LLM and ignores the rest. That’s the whole decision.

One behavior is worth writing into your instructions verbatim, because it’s exactly the silent failure described above: invalid connection parameters are ignored rather than rejected. The spec’s advice is to check configuration.model on the Begin message against what you requested. A coding agent that does that check on connect catches a typo before it becomes a week of mysterious accuracy problems.

3. Typed SDKs, and plugins where you’d rather not touch the socket

Raw WebSocket code is easy for an agent to verify against the spec, but you don’t have to write it. The official SDKs install with pip install assemblyai and npm install assemblyai. The Python SDK’s streaming v3 module gives you typed parameter and event classes, so a misspelled field fails at the editor rather than on the wire, and its disconnect call takes a terminate flag so the final turn is delivered before the session closes. The JavaScript SDK exposes a streaming transcriber with open, turn, error and close events and a close() call that finalizes the last turn.

If you’re building on an orchestrator, the typed surface is the orchestrator’s plugin. The LiveKit guide uses assemblyai.STT with model set to universal-3-6-pro inside an AgentSession. The Pipecat guide uses AssemblyAISTTService with the model set in its Settings. Either way your coding agent is filling in a constructor, not hand-rolling a protocol, and the docs pages give it the exact parameter names to copy.

4. Errors that tell the agent what to do next

The part of an integration a coding agent is worst at is the part it can’t see: what happens when the server says no. If the failure is a generic disconnect, the agent writes a generic retry loop, and a generic retry loop is how you turn one error into a thousand.

AssemblyAI’s streaming errors are documented as close codes with exact text, each with a stated fix. When the server sends an error, the WebSocket close reason is “See Error message for details” and the full text arrives in an Error message just before the close. The ones a voice agent will meet:

The rule that matters most is in the first row. Only retry on the rate-limit text. Every other 1008 means a credential or account problem, and reconnecting unchanged will fail again. The rate limit itself is on new sessions per minute (5 on the free tier, 100+ on paid accounts), not on how many sessions are open. There’s no cap on concurrent open sessions, and paid limits scale up automatically: open 80% or more of your current limit in a minute and the next minute’s limit rises by 10%. The docs name the usual cause of unexpected 1008s directly: code that reconnects in a loop without backoff.

That is precise enough to hand to a coding agent as a spec. Here’s what it produces:

import asyncio
import random


class SessionClosed(Exception):
    def __init__(self, code, last_message):
        super().__init__(f"{code}: {last_message}")
        self.code = code
        self.last_message = last_message


RATE_LIMITED = "Too many concurrent sessions"


async def connect_with_backoff(start_session, attempts=5):
    """start_session() runs one streaming session and raises SessionClosed on an error close."""
    for attempt in range(attempts):
        try:
            return await start_session()
        except SessionClosed as err:
            if err.code == 1008 and RATE_LIMITED in err.last_message:
                # New-session rate limit: wait into the next minute, with jitter.
                await asyncio.sleep(min(60, 2 ** attempt * 5) + random.random())
                continue
            raise  # bad key, expired token, no funds: fix it, don't retry
    raise RuntimeError("rate limited after retries")

The other billing-relevant detail belongs in the same instructions file. Send a Terminate message to close a session. Sessions that aren’t terminated auto-close after three hours and are billed for the full duration.

A working Python loop against the raw protocol

Here’s the streaming side of a voice agent written directly against the spec, with no SDK, so you can check every name against the API reference. It connects to Universal-3.6 Pro Realtime, biases recognition toward a few domain terms, hands each finished turn to your LLM, feeds the agent’s reply back as context, and terminates cleanly. The TTS step is a stub on purpose: plug in whichever text-to-speech you use.

# pip install "websockets>=14"
import asyncio
import json
import os
from urllib.parse import urlencode

import websockets

API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
PARAMS = {
    "speech_model": "universal-3-6-pro",
    "sample_rate": 16000,
    "encoding": "pcm_s16le",
}
URL = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(PARAMS)}"


def synthesize(text):
    """Stub: send text to your TTS provider and play the audio."""
    print("agent:", text)


async def send_audio(ws, audio_chunks):
    # 16 kHz mono PCM16, 50-1000 ms per binary frame, at real-time pace.
    async for chunk in audio_chunks:
        await ws.send(chunk)
    await ws.send(json.dumps({"type": "Terminate"}))  # flush the last turn, stop billing


async def run_session(audio_chunks, llm_reply):
    last_message = ""
    async with websockets.connect(URL, additional_headers={"Authorization": API_KEY}) as ws:
        await ws.send(json.dumps({
            "type": "UpdateConfiguration",
            "keyterms_prompt": ["AssemblyAI", "backorder", "expedited"],
        }))
        sender = asyncio.create_task(send_audio(ws, audio_chunks))
        try:
            async for raw in ws:
                last_message = raw
                msg = json.loads(raw)
                if msg["type"] == "Begin":
                    # Invalid parameters are ignored, not rejected: confirm the model.
                    assert msg["configuration"]["model"] == "universal-3-6-pro", msg
                elif msg["type"] == "Turn" and msg["end_of_turn"]:
                    print("caller:", msg["transcript"])
                    reply = await llm_reply(msg["transcript"])  # your LLM
                    synthesize(reply)
                    await ws.send(json.dumps({
                        "type": "UpdateConfiguration",
                        "agent_context": reply[:1750],  # what the agent just said
                    }))
                elif msg["type"] == "Termination":
                    break
        except websockets.ConnectionClosedError as e:
            code = e.rcvd.code if e.rcvd else None
            raise SessionClosed(code, last_message) from e  # see the backoff example
        finally:
            sender.cancel()

Three details in there are the ones generated code most often gets wrong. The Authorization header has no Bearer prefix. agent_context is the agent’s own spoken reply, capped at 1,750 characters, and you update it mid-stream with UpdateConfiguration after every reply. And the session ends with a Terminate message, not just a closed socket. Wire run_session into connect_with_backoff from the previous section and you have reconnect behavior that matches the docs.

How a coding agent scaffolds each path

There are three common ways to put Universal-3.6 Pro Realtime inside a voice agent. AssemblyAI supplies the speech layer in each; you own the orchestration and the LLM, or route the LLM through LLM Gateway. The prompts below are the ones to start with. Each tells the agent where to read before it writes.

LiveKit

The LiveKit guide sets turn detection to stt so AssemblyAI’s turn detection ends turns, matches Silero’s activation threshold to the STT vad_threshold, and sets the endpointing min_delay to 0 to avoid adding delay on top. A prompt that gets all three right:

Build a LiveKit Agents voice agent in Python.

Before writing code, read https://www.assemblyai.com/docs/agent-instructions.md
and https://www.assemblyai.com/docs/voice-agents/livekit-universal-3-6-pro.
Use assemblyai.STT with model="universal-3-6-pro" and follow the guide's
AgentSession setup exactly: turn_detection="stt", Silero activation_threshold
matched to the STT vad_threshold, endpointing min_delay 0.

Use my OpenAI key for the LLM. Leave the TTS slot as a clearly marked TODO.
Use only parameter names that appear in the docs. If something isn't there,
stop and ask me.

Pipecat

The Pipecat guide constructs AssemblyAISTTService with the model inside its Settings and sets vad_force_turn_endpoint to False, which switches Pipecat over to AssemblyAI’s built-in turn detection (available on the Universal-3 Pro family). Leave that flag out and Pipecat’s default VAD-driven mode decides when the caller is done instead, which is a different agent.

Build a Pipecat voice agent in Python.

Read https://www.assemblyai.com/docs/voice-agents/pipecat-universal-3-5-pro
before writing code. Use AssemblyAISTTService with model "universal-3-6-pro"
in its Settings and vad_force_turn_endpoint=False so AssemblyAI's turn
detection ends turns. After every bot reply, update agent_context with the
reply text.

Read ASSEMBLYAI_API_KEY from the environment. Do not invent parameters.

Your own loop

If you want no framework at all, the Python loop above is the starting point. Point the coding agent at the WebSocket spec and the common session errors page, give it the backoff function as a requirement, and ask it to add the pieces around the edges: microphone or telephony audio in, your LLM call, your TTS out, and barge-in, where SpeechStarted is the signal to stop playback. The protocol is small enough that this usually takes a couple of prompts, and every name the agent writes can be checked against one page.

Where the accuracy levers are

Most of the tuning surface on a voice agent isn’t in the system prompt. It’s in what the speech model knows about the moment it’s transcribing, and on Universal-3.6 Pro Realtime that comes down to a few documented parameters.

• agent_context — the agent’s last spoken reply, up to 1,750 characters, updated mid-stream. If the agent just asked for an email address, the model expects one. Across 20,000 voice-agent files, passing agent_context cut word error rate by 10.2%.
• keyterms_prompt — up to 100 words or phrases to bias toward, included on streaming at no extra cost and updatable mid-session. Pair it with any lookup tool you ship: push the candidate company or product names into keyterms so the transcript doesn’t mangle a name before your tool sees it.
• Voice Focus — near-field for headsets and handsets, far-field for rooms, kiosks and drive-thrus, with a strength threshold that defaults to 0.7. It’s +$0.10/hr.
• Turn detection — combines semantic context with voice activity rather than waiting on silence alone. The turn detection docs cover min_turn_silence and max_turn_silence if you need to tune.

One more spec detail that saves a debugging session: on Universal-3.6 Pro Realtime, pass context through agent_context and keyterms_prompt. The streaming API reference lists the free-text prompt parameter for Universal-3.5 Pro, and since invalid parameters are ignored, a coding agent sending it from memory won’t see an error either way, so have it check the current reference before relying on prompt. The prompting and keyterms documentation has the methodology.

What the speech layer actually scores

Here’s how Universal-3.6 Pro Realtime performs on AssemblyAI’s English voice-agent benchmark, run against 12,460 scripted voice-agent scenarios rather than read audiobooks. Lower is better throughout.

The entity rows are the ones to read if you’re building anything transactional. Aggregate word error rate tells you how the transcript reads; entity error rate tells you whether the agent got the order number, the street, the name and the callback number right. A 14.4% entity error rate versus 30.1% is the difference between an agent that can take an address and one that can’t. Methodology is in the Universal-3.6 Pro Realtime research post.

Independent benchmarks point the same way. On Pipecat’s open STT benchmark, Universal-3.6 Pro Realtime posts a 0.96% pooled semantic word error rate, with the final transcript landing a median 307 ms after the speaker stops, and Coval’s STT benchmark showed it with the lowest word error rate, 2.2% over a 7-day window at launch in September 2026. On AssemblyAI’s own benchmarks page, its realtime voice-agent results rank first of 22 systems, with a 14.41% entity error rate, the lowest in the set.

A reminder for the coding agent writing your evaluation harness: measure the same span on every system. A per-layer transcription number isn’t comparable to a full turn that includes the LLM and speech synthesis.

What it costs

Universal-3.6 Pro Realtime is $0.45/hr, billed per second, pay-as-you-go, with no minimums. Keyterms prompting is included; streaming diarization is +$0.12/hr, Voice Focus +$0.10/hr and Medical Mode +$0.15/hr. There’s no cap on open sessions. New sessions per minute are rate limited, as above, and paid limits scale automatically.

The rest of the bill is yours to choose. The LLM can be any provider, or GPT, Claude, Gemini and other models through LLM Gateway on the same account. For the speech-out side, AssemblyAI’s Text-to-Speech API launches on October 28, 2026, built for conversation rather than narration, so speech in and speech out can come from one vendor. Until its docs are live, keep TTS behind a stub like the one above and let your coding agent fill it in later.

Two cost habits are worth putting in the project instructions so the agent writes them by default: send Terminate when a call ends, and never reconnect in a tight loop. Both are documented, and both show up on the invoice if they’re missing.

Where to go from here

If you want to hear behavior before writing code, the playground runs your own audio through the speech layer. For the bigger picture of what voice agents are being used for and what breaks in production, AI voice agents is the overview. The two pages to keep open while your coding agent works are the WebSocket API reference and llms.txt itself.

Frequently asked questions

Do I need to know how to code to build a voice agent with a coding agent?

No, but you need to read code well enough to notice when the agent has gone somewhere strange. The skill that matters is specification: describing precisely what the agent should do, what it must never do, and which tools it can call. If you can write that clearly, Claude Code, Cursor, GitHub Copilot or Windsurf can write the integration against Universal-3.6 Pro Realtime.

What is AssemblyAI’s llms.txt?

It’s a Markdown index of AssemblyAI’s documentation at https://www.assemblyai.com/docs/llms.txt, written for LLMs and coding agents. It starts with three core machine-readable resources (llms-full.txt with the full docs content, agent-instructions.md with the integration prompt for AI coding agents, and the index itself), then lists every docs page by section with a one-line description.

How do I stop my coding agent inventing API fields?

Point it at current documentation instead of letting it work from training data. Pin https://www.assemblyai.com/docs/agent-instructions.md in AGENTS.md, CLAUDE.md or your editor’s rules file, tell the agent to use the exact names in the WebSocket API reference and to ask rather than infer, and have it check configuration.model on the Begin message, since invalid parameters are ignored rather than rejected.

How should a voice agent handle a 1008 close from the streaming API?

Read the error text that arrives just before the close. Only “Unauthorized Connection: Too many concurrent sessions” should be retried: it’s the new-sessions-per-minute rate limit, so back off and reopen the session in the next minute. Any other 1008, such as an invalid API key, an expired token or insufficient funds, needs the credential or account fixed first.

Does Universal-3.6 Pro Realtime work with LiveKit and Pipecat?

Yes. LiveKit uses assemblyai.STT with the model set to universal-3-6-pro and turn detection set to stt. Pipecat uses AssemblyAISTTService with the model in its Settings and vad_force_turn_endpoint set to False so AssemblyAI’s turn detection ends turns. Both docs guides give the exact parameters a coding agent should copy.

How much does the speech layer cost?

Universal-3.6 Pro Realtime is $0.45/hr, billed per second, with no minimums and no cap on open sessions. Keyterms prompting is included; streaming diarization, Voice Focus and Medical Mode are add-ons. The free tier includes 333 hours of streaming with no credit card. Your LLM and TTS are billed separately by whichever provider you choose.