If you've looked at voice agent platforms before landing here, you may have noticed something: many of them lead with a visual interface. Click to configure. Fill in a field. Pick a voice from a dropdown. It looks approachable, and in some ways it is, at least on day one.
There's another way to build one: describe the agent to a coding agent and let it write the code against three focused APIs. That means streaming speech-to-text with Universal-3.6 Pro Realtime, an LLM through LLM Gateway, and AssemblyAI Text-to-Speech. This post explains why that approach holds up better both while you're building and after you've shipped, and it lays out the engineering workflow that gets a coding agent from an empty folder to an agent you'd trust on real calls.
If you want a quick first build, start with our guide to vibe coding a voice agent. This post is the step after that: how to work with a coding agent on a voice agent you intend to maintain.
What a visual builder actually is
A visual builder is a product interface that lets you configure software by filling out forms rather than writing code. You log in to a web dashboard, navigate to settings, type your agent's personality into a text field, choose a voice from a list, and click save. The platform runs the agent on its infrastructure and hides the underlying code from you entirely.
This model has genuine appeal. If you've never connected to an API or worked in a terminal, a visual interface removes those barriers. You don't need to know what a WebSocket is. You don't need to install anything. You configure, click, and the thing runs.
The tradeoff is that a visual builder can only expose what its designers put fields for. The agent's behavior is bounded by the interface: by what someone decided to include in the form, whatever else is technically possible. When your needs go beyond those fields (a custom tool that looks up a live database, logic that changes based on conversation context, behavior that responds to specific phrases), you quickly reach the edge of what the UI supports. Getting past that edge usually means workarounds, custom integrations, or waiting for the platform to add the feature you need.
For a simple agent, that ceiling might not matter. For an agent that needs to do real work, it does.
What a coding agent is, and why it changes the equation
A coding agent (Claude Code, Cursor, GitHub Copilot, Windsurf) is an AI-powered programming tool that reads instructions and writes code on your behalf. It goes well past chat and autocomplete: it takes a description of what you want to build, produces working project files, installs dependencies, runs commands, and tells you how to run the result.
You don't need to be a professional developer to use one. You do need to be comfortable opening a terminal and running a command. The core mechanic is simpler than it sounds: you describe what you want, and the coding agent builds it.
The reason this changes the equation is that language is a more powerful interface than a form. A form can only accept inputs its designer anticipated. Language can accept any description you can articulate. When you want your agent to do something new, you describe the change, and the coding agent implements it directly in the code, with no UI deciding what's allowed.
Why a cascaded stack suits a coding agent
A voice agent does three jobs: transcribe the caller, decide what to say, and say it. Building those as three separate hops, a cascaded architecture, sounds like more work for a coding agent than one all-in-one service. In practice it's the opposite, for three reasons.
- Each hop has a small, documented contract. Speech-to-text sends
Begin,TurnandTerminationevents, and a finished caller turn is aTurnwithend_of_turnset. LLM Gateway is OpenAI-compatible, so the coding agent already knows the request shape, and tools go on the chat completion request. Text-to-speech holds one session per conversation:generate()text as it arrives,flush()at the end of each sentence,cancel()when the caller interrupts. A coding agent can hold any one of those in its head at once. - Each hop can be built and tested on its own. You can get transcription working against a recording before an LLM exists, and you can test the voice before there's a microphone. Small, separately verified steps are where coding agents are most reliable.
- Every decision is in your code. The system prompt, the tools, the voice and the barge-in logic all live in files you own, so a change is a diff you can read, and a regression is a diff you can revert.
It also helps that all three hops run on one AssemblyAI API key, so there's one secret to manage and one place to look when a request is refused.
The engineering workflow
A single big prompt can get a demo running. An agent you'll maintain needs a little more structure. This is the loop that works.
1. Write the spec before the first prompt
Put the decisions that must not drift into a file the coding agent reads every session: CLAUDE.md for Claude Code, .cursorrules for Cursor, or your tool's equivalent. Name the exact model, SDK classes, voice and build order. A coding agent that has to guess a model name or a parameter will guess from its training data, and voice APIs move faster than training data.
# Voice agent: project spec
## Stack (do not substitute anything here)
- Speech-to-text: AssemblyAI Universal-3.6 Pro Realtime, speech_model="universal-3-6-pro".
Python SDK: RealTimeTranscriber from assemblyai.streaming.v3. Audio: 16 kHz, 16-bit mono PCM.
A finished caller turn is a Turn event with end_of_turn true. Leave the turn-silence
settings at their defaults.
- LLM: AssemblyAI LLM Gateway through aai.LLMGateway(), model "gpt-4.1", stream=True.
Tools are OpenAI-style "tools" on the chat completion request; our code runs them.
- Text-to-speech: AssemblyAI TTS. One aai.TTSSession per conversation, opened when the
call starts. voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3" (Audrey), language="english".
generate() each piece of LLM text as it arrives, flush() at the end of each sentence
(never per token), cancel() when the caller talks over the agent.
- One secret: ASSEMBLYAI_API_KEY, read from the environment. Never hardcode it.
## Ground truth
- Before writing AssemblyAI code, read https://www.assemblyai.com/docs/llms.txt and the
pages it lists for streaming speech-to-text, LLM Gateway and text-to-speech.
- pip install -U assemblyai (1.8.2 or newer). Check every assemblyai import and method
against the installed package before using it. If a name is not there, stop and ask me.
## Build order (one step per session; show me the diff before moving on)
1. Speech-to-text alone: stream a WAV file, print each final turn with its session id.
2. LLM alone: a text loop that streams replies and calls one tool, lookup_order.
3. Text-to-speech alone: speak a fixed sentence and write it to agent.wav.
4. Wire the three together. Barge-in last.
## Definition of done for every step
- Runs against local mock servers with a fake key. Point the SDK at them with
aai.settings.tts_base_url, aai.settings.llm_gateway_base_url and the api_host option
of RealTimeTranscriberOptions. No real key in tests, ever.
- Appends one JSON line per event to telemetry.jsonl, tagged with a conversation id and
each hop's session id.
- Handles the Error event of every hop and logs its code.The voice ID is set explicitly because defaults change. The spec says to leave turn detection at its defaults, because hand-tuned silence thresholds are the first thing a coding agent reaches for when a test sentence gets cut off, and usually the wrong fix.
2. Ground it in the current docs
Point the coding agent at the AssemblyAI docs index and tell it to read the pages for each hop before it writes code. Then make it prove every name: after pip install -U assemblyai, ask it to import each class it plans to use and list the methods it will call. A hallucinated parameter fails right there, before it can turn into a silent bug three files deep.
3. Build one hop per session
Ask for speech-to-text alone first, run it against a recording, and read the final turns it prints. Then the LLM with one tool, as a text loop. Then text-to-speech, speaking a fixed sentence to a file. Only then wire them together, and leave barge-in for last because it touches all three. After each step, read the diff before you accept it. Short sessions keep the coding agent's context focused on one contract at a time.
4. Test against mocks, never a real key
Have the coding agent write small local mock servers, one per hop: a WebSocket that sends scripted Turn events, an HTTP endpoint that streams a canned LLM reply with a tool call, and a WebSocket that answers Generate and Flush with silent audio and a FlushDone. The Python SDK can be pointed at them, so the agent's real code runs end to end with a fake key, in CI, in seconds. Scripted barge-ins and errors become regression tests rather than things you hope to catch on a live call.
5. Instrument from the first commit
Ask for one JSON log line per event, tagged with a conversation ID and each hop's session ID: the final transcript and its timing, LLM time to first token and tool calls, time to first audio, every barge-in, and every error code. It's a few lines when the code is being written and a project when it isn't. Our guide to voice AI observability lists what to capture at each hop and includes a working script.
6. Iterate in plain language
Now the loop the visual builder can't match. Run the agent, notice what you want to change, and describe it: "Have it introduce itself by name." "Add a tool that checks appointment availability." "Ignore 'mhm' and 'yeah' while the agent is talking instead of treating them as interruptions." The coding agent updates the code, the mocks prove nothing else broke, and you run it again. There's no form to navigate, no field to find, and no ceiling.
Stream your own audio and see the final transcripts your agent will act on, no code required.
Better to build, and better to maintain
This is where the coding agent approach pulls clearly ahead of a visual builder, and it's worth being specific about why.
When you build with a visual builder, your agent lives on the platform's infrastructure and is configured through its interface. Changing the agent means navigating back to the form. Adding a capability means checking whether the form has a field for it. Debugging means working with whatever logs and visibility the platform decides to expose.
When you build with a coding agent, your agent is a project that lives in your environment. You own the code. Every aspect of the agent's behavior (the system prompt, the voice, the tools it can call, how it handles turn detection and interruptions) exists as readable, changeable text in files you control, under version control, with tests around it.
That same ownership matters after you've shipped. When something needs to change (a new tool, an updated system prompt, a fix for a turn where callers keep repeating themselves), the change is a conversation with the coding agent and a diff you review. You can also swap any single hop: try a different LLM in LLM Gateway by changing one model name, or change the voice by changing one ID, without touching the other two. As the agent gets more sophisticated, the coding agent approach scales with it. The visual builder stays at whatever complexity its interface was designed to handle.
What each AssemblyAI hop gives the coding agent
The three APIs are documented the same way: a small set of events or methods, explicit names, and complete runnable samples. That makes them easy for a coding agent to read and hard for it to get subtly wrong.
- Speech-to-text: Universal-3.6 Pro Realtime. Our most accurate streaming model, covering 32 languages, with turn detection built in. When choosing a STT API for voice agents, accuracy on the entities your workflow depends on matters most, and it posts a 14.4% entity error rate.
- LLM: LLM Gateway. OpenAI-compatible, with models from several providers behind one endpoint and the same API key. Tool calling uses the standard
toolsparameter, which coding agents already know. - Text-to-speech: AssemblyAI TTS. Built for conversational voice agents and trained on millions of hours of audio, including real conversational data. It delivers frontier-level quality and runs faster than other production voice agent TTS models. It speaks six languages from a library of thousands of voices. One session per conversation accepts LLM text as it streams and cancels cleanly on barge-in.
Because the hops are separate, the coding agent can reason about each one with the docs page for that hop alone, which keeps the generated code short, readable, and easy to change.
Where to start
If you haven't used a coding agent before: pick one (Claude Code is well-suited to this workflow), start with the vibe coding guide to get a first agent talking, then come back here when you want to keep it.
If you have used coding agents but not for voice: copy the spec above into your project, change the tool and the system prompt to match your use case, and run the build order one step at a time. The step most worth your attention is barge-in, so give it its own session and its own mock test.
If you want the conceptual grounding before you start: our explainer on what an AI voice agent is covers how the pieces fit, and the TTS quickstart and streaming speech-to-text page cover each hop in depth.
One API key covers streaming speech-to-text, LLM Gateway, and Text-to-Speech. Hand your coding agent the spec and have a tested voice agent running this afternoon.
Frequently Asked Questions (FAQs)
What is a coding agent and do I need to know how to code to use one?
A coding agent is an AI-powered tool like Claude Code, Cursor, or GitHub Copilot that reads your plain-language instructions and writes working code for you. You don't need to be a developer to use one. You need to be comfortable opening a terminal and running a command, and willing to read the changes it proposes before you accept them.
How long does it take to build a voice agent with a coding agent?
A first working agent can run the same afternoon. Building it hop by hop, with a spec file, mock tests and logging, takes a little longer up front and saves far more time the first time something breaks on a real call.
What's the difference between building with a visual builder vs. a coding agent?
A visual builder limits you to the fields and options its designers included in the form. A coding agent lets you describe any change in plain language and implements it directly in code you own. When you need a custom tool, non-standard logic, or behavior the UI doesn't have a field for, the coding agent approach has no ceiling.
Which AssemblyAI APIs does the voice agent use?
Three, all on one API key: Universal-3.6 Pro Realtime for streaming speech-to-text (speech_model="universal-3-6-pro"), LLM Gateway for the language model, and AssemblyAI Text-to-Speech for the agent's voice. Tools and function calling live in the LLM request.
How much does it cost?
Each hop is billed separately. Universal-3.6 Pro Realtime streaming speech-to-text is $0.45 per hour of session time, LLM Gateway bills by the tokens of the model you choose, and Text-to-Speech is $15 per 1M characters, billed on input characters. See the pricing page for current rates.
Do I need to understand WebSockets to build this?
No. The Python and JavaScript SDKs manage the speech-to-text and text-to-speech connections, and the coding agent writes the event handling from the docs. It helps to know the few events your agent depends on, such as a final Turn from speech-to-text and FlushDone from text-to-speech, so you can read the logs when something goes wrong.