The Thinker and the Responder: Bring your own response generation to a voice agent

The two roles, defined

If you've read Voice agent architectures explained, you already know the three-verb version of this: a voice agent hears, thinks, and speaks. Thinker and Responder is the same anatomy cut along a different line, not by what each step does, but by who owns it and for how long. Hearing and speaking turn out to be the same job. Thinking is the odd one out.

The Responder is AssemblyAI's managed realtime engine. It owns:

• voice activity detection (input.speech.started, input.speech.stopped)
• streaming transcription (transcript.user.delta, then transcript.user), running on Universal-3.5 Pro Realtime
• turn detection, deciding when the caller has actually finished
• text-to-speech playback (reply.started through reply.done)
• interruption and barge-in handling, when the caller talks over the agent mid-reply

The Thinker is whatever decides what to say. That's it. It comes into existence when turn detection fires, and it's gone the moment the last token is written. Everything else on that list happens without it.

Here's how the demo repo I built for this puts it, in a comment in the code:

The responder holds the call: it is listening, deciding a turn ended, or speaking, which is nearly all of the time. The thinker is this process, and it only exists between a finished question and a finished answer.

Here's the same split drawn out, for each of the two ways a call reaches the agent. Everything inside the AssemblyAI Voice Agent Platform block is the Responder; the customer-owned dialogue service on the right is the Thinker.

Web application over the WebSocket API

PSTN telephony with Twilio or Telnyx

Why split it this way

Because the two halves reward completely different kinds of engineering effort.

Turn-taking, noise handling, and speech synthesis are difficult, latency-bound, and, for almost every team, not where the product lives. Nobody's customers pick them because their barge-in is 40 ms tighter. But they will absolutely leave if the agent talks over people, so the work has to be done properly by someone.

The other half is the opposite. What the agent decides to say is the product. It's your domain logic, your data, your tone, your escalation rules, your compliance boundaries. And in this architecture it's a small, swappable slot.

Getting the input leg right is what makes that slot worth having. If transcription mishears the order number, your Thinker answers the wrong question perfectly. Fireflies ran into exactly this while building out their agent pipeline:

We were searching for the best realtime ASR model for our voice agent pipeline in Fireflies. The new Universal 3.5 Pro speech model from Assembly is best so far in terms of accuracy, latency and language switching.

Now the interesting part. Because the boundary between the two roles is a wire protocol rather than a plugin API, the Thinker can be almost anything: a hosted frontier model, a small fine-tune, a deterministic rules engine, a retrieval pipeline, an existing agent framework, or a human being typing. The Responder can't tell the difference and doesn't try.

The demo proves that the least flattering way possible. Two of its three "models" are a handful of regexes and a person clicking buttons. Both are valid.

The contract between them: chat completions

This is the whole trick, so it's worth being precise about it.

When you create an agent with POST /v1/agents, you can include an llm array. Here's the block from the demo's agent config:

"llm": [
  {
    "base_url": "${BYO_LLM_URL}",
    "model": "assembly-rules",
    "api_key": "${BYO_LLM_KEY}"
  }
]

Three fields, and only two of them mean what you'd expect. base_url is where the Responder sends its requests. api_key is a credential you invent yourself, so your endpoint can tell a real request from a stray one.

model is the interesting one, because it's a string you make up. assembly-rules isn't a model that exists anywhere. Nothing validates it and nothing looks it up. My endpoint advertises that name, and AssemblyAI hands it straight back to me in the model field of every request it sends. It's a label for my own routing, not an instruction to anybody else.

Which points at what this section is really about. Your endpoint doesn't need to be OpenAI, and there doesn't need to be a language model behind it at all. It needs to speak the schema. A handful of regexes, a lookup table, a state machine, an existing agent framework, a person typing: each one becomes a legitimate voice agent brain the moment it answers in that shape.

Three constraints are worth knowing before you wire one up:

base_url must be public HTTPS

Non-HTTPS URLs and private or loopback hosts are rejected. This surprises people until you notice the direction of travel: the Responder calls out to your endpoint. Your laptop is not reachable from AssemblyAI's servers, even for a browser session where the audio never leaves your machine. Local development needs a tunnel: the demo shells out to cloudflared and rewrites .env on every run, because the hostname changes each time.

api_key is write-only

It's encrypted at rest and never returned by any read. GET and list responses give you back base_url and model and nothing else. It isn't a secret AssemblyAI needs to read; it's a credential you set so you can check incoming requests. Which you should:

function authorized(req) {
  const header = req.headers.authorization || req.headers['x-api-key'] || ''
  return header.replace(/^Bearer\s+/i, '').trim() === TOKEN
}

One entry, no fallbacks

llm is an array, but only a single entry is accepted today. Fallback chains aren't supported yet, so if you want failover between providers it lives behind your endpoint, not in this field. (There's a pattern for that in how to add automatic LLM fallbacks to a voice pipeline.)

Then, every turn, the Responder does something completely ordinary. It sends POST {base_url}/chat/completions with the standard OpenAI request body: a messages array of system / user / assistant / tool roles, the model string you configured, and stream: true.

Worth pinning down, because it trips people up: the Responder appends /chat/completions to whatever you gave it. The /v1 you see in most examples is part of base_url, not part of the route. Point base_url at https://your-host/v1 and the route you implement is /v1/chat/completions; point it at https://your-host and it's /chat/completions. Same contract either way, which is why the OpenAI and LLM Gateway examples in the docs both carry the /v1 themselves.

That request is the entire integration surface. Match its shape and you're a voice agent brain, whatever you happen to be underneath.

Why streaming isn't optional

Streaming is a hard requirement here, and the reason is specific to voice.

Batch inference means waiting for the full completion, then speaking. Token streaming means speaking as tokens arrive. On a phone call the caller isn't waiting for your completion. They're listening to silence. What they experience is the gap before the first word, not the time to the last one.

So the metric that matters is time-to-first-token. Total completion time barely registers by comparison, because playback and generation overlap once you're streaming. (Streaming LLM responses in a voice pipeline has the full wire-level walkthrough, including how to flush sentences into TTS.)

Here's what I measured while building the demo, going through AssemblyAI's LLM Gateway:

Single runs on one machine, not a benchmark. But look at the last two rows, because the ordering there is the whole point: the same model, one config field apart, spans 810 ms to 3.9 seconds.

Reasoning models think before they answer, and on a call that thinking is dead air. Nobody waits 3.9 seconds in silence and assumes the line is fine. If your Thinker wraps a reasoning model, cap the effort:

const forVoice = (model) => (/^gpt-5/.test(model) ? { reasoning_effort: 'minimal' } : {})

Capping effort is the blunt fix, and it costs you the reasoning. There's a better one that costs you nothing, further down in what streamed tokens look like on the wire: you can start speaking before the model has finished thinking.

And a sharper trap: not every model streams at all. In the gateway, Gemini errors outright on stream: true and gpt-oss refuses it. That quietly disqualifies them from voice regardless of how good they are, and you won't find out from a quality benchmark. Check for streamed chat completions before you check for anything else.

What streamed tokens look like on the wire

The response side is standard OpenAI server-sent events: one event per delta, closing exactly once.

The thing worth emphasizing is how much freedom you have inside that shape. The Responder never asks what produced the tokens. It opens a request, reads whatever well-formed SSE comes back, and speaks it. How that stream gets filled is entirely yours.

Which makes the stream a pacing tool and not just a transport. You don't have to know the answer before you start talking. You can write a few words the instant the request lands, hold the connection open, and do the slow work while those words are already being spoken.

const send = (content) =>
  res.write(`data: ${JSON.stringify({ choices: [{ delta: { content } }] })}\n\n`)

// Costs nothing to produce, so the caller hears audio almost immediately.
send('Let me pull that up.')

// Same request, same open stream. This runs while those words are being spoken.
const answer = await slowModel(messages)
for await (const token of answer) send(token)

res.write('data: [DONE]\n\n')
res.end()

That's the real answer to the 3.9-second reasoning model above. Capping effort trims the wait; this takes it out of the caller's experience altogether. The acknowledgement costs nothing to generate, so audio starts almost immediately, and the expensive call happens behind speech that's already playing. One turn, one stream, no extra round trip.

From there it generalizes about as far as you care to take it. Answer from cache when you have one and fall through to a model when you don't. Race a cheap model against a good one, speak the cheap one, and let the good one take over mid-reply if it lands in time. Say "one moment" while a tool call resolves, then carry on with the result. Put moderation in front of generation and never write the tokens that fail it. Decide the right move is silence, and close the stream having said nothing at all.

None of that is a feature you have to ask for. It falls out of owning the writes while the Responder only reads. The rest of this section is the wire format that makes it concrete.

The demo's model mode is one concrete version: it proxies to the LLM Gateway and forwards the upstream response unmodified, chunk by chunk, as it arrives.

res.writeHead(200, { 'content-type': 'text/event-stream', connection: 'keep-alive' })
for await (const part of upstream.body) {
  if (ttft === null) ttft = since()
  res.write(part)
}
res.end()

Content shows up as it's actually produced. A chunk lands and goes out immediately, the connection sits open for however long the next piece takes, and the whole thing closes once, at the end, with finish_reason: "stop" followed by [DONE]:

data: {"choices":[{"delta":{"role":"assistant","content":""}}]}

data: {"choices":[{"delta":{"content":"Sorry "}}]}

… more tokens, each forwarded the instant it's ready …

data: {"choices":[{"delta":{"content":"about "}}]}

data: {"choices":[{"delta":{"content":"that."}}]}

data: {"choices":[{"delta":{},"finish_reason":"stop"}]}

data: [DONE]

The demo's other two modes are the exception, and it's worth being explicit about why. In rules and you mode, the endpoint already has the entire reply as one finished string before it sends a single byte: a regex matched instantly, or a human already clicked send. There's nothing left to stream, so it re-splits the finished string into words and writes them back to back with no delay, purely to preserve the shape:

for (const word of text.match(/\S+\s*/g) ?? []) {
  res.write(`data: ${chunk(model, { content: word })}\n\n`)
}
res.write(`data: ${chunk(model, {}, 'stop')}\n\n`)
res.write('data: [DONE]\n\n')

That's a stand-in for illustration, not the general case. A real Thinker keeps the connection open and interleaves writes with generation over time. Don't copy the word-splitter into production and think you've implemented streaming.

Tool calls ride the same channel: a tool_calls delta in the stream, in OpenAI's function-calling format:

res.write(`data: ${chunk(model, {
  role: 'assistant',
  content: '',
  tool_calls: [{
    index: 0,
    id: 'call_byo_' + turns,
    type: 'function',
    function: { name: call.name, arguments: JSON.stringify(call.arguments) },
  }],
})}\n\n`)
res.write(`data: ${chunk(model, {}, 'tool_calls')}\n\n`)

No separate integration surface. Whatever tools your agent has, they arrive and depart through the same stream.

Building the minimal Thinker

Here's the smallest thing that satisfies the contract.

1. Stand up an HTTP server with one route: POST /chat/completions, hanging off whatever path your base_url ends in. That's the only endpoint the Responder will ever hit.
2. Read the incoming messages and pull out the caller's last utterance.
3. Decide a reply. The demo does this three interchangeable ways behind the identical endpoint, switchable mid-call:
   • rules: about 20 regexes over a small state machine (a food-delivery claim object tracking issue, photos, verification, and outcome). Answers in a fraction of a millisecond.
   • model: proxies the same request to the LLM Gateway, swapping in a real model.
   • you: every request blocks on a human clicking a quick-reply button or typing free text. Including an explicit say nothing option, because silence is a real answer on a call and no model will ever give it to you.
4. Stream the decided text back as SSE chunks.
5. Publish an agent whose llm.base_url points at the endpoint, via POST /v1/agents.

Those three modes are illustrations of the contract, not its boundary. A production endpoint behind base_url can do anything that emits valid chat-completions SSE: race multiple models, fall back from one to another, mix a state machine with a model for the parts that must be deterministic, put retrieval or moderation or business logic in front of generation. The Responder only ever sees the resulting stream.

One detail from you mode that's worth stealing regardless, and it's the speak-first pattern again from the other direction: the agent gives up on a request that has gone quiet for a few seconds and re-sends the turn. So the endpoint answers the socket immediately with an empty assistant delta and keeps it warm with a heartbeat while the human types. Writing something early protects you twice over. It starts audio sooner, and it stops the Responder concluding your endpoint has died.

Managed model vs. custom LLM is one field

This is the part I find genuinely satisfying.

In the demo repo, agents/food-claim.jsonc and agents/food-claim-managed.jsonc are identical files except that the managed twin has no llm block at all. Delete those seven lines and AssemblyAI's own model answers instead. Send "llm": [] on a PUT and an existing agent reverts to managed mid-flight.

Everything else in the config is Responder-owned and identical either way: system_prompt, greeting, voice.voice_id, and the whole input block: transcription_mode, language_codes, voice_focus, keyterms, turn_detection.*.

Which means swapping the Thinker changes what gets said and never how the call behaves. Turn detection thresholds, interruption handling, and voice all stay exactly where they were, because they belong to the other role. The demo makes this literal: click the Thinker box on screen mid-call and the same conversation carries on with a different brain, same voice, same timing, same photos.

Managed and custom aren't two products with a migration between them. They're one field.

Two ways a call reaches the agent

Worth saying up front: this is a transport choice, not an architecture choice. The contract above doesn't change either way.

WebSocket, for browsers and apps

The client opens wss://agents.assemblyai.com/v1/ws with a short-lived token, sends session.update with { agent_id } to say which agent answers, then streams raw PCM frames as input.audio. Back come the session events the pipeline is built from: session.ready, transcript.user.delta, transcript.user, transcript.agent, reply.audio, and reply.done, with status: "interrupted" when the caller barges in. Which agent answers is a client-side decision, made at connect time.

SIP, for phone numbers

A number is bound to one agent. You point a SIP trunk's origination URI at sip:sip.assemblyai.com, import the number with POST /v1/phone-numbers/import, and attach an agent with PUT /v1/phone-numbers/{number}/agent. (Bring your own number has the Twilio walkthrough.)

That binding changes how you swap Thinkers. There's no client sending session.update on a phone call, so switching brains means moving the number to a different agent, a server-side operation, not a toggle.

Telephony also unlocks one tool shape the browser can't have: keypad collection. Declare dtmf_collected_arguments on a tool and the Responder gathers the digits itself, with min_digits, max_digits, a terminator, and a sensitive flag. Marked sensitive, the digits never reach your model or the transcript at all, which is useful for anything card- or PIN-adjacent. Try to declare it on a WebSocket agent and the API refuses the session outright, with a message explaining that the value could never be collected. I found that out the direct way.

Rough gut-check: browser demos and in-product voice want WebSocket. Support lines, IVR replacement, and outbound calling want SIP.

What the Thinker never sees

Go back through the Responder's job list. None of it is visible to your endpoint. Your /chat/completions route sees finished turns as messages and nothing else: no audio, no partial transcripts, no VAD events, no knowledge that the caller interrupted.

That's the design working. But it does mean two things need planning for.

Other modalities have to be described, not passed. In the demo, a caller sends a photo mid-call. The gateway silently drops image content, so a custom model can't see it either. What actually happens is a system message: "The caller has just sent a photo of their order. It shows X." Text in, text out. If your product involves anything but speech, that translation is your job. (And if you're tempted to have the model describe the image instead: I passed a solid red square through four models and got back "blue," "brown," and "white.")

Cross-turn state lives on your side of the wire. A claim being assembled, a review being collected, a verification that already happened: the Responder isn't holding any of it. The demo's claim and review objects are the whole memory of those conversations, and they live in the endpoint's process. The Thinker's job is deliberately narrow; narrow means stateless unless you make it otherwise.

Lessons for anyone plugging in their own engine

• Streaming is a hard requirement. Verify your model and provider actually support stream: true before you wire anything. Some refuse, and quality benchmarks won’t tell you.
• Watch TTFT, not just quality. A smarter model with a 3-second first token loses to a good-enough model with a 500 ms one. On a call, that isn’t close.
• Turn reasoning down or off. Same model, one field, 810 ms versus 3.9 seconds.
• Your Thinker doesn’t have to be ML. A rules engine, a lookup table, a state machine, a human. Anything returning text in the right shape is a legitimate model. Genuinely useful for demos, fallback paths, and narrow domains where a state machine beats a general model.
• Treat your endpoint as public the moment it’s live. Check the api_key the agent was published with on every request. A quick tunnel URL is a public URL.
• Tool calls need no separate integration. They ride the same chat.completions stream, keypad collection included.
• Keep your own conversation state. The Responder isn’t remembering anything for you.

The model is the replaceable part

The Voice Agent API's real trick isn't the model it ships with. It's that the model is an HTTP contract you can replace, and everything that makes a phone call feel like a phone call is handled for you either way.

That's a more unusual property than it sounds. Most of the interesting work in voice right now is happening in the think step: agent frameworks, retrieval, tool orchestration, small fine-tunes that beat frontier models on one narrow job. An architecture where that step is a URL rather than a vendor decision is one you can keep building against as the field moves. The hard, unglamorous, latency-bound parts stay solved.

If you want the hands-on version, the demo repo is dan-ince-aai/voice-agent-byo-llm-demo. Clone it, cp .env.example .env, add an AssemblyAI API key, npm start, and take a live call, flipping between the rules engine, a gateway model, and typing the replies yourself, watching the Thinker and Responder light up independently on screen as you go.

Answering your own voice agent by hand is a strange experience. It also makes the split obvious in about thirty seconds, which is more than this post managed in four thousand words.

For the reference material: Connect your own LLM covers the contract, and Create an agent has every field.

Frequently asked questions