Build a voice agent without Pipecat or LiveKit

If you've started building a voice agent in the last year, you've hit this question fast: do I need Pipecat or LiveKit?

The internet says yes. Every tutorial reaches for an orchestration framework before it writes a line of agent logic. And for good reason — those frameworks are genuinely excellent, and AssemblyAI ships drop-in plugins for both. If you're already running one, keep running it.

But "do I need a framework" is the wrong first question. The real question is: what moves the audio, and what wires the pipeline together? Answer those two and the framework question answers itself. A lot of the time, the honest answer is: you don't need one.
This post is an architecture overview of the framework-free path. We'll cover the four things people actually mean when they ask about going without Pipecat or LiveKit:

• What a voice pipeline framework actually does for you (and what it costs).
• The transport question everyone conflates — SIP vs. WebSocket.
• Connecting a phone number with Twilio.
• Whether the framework-free architecture holds up at enterprise scale.

Let's start with what you'd be removing.

What a framework actually does for you

A voice agent has to do a few things in a tight loop, every few hundred milliseconds: turn speech into text, decide what to say, turn that back into speech, and move audio in and out — while handling interruptions when the caller talks over the bot.

Historically, each of those was a different vendor. Speech-to-text from one provider, an LLM from another, text-to-speech from a third. Something has to sit in the middle and conduct: route partial transcripts to the LLM, stream the LLM's tokens to the TTS engine, manage turn-taking, and handle barge-in. That conductor is what Pipecat and LiveKit Agents give you, plus a transport layer to get audio to and from the user.

That's real work, and the frameworks do it well. The thing is, most of it is only hard work when the pieces don't know anything about each other.

Here's the cost side, because it's easy to miss when you're following a quickstart. A framework is another dependency in your stack — one more thing to deploy, version, monitor, and reason about when something goes wrong at 2am. You're still wiring three model vendors together, so you still own three sets of API keys, three billing relationships, three failure modes, and three latency budgets that stack. The framework hides that complexity. It doesn't remove it.

So the question becomes: what if the three stages came from one vendor, and each one already handled its own hard part?

The architecture without a framework

Here's the shift. AssemblyAI now covers every stage of the cascade behind a single API key:

• Speech-to-text: Universal-3.6 Pro Realtime over a WebSocket. Turn detection is built into the model, so you get a Turn event with end_of_turn when the caller is done, instead of running a separate turn-detection plugin.
• The LLM: LLM Gateway, an OpenAI-compatible endpoint in front of Claude, GPT, Gemini and other models. Tool calling lives in this request.
• Text-to-speech: the AssemblyAI Text-to-Speech API over a WebSocket, one session per conversation. A cancel() stops speech the moment the caller cuts in.

When turn-taking lives in the speech-to-text model and barge-in is one call on the text-to-speech session, what's left to orchestrate is a loop: final transcript in, LLM reply out, reply spoken, stop if interrupted. That fits in one file.
Compare the two topologies.

With a framework (cascading, multi-vendor):

         ┌─────────── your orchestration framework ────────────┐
Caller ─▶│ transport ─▶ STT vendor ─▶ LLM vendor ─▶ TTS vendor │─▶ Caller
         └─────────────────────────────────────────────────────┘
          (you deploy, scale, and monitor this whole box)

Without a framework (one vendor, your own loop):

Caller ─▶ your app ─┬─▶ Universal-3.6 Pro Realtime   WebSocket: audio in, Turn events out
                    ├─▶ LLM Gateway                  HTTPS: transcript in, reply and tool calls out
                    └─▶ AssemblyAI Text-to-Speech    WebSocket: text in, audio out ─▶ Caller
          (one API key, one bill, and a loop you can read in one sitting)

Universal-3.6 Pro Realtime is the speech layer, and it's the most accurate realtime model AssemblyAI ships: a 5.19% mean word error rate and a 14.4% entity error rate on AssemblyAI's English voice-agent benchmark, across 32 languages. On the way out, AssemblyAI-TTS-1 is built for conversational voice agents: it was trained on millions of hours of audio, including real conversational data, and it speaks six languages with thousands of voices to choose from. It delivers frontier-level quality, runs faster and costs less than other production voice agent TTS models, and has unlimited concurrency. It's framed as invisible infrastructure on purpose: one key, one bill, one set of logs.

What you build instead of an orchestration layer is the loop itself. Let's write it — first on your own machine, then over the phone.

The whole loop, in one file

Install the Python SDK and a small audio library, set your key, and run the agent:

pip install -U assemblyai sounddevice   # assemblyai 1.8.2 or newer
export ASSEMBLYAI_API_KEY=your_key_here
python agent.py

Here's the full agent. The three numbered sections are the three stages; everything else is the loop that joins them:

# agent.py: speech-to-text -> LLM -> text-to-speech, with no orchestration framework.
import json
import os
import queue
import threading

import assemblyai as aai
import sounddevice as sd
from assemblyai.streaming.v3 import (
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    TurnEvent,
)

API_KEY = os.environ["ASSEMBLYAI_API_KEY"]  # one key for STT, LLM Gateway and TTS
LLM_MODEL = "claude-haiku-4-5-20251001"  # any model from the LLM Gateway model list
MIC_RATE = 16000

user_turns = queue.Queue()
agent_speaking = threading.Event()
barge_in = threading.Event()

# 1. Speech-to-text: Universal-3.6 Pro Realtime decides when the caller is done.
def on_turn(client: RealTimeTranscriber, event: TurnEvent):
    words = event.transcript.split()
    if agent_speaking.is_set() and len(words) >= 2:
        barge_in.set()  # the caller is talking over the agent
    if event.end_of_turn and words:
        user_turns.put(event.transcript)

stt = RealTimeTranscriber(api_key=API_KEY)
stt.on(RealTimeEvents.Turn, on_turn)
stt.connect(RealTimeParameters(speech_model="universal-3-6-pro", sample_rate=MIC_RATE))

def microphone():
    with sd.RawInputStream(samplerate=MIC_RATE, channels=1, dtype="int16") as mic:
        while True:
            data, _ = mic.read(MIC_RATE // 20)  # 50 ms chunks
            yield bytes(data)

threading.Thread(target=stt.stream, args=(microphone(),), daemon=True).start()

# 2. The LLM, through LLM Gateway. Tool calling lives in this request.
llm = aai.LLMGateway(api_key=API_KEY)
TOOLS = [{
    "type": "function",
    "function": {
        "name": "lookup_order",
        "description": "Look up an order's status by order number.",
        "parameters": {
            "type": "object",
            "properties": {"order_id": {"type": "string"}},
            "required": ["order_id"],
        },
    },
}]
messages = [{"role": "system", "content": (
    "You are a friendly support agent on a voice call. Reply in one or two "
    "short spoken sentences, with no markdown."
)}]

def lookup_order(order_id):
    return {"order_id": order_id, "status": "shipped", "eta": "Thursday"}  # your system here

def think(text):
    messages.append({"role": "user", "content": text})
    while True:
        reply = llm.chat.completions.create(
            model=LLM_MODEL, messages=messages, tools=TOOLS, max_tokens=300,
        ).choices[0].message
        if not reply.tool_calls:
            messages.append({"role": "assistant", "content": reply.content or ""})
            return reply.content or ""
        messages.append({"role": "assistant", "content": reply.content, "tool_calls": [
            {"id": c.id, "type": "function",
             "function": {"name": c.function.name, "arguments": c.function.arguments}}
            for c in reply.tool_calls
        ]})
        for call in reply.tool_calls:
            args = call.function.arguments
            args = json.loads(args) if isinstance(args, str) else args
            messages.append({"role": "tool", "tool_call_id": call.id,
                             "content": json.dumps(lookup_order(**args))})

# 3. Text-to-speech: one session for the whole conversation.
config = aai.TTSConfig(
    voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3",  # Audrey
    language="english",
)
speaker = sd.RawOutputStream(samplerate=24000, channels=1, dtype="int16")  # native TTS rate

try:
    with aai.TTSSession(config, api_key=API_KEY) as tts, speaker:
        print("Listening. Use headphones so the agent doesn't hear itself.")
        while True:
            try:
                text = user_turns.get(timeout=60)
            except queue.Empty:
                tts.keep_alive()  # the session closes after two idle minutes
                continue
            print(f"you:   {text}")
            answer = think(text)
            print(f"agent: {answer}")

            barge_in.clear()
            agent_speaking.set()
            for chunk in tts.speak(answer):  # raw 24 kHz 16-bit PCM
                if barge_in.is_set():
                    tts.cancel()  # stop the reply the moment the caller cuts in
                    break
                speaker.write(chunk)  # blocks while it plays, so barge-in stays responsive
            agent_speaking.clear()
except KeyboardInterrupt:
    pass
finally:
    stt.disconnect(terminate=True)

Notice there's no pipeline graph, no service registry, no turn-detection plugin to configure. Three things carry the whole design:

• The speech-to-text model owns turn-taking. A Turn event with end_of_turn set is a finished user turn, so the loop just waits on a queue. Leave the turn-silence settings at their defaults unless you have a reason to change them.
• The LLM call owns tools. Tool definitions go in the request, the model asks for lookup_order, your code runs it, and the result goes back in the next request. There's no separate tool protocol to learn.
• The text-to-speech session owns playback control. One session stays open for the whole conversation. speak() sends a reply and yields its audio as raw 24 kHz 16-bit PCM, and cancel() stops it mid-sentence. The session closes after two idle minutes, which is what the keep_alive() is for.

Two practical notes. Use headphones: a terminal app gets no echo cancellation, so the microphone would hear the agent and interrupt it. And LLM Gateway uses the same API key as everything else, but it may need a paid account; if it isn't available on yours, point think() at your own OpenAI or Anthropic client and leave the rest as it is. For lower latency, stream the LLM's tokens into the session with generate() and flush() instead of speak(), so speech starts on the first complete sentence. The text-to-speech quickstart shows that pattern.

The transport question: SIP vs. WebSocket

Here's where most of the confusion lives, so let's settle it.
Audio has to physically move between the user and your agent. There are three common ways to move it:

• WebRTC — the browser-and-app real-time standard. It handles NAT traversal, jitter buffering, and echo cancellation. This is LiveKit's and Daily's home turf, and it's the right tool for multi-party rooms, video, and rich client SDKs.
• SIP — the signaling protocol of the telephone network. If you're terminating raw PSTN calls or running your own telephony infrastructure, you're in SIP territory.
• WebSocket — a plain, bidirectional socket. No media server, no SFU, no signaling dance. You send audio frames, you receive audio frames.

AssemblyAI takes and returns audio over a WebSocket. That's the entire transport story for speech-to-text and text-to-speech alike.

"But my users are on the phone — don't I need SIP?" This is the part people get wrong. You don't speak SIP yourself, and you don't stand up a WebRTC media server either. Your telephony provider does that for you. Twilio terminates the PSTN call, handles the SIP side, and hands you the call's audio over — you guessed it — a WebSocket. So the agent is WebSocket on the user side and WebSocket on the AssemblyAI side. It's sockets all the way through.

That's why you can skip the framework's transport layer for telephony: the carrier already converted the hard part into a WebSocket before it reaches you. Your job is to bridge sockets.

Connecting a phone number with Twilio

Let's make the phone case concrete, because it's the one that sounds like it should need the most machinery and actually needs the least.
The end-to-end path is four hops:

Caller ↔ Twilio Media Streams ↔ Your server ↔ AssemblyAI (STT + LLM Gateway + TTS)

Twilio handles the phone network. Your server is a thin bridge that runs the loop. AssemblyAI handles speech in both directions and the LLM in between. And here's the detail that removes a whole class of pain: Twilio streams 8 kHz G.711 μ-law, Universal-3.6 Pro Realtime accepts that encoding directly (pcm_mulaw at 8000 Hz), and a text-to-speech session opened with pcm_mulaw at 8000 Hz returns audio already in Twilio's format. The model's native output is 24 kHz 16-bit PCM; the server resamples and encodes it before it reaches you. So your bridge forwards audio as-is, with no transcoding or resampling in your code.
When a call comes in, Twilio hits a webhook on your server. You return TwiML that opens a Media Streams WebSocket. Here's the top of the Node.js server, with PUBLIC_HOSTNAME set to your server's public address:

// server.mjs: npm install assemblyai express ws
import http from "node:http";
import express from "express";
import { WebSocketServer } from "ws";
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });
const app = express();

// Twilio calls this webhook when the number rings. The TwiML opens a Media Stream.
app.post("/twiml", (req, res) => {
  const host = process.env.PUBLIC_HOSTNAME.replace(/^https?:\/\//, "");
  res.type("text/xml").send(
    `<Response>
  <Connect>
    <Stream url="wss://${host}/media-stream" />
  </Connect>
</Response>`,
  );
});

const server = http.createServer(app);
const wss = new WebSocketServer({ server, path: "/media-stream" });
server.listen(process.env.PORT || 8080);

When Twilio opens that Media Streams socket, your server opens two AssemblyAI sessions for the call — speech-to-text and text-to-speech — both set to μ-law at 8 kHz to match Twilio:

wss.on("connection", (twilio) => {
  let streamSid = null;
  let speaking = false;          // is the agent talking right now?
  let replies = Promise.resolve(); // replies are spoken one after another
  const messages = [{
    role: "system",
    content: "You are a friendly support agent on a phone call. Reply in one or two short spoken sentences.",
  }];

  // Speech-to-text: Twilio's 8 kHz mu-law goes in exactly as it arrives.
  const stt = client.streaming.transcriber({
    speechModel: "universal-3-6-pro",
    encoding: "pcm_mulaw",
    sampleRate: 8000,
  });

  // Text-to-speech: one session per call, already in Twilio's format.
  const tts = client.tts.session({
    voiceId: "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", // Audrey
    language: "english",
    encoding: "pcm_mulaw",
    sampleRate: 8000,
  });

  const ready = Promise.all([stt.connect(), tts.connect()]);
  const keepAlive = setInterval(() => tts.keepAlive(), 60_000); // TTS idles out after 2 minutes

Once both sessions are connected, the whole bridge is a forwarding rule in each direction plus one for interruptions. The LLM call sits in the middle, with its tools:

  // Caller -> speech-to-text: forward each Twilio media frame unchanged.
  twilio.on("message", async (data) => {
    const msg = JSON.parse(data.toString());
    if (msg.event === "start") streamSid = msg.start.streamSid;
    if (msg.event === "media" && msg.media.track === "inbound") {
      await ready;
      stt.sendAudio(Buffer.from(msg.media.payload, "base64"));
    }
  });

  // Speech-to-text -> LLM -> text-to-speech -> caller.
  stt.on("turn", (turn) => {
    const text = turn.transcript.trim();
    if (!text) return;
    // Barge-in: the caller talks over the agent. Stop the speech and clear Twilio's buffer.
    if (speaking && (turn.end_of_turn || text.split(/\s+/).length >= 2)) {
      tts.cancel();
      twilio.send(JSON.stringify({ event: "clear", streamSid }));
    }
    if (turn.end_of_turn) replies = replies.then(() => respond(text)).catch(console.error);
  });

  async function respond(text) {
    messages.push({ role: "user", content: text });
    const answer = await think(messages); // LLM Gateway, with tool calls
    speaking = true;
    try {
      for await (const chunk of tts.speak(answer)) {
        twilio.send(JSON.stringify({
          event: "media",
          streamSid,
          media: { payload: Buffer.from(chunk).toString("base64") },
        }));
        await sleep(chunk.byteLength / 8); // pace at the wire rate: 8 bytes of mu-law per ms
      }
    } finally {
      speaking = false;
    }
  }

  twilio.on("close", async () => {
    clearInterval(keepAlive);
    await ready.catch(() => {});
    await Promise.allSettled([stt.close(), tts.close()]);
  });
});

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

// The LLM, through LLM Gateway (OpenAI-compatible). Tools are part of the request.
const TOOLS = [{
  type: "function",
  function: {
    name: "lookup_order",
    description: "Look up an order's status by order number.",
    parameters: {
      type: "object",
      properties: { order_id: { type: "string" } },
      required: ["order_id"],
    },
  },
}];

const lookupOrder = ({ order_id }) => ({ order_id, status: "shipped", eta: "Thursday" }); // your system here

async function think(messages) {
  while (true) {
    const res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
      method: "POST",
      headers: { authorization: process.env.ASSEMBLYAI_API_KEY, "content-type": "application/json" },
      body: JSON.stringify({ model: "claude-haiku-4-5-20251001", messages, tools: TOOLS, max_tokens: 300 }),
    });
    const message = (await res.json()).choices[0].message;
    if (!message.tool_calls?.length) {
      messages.push({ role: "assistant", content: message.content ?? "" });
      return message.content ?? "";
    }
    messages.push({ role: "assistant", content: message.content, tool_calls: message.tool_calls });
    for (const call of message.tool_calls) {
      const args = JSON.parse(call.function.arguments || "{}");
      messages.push({ role: "tool", tool_call_id: call.id, content: JSON.stringify(lookupOrder(args)) });
    }
  }
}

That's a phone agent with no SIP stack, no media server and no orchestration framework: a webhook, a socket bridge and one loop. Two details are worth copying. The reply audio is paced at the wire rate, 8,000 bytes of μ-law per second, so Twilio's buffer stays short and a barge-in stops the agent almost at once. And on barge-in the bridge does two things: cancel() stops synthesis on the AssemblyAI side, and Twilio's clear message drops anything Twilio has already queued. The same pattern applies to other carriers — Twilio, Vonage, and the rest all hand you a media WebSocket.

What if you only want to swap one layer?

Not everyone is building net-new. Maybe you already have an LLM and a TTS voice you love, and you only want better transcription — the rip-and-replace-one-piece path. You can do that without a framework too.
Connect straight to the streaming speech-to-text WebSocket and keep the rest of your stack exactly as it is:

import json
from urllib.parse import urlencode
import websocket  # pip install websocket-client

API_KEY = "YOUR_API_KEY"
# Streaming uses speech_model (singular). universal-3-6-pro is the flagship realtime model.
params = {"speech_model": "universal-3-6-pro", "sample_rate": 16000}
ENDPOINT = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(params)}"

def on_message(ws, message):
    data = json.loads(message)
    if data.get("type") == "Turn":
        if data.get("end_of_turn"):
            print("Final:", data.get("transcript"))   # → send to your own LLM
        else:
            print("Partial:", data.get("transcript"), end="\r")
    elif data.get("type") == "SpeechStarted":
        print("Barge-in: user started speaking")       # → interrupt your TTS

# Authenticate with the raw key in the header — no "Bearer" prefix.
ws = websocket.WebSocketApp(ENDPOINT, header={"Authorization": API_KEY},
                            on_message=on_message)
ws.run_forever()
# Stream 16 kHz PCM16 audio in ~50 ms chunks as binary WebSocket frames.

You hold the loop — final transcripts to your LLM, tokens to your TTS, SpeechStarted or the first words of a new turn to trigger barge-in. If you want a longer build, our real-time transcription in Python walkthrough goes step by step. The same goes the other way: if your speech-to-text is settled and you only want a new voice, the text-to-speech interfaces drop into an existing loop on their own.

A couple of details that trip people up: streaming uses speech_model (singular) — that's the opposite of the pre-recorded API, which takes a speech_models array for fallback routing. And universal-3-6-pro handles turn detection in the model, so end_of_turn_confidence_threshold is a no-op on it.

Does this hold up at enterprise scale?

A fair pushback: "a hand-rolled loop is fine for a demo, but does it survive production?" This is where keeping the pipeline with one vendor actually helps instead of hurts.

One vendor, fewer seams. A three-vendor cascade has three accounts that can rate-limit you, three status pages to watch, and three sets of logs to stitch together during an incident. Here there are still three stages, but one API key, one bill, one support channel, and session IDs you can quote in one place. When you're the one carrying the pager, that matters more than any benchmark.

Concurrency that scales with you. Both streaming services count the new sessions you open each minute, and sessions already running don't count against that limit, so a busy contact center isn't capped by concurrent calls. See the text-to-speech limits for the session rules, including the one-hour maximum and the two-minute idle timeout.

Pricing you can forecast. Each stage is metered on its own unit: streaming speech-to-text by the hour ($0.45/hr for Universal-3.6 Pro Realtime, with context and keyterm prompting included), LLM Gateway by the tokens of the model you choose, and text-to-speech by input characters at $15 per 1M characters. All three land on one bill, and current rates are on the pricing page.

The compliance and residency boxes. SOC 2 Type 2, EU data residency for all three stages (streaming.eu.assemblyai.com for speech-to-text, llm-gateway.eu.assemblyai.com for the LLM, which serves Claude and most Gemini models in the EU, tts.eu.assemblyai.com for text-to-speech), and a self-hosted speech-to-text option if the audio can't leave your VPC. For teams handling protected health information (PHI), AssemblyAI offers a Business Associate Addendum (BAA); confirm which products your BAA covers before you route PHI through them.

None of this is a reason to avoid a framework — plenty of large deployments run Pipecat or LiveKit happily. It's a reason to know that the framework-free path isn't a toy. For an honest look at where any voice agent architecture starts to strain, see where voice agent stacks start showing their limits.

When you should use Pipecat or LiveKit

Skipping the framework is the right call for a large share of voice agents — especially phone agents, support bots, and anything where the job is a clean listen, think, speak loop. But not all of them. Reach for Pipecat or LiveKit when:

• You need real WebRTC features — multi-party rooms, video alongside voice, screen share, or polished mobile and web client SDKs. This is LiveKit's core strength, and a WebSocket bridge won't replicate it.
• You want best-of-breed per stage across vendors — a specific third-party voice, a self-hosted LLM, or a custom turn-detection model, mixed and matched per use case. Frameworks are built for exactly this kind of swapping.
• You already run one in production. If Pipecat or LiveKit is in your stack, don't rip it out. AssemblyAI ships first-class speech-to-text plugins for both, so you can run Universal-3.6 Pro Realtime inside the framework you already trust, and the text-to-speech API has a LiveKit plugin too.
• You need fine-grained control over the pipeline graph — custom processors, intricate branching, frame-level manipulation.

The decision is less "framework vs. no framework" and more "how much of the pipeline do I actually need to control."

Choosing your path

The takeaway

The voice agent ecosystem defaulted to orchestration frameworks for a sound reason: for years, building an agent meant gluing three vendors together, and that glue is real engineering. Frameworks made the glue manageable.

But the default quietly assumes each stage knows nothing about the others. Once the speech-to-text model decides when the caller is done, the LLM request carries the tools, and the text-to-speech session stops on a single cancel() — and your carrier hands you telephony audio over a WebSocket too — the orchestration layer doesn't need a framework. What's left is a socket on each side and one loop to join them.

So before you add Pipecat or LiveKit to your next voice agent, ask the two questions that actually matter: what's moving my audio, and how much of the pipeline do I need to control? If the answers are "a WebSocket" and "just the loop," you've already got your architecture.

Frequently asked questions

Do you need Pipecat or LiveKit to build a voice agent?

No. Pipecat and LiveKit orchestrate a multi-vendor pipeline — they wire together separate speech-to-text, LLM, and text-to-speech services and manage turn-taking and transport between them. When the speech-to-text model handles turn detection and the text-to-speech session handles barge-in, as Universal-3.6 Pro Realtime and the AssemblyAI Text-to-Speech API do, the remaining orchestration is a short loop you can write yourself. Reach for a framework when you need WebRTC features like multi-party rooms or video, or fine-grained control over each pipeline stage.

Do voice agents use SIP or WebSocket for audio transport?

Voice agents can move audio over WebSocket, WebRTC, or SIP, and the three solve different problems. AssemblyAI takes and returns audio over a plain WebSocket — no media server or SFU required. For phone calls you don't implement SIP yourself: your telephony provider (such as Twilio) terminates the PSTN and SIP side and hands you the call's audio over a WebSocket, so the whole agent ends up being WebSocket on both ends.

What does a framework-free voice agent need to handle itself?

Four things: forwarding audio to the speech-to-text socket, sending each finished user turn to the LLM, speaking the reply on the text-to-speech session, and stopping that speech when the caller interrupts. With Universal-3.6 Pro Realtime, turn detection arrives as a Turn event with end_of_turn set, tool calling happens inside the LLM request, and barge-in is a cancel() on the text-to-speech session, so each of the four is a few lines of code.

Can you build a Twilio voice agent without a framework?

Yes. Twilio terminates the phone call and streams 8 kHz G.711 μ-law audio to your server over a Media Streams WebSocket. Universal-3.6 Pro Realtime accepts that audio as-is with encoding=pcm_mulaw and sample_rate=8000, and AssemblyAI Text-to-Speech returns μ-law at 8 kHz when you open the session with the same settings, so there's no transcoding in your code. The whole bridge is a webhook that returns TwiML, two AssemblyAI sessions per call, and a loop that forwards audio in each direction and sends Twilio a clear on barge-in.

Does a voice agent without an orchestration framework scale to production?

Yes. AssemblyAI's streaming services count new sessions per minute, and sessions already running don't count against that limit, so concurrent calls aren't the ceiling. Keeping all three stages with one vendor also reduces operational risk: one API key, one bill, one set of logs, and session IDs you can quote to one support team. Speech-to-text, LLM Gateway and text-to-speech each have an EU endpoint, and speech-to-text has a self-hosted option for teams with strict data requirements.

How do I build a voice agent without Pipecat or LiveKit?

Open a Universal-3.6 Pro Realtime streaming session and send it your audio; when a Turn event arrives with end_of_turn set, send the transcript to an LLM through LLM Gateway (with your tools in the request), then speak the reply on an AssemblyAI Text-to-Speech session that stays open for the whole conversation. If the caller starts talking while the agent is speaking, call cancel() on the session. The same loop works from your own machine, from a server, or over the phone through Twilio.