The ongoing engineering cost of a multi-vendor voice agent stack

You shipped your voice agent. Users are in it. Things are working.

Then Deepgram pushes a breaking change to their streaming API. Your LLM provider has a latency spike. Your TTS vendor silently renames their voice IDs—the ones your configuration has been referencing for six months. None of these failures are yours. All of them are your problem.

That's week two of a multi-vendor voice agent stack—and nobody tells you about it during the evaluation phase. You evaluated each vendor in isolation, got their demos working, built the connection between them, and shipped. What you didn't evaluate was the operational surface area of running four vendors as though they were one product. The evaluation showed you what each component could do. It didn't show you what it costs to keep them working together.

This post is about that cost. Not in the abstract—in the specific: the dashboards, the invoices, the log streams, the on-call rotations, and the engineering time that accumulates around a multi-vendor stack even when nothing is obviously wrong.

What you're actually signing up for

A typical voice agent stack has four distinct components: speech recognition (STT), a language model (LLM), text-to-speech (TTS), and an orchestration layer to coordinate them. In most implementations, each comes from a separate vendor. That means:

SurfaceWhat it adds to your team's loadFour onboarding flowsA new account, new credentials to rotate, new docs to navigate, and a new rate-limit model to understand per vendor—often seven flows before you write a meaningful line of product code.Four billing relationshipsSTT bills per audio minute, LLM per input/output token, TTS per character or request. Modeling a single 5-minute call needs three pricing pages and a spreadsheet of assumptions that break under real traffic.Four observability contextsWhen a turn goes wrong, you debug across four log streams that don't share context. Correlating them is your job.Four failure surfacesUptime is the product of your vendors' uptimes. Four vendors at 99.9% each yields 99.6% before you ship a line of your own code.Four release cyclesEvery model update, deprecated endpoint, or streaming schema change lands on the vendor's calendar, not yours.

None of these costs appear in a vendor demo. All of them show up in production.

The operational burden is the architecture

Here's what gets missed in discussions about cascaded versus unified voice stacks: the hard part is rarely the code you write to connect the components. The hard part is the ongoing coordination the architecture requires—before launch, at launch, and on every ordinary day afterward when nothing is visibly broken.

Take interruption handling as a concrete example. Getting a voice agent to stop speaking when a user interrupts is conceptually simple: detect that the user is talking, cancel the current TTS output, and resume listening. In practice, across a cascaded stack, you manage state across all three components simultaneously. You flush the audio buffer, discard any TTS requests already in flight at the vendor level, signal the LLM to abandon the current generation, and ensure the STT layer resumes cleanly—in the right sequence, with the right timing, without any of the components having visibility into what the others are doing.

If you've built on an orchestration framework like LiveKit's agent layer, you'll recognize this: the interruption and turn detection logic is distributed across several internal gates—VAD thresholds, word-count minimums, accumulated turn buffers—each making decisions independently based on their slice of the conversation. Getting them to behave coherently takes configuration, and then re-configuration as your use case evolves. When something changes upstream—a vendor updates their streaming event format, or you change the LLM—the interaction between those gates can shift in ways that aren't obvious until they're audible.

This isn't a criticism of orchestration frameworks. LiveKit and Pipecat are well-built tools that give developers significant control precisely because they expose that complexity. The question is whether your team wants to own that complexity or route around it.

A well-integrated pipeline handles coordination differently. Because the STT, turn detection, LLM, and TTS share context inside the same system, the handoffs happen at the infrastructure level rather than in glue code you write and maintain. The pipeline knows what the agent is about to say, what the user just said, and when to stop. You configure thresholds. You don't manage state across four systems.

Some teams are genuinely well-suited to owning the coordination layer: teams running voice at scale where per-unit cost optimization at the STT or TTS tier meaningfully moves margin, teams with proprietary fine-tuned LLMs that can't route through a third-party gateway, teams whose compliance requirements mandate full control of every vendor relationship. For those teams, the multi-vendor architecture is the right call, and the operational overhead is a conscious, planned tradeoff.

For teams where voice is an interface rather than the core product—lead qualification, appointment scheduling, clinical intake, sales training—the operational cost of the multi-vendor stack is overhead with no corresponding return. Every hour maintaining vendor coordination is an hour not spent on what the product actually does. If you're shipping meeting intelligence or AI notetakers, that math gets brutal fast.

What "collapsed" actually means

The phrase "collapsed the stack" is the most accurate description of what the Voice Agent API does architecturally. The components of a voice agent pipeline didn't disappear. They've been unified, not eliminated.

The internal pipeline runs: voice focus (noise cancellation) → Universal-3 Pro Streaming (real-time speech recognition) → turn detection → LLM Gateway (reasoning) → TTS (voice generation) → audio out. Every stage is present, named, and visible. If you want to understand what happened during a conversation, the logs show the full sequence from session open to close: what the user said, what the model transcribed, what the LLM generated, and what the TTS spoke. One stream. One view. One place to look when something needs debugging.

What collapsed is the coordination burden—the glue code you'd otherwise write to move data between components that don't share context, the state machines you'd otherwise build to manage turn-taking across systems with separate internal clocks, and the failure surfaces that emerge whenever data crosses a boundary between vendors.

The API surface reflects this. The OpenAI Realtime API—a genuinely well-designed product—exposes 30+ event types. The Voice Agent API exposes approximately six. That isn't a constraint on what you can build. It's evidence that the complexity has moved inside the system rather than being distributed to developers. When the pipeline handles coordination internally, the external surface can be simpler without being less powerful. The full event reference lives in the Voice Agents docs.

There's a practical consequence for observability. With the Voice Agent API, a failing turn surfaces as a traceable sequence in a single log stream. You see what came in, what was decided at each stage, and what went out. You don't correlate across vendors. You don't wonder which provider introduced the latency. One team owns the full stack, which means when something is wrong, there's one place to report it. Deeper context on accuracy and speaker diarization sits in the Speech Understanding layer.

The math, actually

$4.50 per hour. That's the flat rate for the complete pipeline—speech recognition, language model reasoning, and voice generation—billed per second of session duration. Full breakdown lives on the pricing page.

From there, the arithmetic is direct. A 5-minute call costs $0.375. A hundred concurrent calls running for eight hours costs $3,600. A clinical intake workflow handling a thousand calls a day at four minutes each runs $300. You know your call volume. You know your average session length. The cost-per-call is a one-line formula, and it doesn't change as you change the system prompt, add tools, or tune the agent's behavior.

Compare this to the same calculation on a multi-vendor stack. STT cost depends on audio duration. LLM cost depends on turn count, system prompt length, conversation history handling, and the ratio of input to output tokens—a ratio that shifts every time you change the prompt or the agent's verbosity. TTS cost depends on character count, which changes with response style. Modeling a cost-per-call before real traffic means making assumptions about all three, and the assumptions tend to be optimistic.

This matters more than it sounds when you're trying to sign off on a deployment or make the case internally for scaling a voice product. A number you can derive in one line is a number you can defend. A number that requires a multi-tab spreadsheet becomes a negotiation every time it comes up. Accuracy assumptions matter too—see how word error rate shifts pipeline economics.

Unified billing isn't a pricing trick. It's what happens when one system is responsible for the full pipeline.

This isn't an argument against complexity

There are teams for whom owning every layer is the right architecture: teams running voice at millions of calls per month where STT and TTS margin improvements compound meaningfully, teams with custom fine-tuned models that require specific routing, teams whose enterprise contracts require a specific vendor configuration.

For those teams, the multi-vendor stack isn't a burden—it's a deliberate investment, staffed and planned accordingly. The operational costs are known and accepted as part of what the architecture provides.

This is an argument for entering that investment with eyes open. A four-vendor voice infrastructure is a real engineering product with real operational costs that don't appear in demos. If your team is resourced to build and maintain it, that's the right call for your situation. For background on the design space, the voice agents guide walks through the tradeoffs.

If your team's job is anything other than voice infrastructure itself—if voice is how your product communicates rather than what your product is—the infrastructure being invisible isn't a compromise. It's the point. The Voice Agent API isn't for developers who want less understanding of their stack. It's for developers who want more of their time back for the thing they're actually building.

FAQs

What is a multi-vendor voice agent stack?

A multi-vendor voice agent stack is a voice AI pipeline assembled from separate providers for speech recognition (STT), language model reasoning (LLM), text-to-speech (TTS), and orchestration. Each vendor has its own API, billing, logs, and release cadence, and the developer is responsible for coordinating them into a single working product.

How does a collapsed voice agent stack work?

A collapsed stack runs STT, turn detection, LLM reasoning, and TTS inside one pipeline that shares context across stages. The handoffs between components happen at the infrastructure level rather than in glue code, so a developer configures thresholds instead of managing state across four systems. AssemblyAI's Voice Agent API runs voice focus, Universal-3 Pro Streaming, turn detection, an LLM Gateway, and TTS as a single service.

When should a team use a multi-vendor voice stack instead of a unified one?

A multi-vendor stack makes sense when voice infrastructure is itself the product, when per-unit STT or TTS margin optimization meaningfully moves economics at scale, or when proprietary fine-tuned LLMs require specific routing. For teams building voice as an interface—lead qualification, scheduling, intake, sales coaching—a unified pipeline removes operational overhead with no corresponding return.

How much does the AssemblyAI Voice Agent API cost?

The Voice Agent API is $4.50 per hour, billed per second of session duration, covering speech recognition, LLM reasoning, and voice generation in a single line item. A 5-minute call costs $0.375. Cost-per-call doesn't change as you tune the prompt, add tools, or adjust agent verbosity. See the pricing page for full details.

What's the difference between the Voice Agent API and the OpenAI Realtime API?

The OpenAI Realtime API exposes 30+ event types and gives developers fine-grained control over a speech-to-speech model. The AssemblyAI Voice Agent API exposes around six events because the coordination logic—turn detection, interruption handling, pipeline state—runs inside the service rather than in the developer's code. Both ship real-time voice; they differ in where the complexity lives.

How do I debug a failing turn in a unified voice pipeline?

With a unified pipeline, a failing turn surfaces as a traceable sequence in a single log stream showing the transcript, turn-detection decision, LLM response, and TTS output for that turn. You don't correlate across vendor dashboards or guess which provider introduced latency. Implementation details live in the AssemblyAI docs and the streaming reference.