AssemblyAI LLM Gateway vs. OpenRouter vs. LLM Gateway.io: Pricing, security, and reliability compared

An LLM gateway sits between your application and a set of model providers, exposing one API, one key and one bill in place of several. All three of the gateways in this comparison do that. The question worth answering is not which one routes to the most models — that number changes every month and every vendor is chasing it — but which one is built around the workload you are actually running. If your workload is voice, the answer is narrower than the general comparison suggests, because voice products are dominated by short, latency-sensitive rewrite calls that general-purpose gateways route to models that were never optimised for them.

This post compares AssemblyAI’s LLM Gateway, OpenRouter, and LLM Gateway.io on the things that separate them: model coverage, fallback behaviour, latency on short tasks, pricing shape, and data residency. It is written for developers building voice applications, so it weights voice-specific concerns heavily. If you are building a general chat product, OpenRouter’s breadth will matter more to you than it does here, and the post says so.

What an LLM gateway is for

Three problems push teams toward a gateway.

Provider sprawl. You start with one model, add a cheaper one for a background job, add a third because a customer requires a specific vendor, and now you are maintaining three SDKs, three key rotations, three rate-limit behaviours and three billing relationships.

Availability. Model providers have incidents. If a single provider outage takes down your product, you have coupled your uptime to someone else’s. A gateway with automatic fallback turns a provider incident into a latency blip.

Portability. Model quality-per-dollar moves fast. A gateway lets you change the model behind a feature by changing a string, rather than by shipping a refactor.

Every gateway in this comparison addresses all three. Where they diverge is in what they assume you are doing with the models once you are connected.

AssemblyAI LLM Gateway

AssemblyAI’s LLM Gateway is an OpenAI-compatible chat completions endpoint that currently fronts 37 models. It replaced LeMUR, which was deprecated on 2026-03-31. There are two regional endpoints:

US:  https://llm-gateway.assemblyai.com/v1/chat/completions
EU:  https://llm-gateway.eu.assemblyai.com/v1/chat/completions

A request looks like any OpenAI-compatible call, which means most existing client code works with a base-URL change:

import requests

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers={"Authorization": "<YOUR_API_KEY>"},
    json={
        "model": "qwen3.5-4b-32k-fast",
        "messages": [
            {"role": "system", "content": "Clean up this dictated text. Keep the speaker's tone."},
            {"role": "user", "content": transcript},
        ],
        "temperature": 0.2,
        "max_tokens": 512,
    },
)

Note the auth header: the raw key, with no Bearer prefix. The Voice Agent API uses Bearer; this one does not.

The features that distinguish it operationally:

• Automatic fallback to up to two backup models, with a 500 ms retry default and per-fallback overrides for prompt, temperature and max tokens. The override matters more than it sounds — a fallback model often needs a different prompt to produce comparable output, and a gateway that only swaps the model name tends to degrade output quality at exactly the moment you were trying to protect it.
• Prompt caching on Anthropic and OpenAI models, with cache-read pricing typically an order of magnitude below the prompt rate.
• US and EU multi-region failover, with the EU endpoint keeping data in the EU at the same price.
• One bill alongside transcription. If you are already sending audio to AssemblyAI, the gateway is on the same account and the same invoice.

Full request and response reference is in the chat completions documentation, and the product background and design rationale are covered in reintroducing LLM Gateway.

Qwen3.5 4B Fast: the model that makes this a different category

This is the part of the comparison that a general-purpose gateway structurally cannot match, and it is worth understanding before looking at any feature table.

Voice products are not chat products. The dominant LLM call in a voice product is short, repetitive and latency-critical: clean up a dictated sentence, rewrite a transcript turn into something sendable, apply live formatting, summarise the turn that just ended. These calls run constantly, they run on small inputs, and they run inside a budget where a human is waiting. Routing them to a frontier general-purpose model is the wrong trade twice over — you pay frontier prices and you wait frontier latency for a task that does not need frontier reasoning.

Qwen3.5 4B Fast (qwen3.5-4b-32k-fast) is the only model on the entire 37-model roster whose provider is AssemblyAI. Every other model routes to OpenAI, AWS Bedrock, Vertex or Fireworks. This one runs on AssemblyAI’s own GPUs, in a latency-optimised 32k-context configuration, purpose-built for that class of rewrite work. The figures:

Be clear about what it is not. Qwen3.5 4B Fast does not support tools, tool_choice or response_format. It is not an agentic model and it will not orchestrate anything. It is a fast rewrite model: dictation cleanup, transcript rewriting, live formatting, turn summarisation. If your task needs function calling or guaranteed structured output, route it to one of the general-purpose models on the same gateway — that is what the fallback and routing configuration is for. Picking the right tool here is worth more than picking the biggest one.

Self-hosting is what makes the latency number possible. A gateway that only proxies to third-party APIs inherits every provider’s queueing, cold-start and rate-limit behaviour; it cannot tune the serving configuration for short-context, low-latency work because it does not control the serving stack. Details are in the Qwen 4B launch post.

The Dictation API: the gateway thesis in a single request

If you want to see what a voice-native gateway buys you, look at the Dictation API, which launched on 2026-09-15 and is generally available and self-serve.

Architecturally, the Dictation API is LLM Gateway running a cleanup pass on a self-hosted small model in the same request as transcription. You POST audio once and get back both the verbatim transcript and the cleaned-up rewrite. There is no second call, no orchestration layer of your own, and no token accounting:

import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

config = aai.DictationConfig(
    stt_prompt="A doctor dictating a patient visit note.",
    keyterms_prompt=["amoxicillin", "lisinopril", "metoprolol"],
    llm_instruction=(
        "Remove filler words and rewrite as a concise clinical chart note."
    ),
)

result = aai.DictationTranscriber().transcribe_live("clip.wav", config)

transcript = result.text
rewrite = result.final_text  # the cleaned-up text, falling back to the transcript

Said: “um so can we uh move the the meeting to thursday i think friday works better actually”. Returned: “Can we move the meeting to Friday? That works better.” Transcription models are verbatim by design, but what was said is not what anyone wants to send.

The pricing is where the architecture shows up on the invoice. From the pricing page: “Every feature is included in the $0.62/hr rate. One line on your bill covers the whole request — there is no second rate to model and no token math to do.” $0.62/hr flat. Volume discounts apply at any tier.

Compare that to assembling the same product on a general-purpose gateway: a transcription vendor billed per hour, a gateway billed per token, an orchestration layer you wrote and now operate, and a cost model where your unit economics depend on how verbose your users are. The rewrite is best-effort with a 5-second internal deadline — on failure the request still returns 200 with the transcription intact and llm_response: null, so a slow model degrades the output rather than breaking the call. Documentation: Dictation API docs and transcript rewriting.

That is the post’s thesis demonstrated rather than asserted: the value of a voice-native gateway is not the routing table, it is what becomes possible when the gateway and the speech layer are the same system.

OpenRouter

OpenRouter is the broadest general-purpose gateway in common use. Its proposition is coverage: a very large catalog spanning most commercially available models plus a long tail of open-weight ones, a unified OpenAI-compatible API, and pass-through pricing with a margin. It also exposes routing preferences so you can express provider priority, and it publishes per-model throughput and latency data, which is genuinely useful when you are choosing between providers serving the same open-weight model.

Where it fits: if your product needs access to an unusually wide range of models — because you are running evaluations across many of them, because customers choose their own model, or because you depend on niche open-weight releases — breadth is the feature and OpenRouter is strong at it.

Where it is a weaker fit for voice: it is a routing layer over other people’s inference. There is no self-hosted, latency-tuned small model underneath it that was built for transcript rewriting, and there is no speech layer to combine with, so the transcription-plus-rewrite pattern remains two vendors, two bills and an orchestration layer you own.

On compliance and data residency, see the note below — we do not characterise another vendor’s posture here, and you should confirm current terms directly with them.

LLM Gateway.io

LLM Gateway.io is the lighter-weight option of the three, oriented toward teams that want a straightforward proxy in front of a handful of providers without adopting a large platform. The typical reason to pick it is simplicity and control over the deployment, rather than either the breadth that OpenRouter offers or the vertical integration that AssemblyAI offers.

The same structural point applies as for OpenRouter: it is a routing layer, not an inference provider, so the latency floor on a short rewrite call is set by whichever upstream provider serves it. For a voice product, that is the number that matters most.

As with OpenRouter, confirm compliance terms and regional endpoint availability against their own current documentation rather than a third-party summary.

Feature comparison

A note on the compliance rows. “Not publicly stated” means exactly that: we are not characterising another company’s compliance posture in a blog post. Certifications, BAA availability and regional endpoints change, and a stale claim in either direction is unhelpful to you and unfair to them. Check each vendor’s own security or trust documentation, and get the current position in writing during procurement. AssemblyAI’s own posture is documented on the security page.

Model families and provider coverage

The previous version of this post listed every model version by name. That list was accurate the week it was published and wrong a month later, which is a bad trade for a page people arrive at from search. Here is the durable version: the families, the counts, and where each one runs.

That is 37 models total. For the exact roster, model IDs, supported parameters, context limits and current per-token rates, the LLM Gateway documentation and the pricing page are the source of truth — they update when the catalog does, and this post will not.

Two pricing details worth knowing before you model your costs

There is a +10% regional surcharge on the proprietary third-party models — the Anthropic, Google Gemini and OpenAI families. The open-weight models carry no surcharge at all. That includes Gemma, GPT-OSS, the whole Qwen family, Minimax M3 and Kimi K3. On a high-volume rewrite path this is not a rounding error: it compounds on top of a per-token rate that is already an order of magnitude apart between tiers.

Cache-read pricing exists on the Anthropic, OpenAI and Gemini models, and it is roughly a tenth of the prompt rate. Claude Opus 5, for instance, is $5 per 1M prompt tokens but $0.50 per 1M on a cache read. If your voice product sends the same long system prompt on every turn — and most do — caching is the single largest lever on your bill after model selection.

The practical selection heuristic for a voice product is simpler than the catalog suggests: use the self-hosted small model for the high-volume rewrite path, a mid-tier general model for anything that needs reasoning over a whole conversation or genuine tool calling, and a frontier model only where the output is customer-visible and quality-critical. Configure the second as the fallback for the first and you have covered availability at the same time.

Where the speech layer fits

The reason a voice-native gateway is worth considering at all is that in a voice product the LLM call is never the whole job. Something has to turn audio into text first, and that layer sets the latency floor and the accuracy ceiling for everything downstream.

On AssemblyAI, the same account gives you:

• Universal-3.5 Pro Realtime for streaming, at $0.45/hr, emitting transcripts continuously as audio arrives and deciding end-of-turn from what has been said rather than from a silence timer.
• Universal-3.5 Pro for pre-recorded audio at $0.21/hr, across 18 languages with native code-switching.
• The Sync API at $0.45/hr for short clips, with roughly 134 ms p50 on a two-second clip from request to finished transcript.
• The Voice Agent API at a flat $4.50/hr for a complete speech-in-to-speech-out turn at approximately 1 second end to end.

Pairing any of those with a gateway call is one account, one key, one invoice and one vendor to escalate to at 3am. That is not an architectural argument, but it is the operational one that decides most of these evaluations in practice.

Which one should you pick

Pick OpenRouter if breadth of model access is the requirement — you are evaluating across many models, your customers choose their own, or you depend on niche open-weight releases. That is what it is built for and it does it well.

Pick LLM Gateway.io if you want a lightweight proxy over a small set of providers and prefer to keep the platform surface area minimal.

Pick AssemblyAI’s LLM Gateway if you are building a voice product. The deciding factors are Qwen3.5 4B Fast at 612 ms average on rewrite tasks and 94% cheaper per hour of audio, no regional surcharge on the open-weight tier, the ability to collapse transcribe-plus-rewrite into a single billed request via the Dictation API at $0.62/hr flat, a dedicated EU endpoint at the same price, and one bill covering both speech and language models.

If you are unsure, the test that settles it takes an afternoon: take a hundred real utterances from your product, run your rewrite prompt through each gateway, and measure p50 and p95 latency plus cost per hour of audio. Voice workloads are unusual enough that general benchmarks will not predict your result.

Get started

The fastest way to evaluate a gateway for a voice product is to run your own rewrite prompt through it on your own utterances and look at p95 latency and cost per hour of audio, not per token.

Further reading: the Dictation API launch post · chat completions reference · pricing

Frequently asked questions

What is an LLM gateway?

An LLM gateway is a service that sits between your application and multiple model providers, exposing a single API, key and bill in place of several. It typically adds automatic fallback when a provider fails, routing rules, caching and usage visibility. The practical benefit is that changing the model behind a feature becomes a configuration change rather than a code change.

How many models does AssemblyAI’s LLM Gateway support?

37 models, spanning nine Anthropic Claude models, nine Google Gemini models, eleven OpenAI GPT models, six open-weight models including Gemma, GPT-OSS and the Qwen family, and two Fireworks-hosted models in Minimax M3 and Kimi K3. The LLM Gateway documentation carries the current roster, model IDs and per-token rates.

What makes Qwen3.5 4B Fast different from other models on a gateway?

It is the only model on the roster hosted by AssemblyAI itself, running on AssemblyAI’s own GPUs in a latency-optimised 32,768-token configuration rather than proxied to a third party. It is built for the short rewrite tasks at the centre of voice products — dictation cleanup, transcript rewrite, live formatting, turn summarisation — and averages 612 ms on those tasks, 1.9× faster than GPT-4.1 and 94% cheaper per hour of audio, at $0.10 / $0.50 per million prompt / completion tokens with no regional surcharge. It supports max_tokens, temperature and stream, but not tool calling or structured output — it is a rewrite model, not an agentic one.

Is AssemblyAI HIPAA-compliant?

AssemblyAI is a business associate under HIPAA, not a covered entity, and offers a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure AssemblyAI appropriately safeguards PHI. AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information. The BAA can be reviewed and signed self-serve without a sales call — see the BAA FAQ.

Can I use an LLM gateway with protected health information?

You can use AssemblyAI’s LLM Gateway for workloads involving PHI with a BAA in place; AssemblyAI is a business associate under HIPAA and offers a standard Business Associate Addendum. For other gateways, confirm BAA availability directly with the vendor — we do not characterise another company’s compliance posture here. Supporting certifications on the AssemblyAI side include SOC 2 Type 2, ISO 27001:2022 and PCI DSS v4.0.

Is there a free tier?

Yes. AssemblyAI’s free tier includes 185 hours of pre-recorded transcription and 333 hours of streaming, with no credit card required. That is enough to run a realistic pilot across both the speech layer and the gateway rather than a toy test.

Can I keep my data in the EU?

Yes. LLM Gateway has a dedicated EU endpoint at llm-gateway.eu.assemblyai.com/v1/chat/completions, and the speech APIs have EU endpoints at api.eu.assemblyai.com and streaming.eu.assemblyai.com. Pricing is the same and data stays in the EU. Self-hosted deployment into a customer VPC is also supported. For the other gateways, regional endpoint availability is not publicly stated in a form we would rely on — check with the vendor.