LLM Gateway Overview

​

Overview

​

Quickstart

import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers = headers,
    json = {
        "model": "claude-sonnet-4-6",
        "messages": [
            {"role": "user", "content": "What is the capital of France?"}
        ],
        "max_tokens": 1000
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])

const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [{ role: "user", content: "What is the capital of France?" }],
      max_tokens: 1000,
    }),
  }
);

const result = await response.json();
console.log(result.choices[0].message.content);

from openai import OpenAI
import openai.types.chat.chat_completion as types

client = OpenAI(
    base_url="https://llm-gateway.assemblyai.com/v1",
    api_key="<YOUR_API_KEY>",
)

import json

messages = [{"role": "user", "content": "What is the capital of france"}]
response = client.chat.completions.create(model="claude-sonnet-4.6", messages=messages)
print(response.choices[0].message.content)

​

Non-standard LLM Enhancements

• Automatic retries
  • 5xx responses will automatically be retried on our side by default
  • configure with fallback_config
• Model Fallbacks
  • Configure alternative models/providers in case a model isn’t working
  • Even change the prompt and/or params to better suit a different model
  • See docs
• Automatically apply LLM Gateway requests on each real-time STT turn for reduced E2E latency
  • see docs
• Post Processing
  • Automatically apply post processing to the LLM response
  • Repair broken JSON in tool calls and structured output with json-repair
  • see docs
• Inject Transcript Text
  • Call LLM Gateway with a transcript_id and we will handle fetching and adding the transcript text to the LLM message.
  • Useful if you don’t want to store the transcript.
  • see docs

​

Regional compliance

• Basic Chat Completions - Simple request/response interactions
• Streamed Responses - Stream output as it’s generated (OpenAI models)
• Multi-turn Conversations - Maintain context across multiple exchanges
• Structured Outputs - Constrain responses to a specific JSON schema
• Tool/Function Calling - Enable models to execute custom functions
• Agentic Workflows - Multi-step reasoning with automatic tool chaining
• Unified Interface - One API for Claude, GPT, Gemini, Qwen, Kimi, and more
• Post-processing - Automatically repair malformed JSON responses with built-in JSON repair

​

Global model routing

​

Rate limits

​

Logging and troubleshooting

• request_id from the response body
• The model parameter used
• The API region (US: llm-gateway.assemblyai.com, EU: llm-gateway.eu.assemblyai.com)
• A timestamp for when the request was sent
• The full error response body when a non-2xx status code is returned

​

Next steps

• Basic Chat Completions - Learn how to send simple messages and receive responses
• Multi-turn Conversations - Maintain context across multiple exchanges
• Structured Outputs - Constrain model responses to follow a specific JSON schema
• Tool Calling - Enable models to execute custom functions
• Agentic Workflows - Build multi-step reasoning applications
• Post-processing - Automatically repair malformed JSON in model responses
• Prompt Caching - Save money on similar calls with prompt caching