> ## Documentation Index
> Fetch the complete documentation index at: https://assemblyai.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Circuit breaker

## Overview

The circuit breaker checks how a model is performing before the LLM Gateway sends your request to it. If the model is slower or failing more than you allow, the gateway skips it. It moves to your next [fallback model](https://assemblyai.com/docs/llm-gateway/fallback.md), or rejects the request if you have none.

[Fallbacks](https://assemblyai.com/docs/llm-gateway/fallback.md) react to a request that has already failed. The circuit breaker reacts to a model that is degrading, so a slow or failing model costs you nothing extra.

The circuit breaker is available on [chat completions](https://assemblyai.com/docs/llm-gateway/chat-completions.md) and [decisions](https://assemblyai.com/docs/llm-gateway/decisions.md) requests.

## Basic usage

Set at least one threshold in `fallback_config.circuit_breaker_config`, and add a fallback to move to when the breaker trips:

> [tab] Python - HTTP
>
> ```python theme={null}
> import requests
>
> headers = {
>     "authorization": "<YOUR_API_KEY>"
> }
>
> response = requests.post(
>     "https://llm-gateway.assemblyai.com/v1/chat/completions",
>     headers=headers,
>     json={
>         "model": "gemini-2.5-flash-lite",
>         "messages": [
>             {"role": "user", "content": "Tell me a short fairy tale."}
>         ],
>         "max_tokens": 400,
>         "fallbacks": [
>             {"model": "claude-sonnet-4-6"}
>         ],
>         "fallback_config": {
>             "circuit_breaker_config": {
>                 "percent_threshold": 100,
>                 "failure_threshold": 20
>             }
>         }
>     }
> )
>
> result = response.json()
> print(result["choices"][0]["message"]["content"])
> ```

> [tab] JavaScript
>
> ```javascript theme={null}
> const response = await fetch(
>   "https://llm-gateway.assemblyai.com/v1/chat/completions",
>   {
>     method: "POST",
>     headers: {
>       authorization: "<YOUR_API_KEY>",
>       "content-type": "application/json",
>     },
>     body: JSON.stringify({
>       model: "gemini-2.5-flash-lite",
>       messages: [{ role: "user", content: "Tell me a short fairy tale." }],
>       max_tokens: 400,
>       fallbacks: [{ model: "claude-sonnet-4-6" }],
>       fallback_config: {
>         circuit_breaker_config: {
>           percent_threshold: 100,
>           failure_threshold: 20,
>         },
>       },
>     }),
>   }
> );
>
> const result = await response.json();
> console.log(result.choices[0].message.content);
> ```

In this example, the gateway skips `gemini-2.5-flash-lite` and uses `claude-sonnet-4-6` when either of these is true:

* The model's current median latency is more than 100% above its normal latency (more than double).
* More than 20% of its recent requests failed.

## Thresholds

The circuit breaker only runs when you set at least one of these thresholds. You can set both. The breaker trips when either one is exceeded.

| Key | Type | Description |
| - | - | - |
| `percent_threshold` | number | How far the model's current median (p50) latency may rise above its baseline, in percent. For example, `100` trips when the current latency is more than double the baseline. Must not be negative. |
| `failure_threshold` | number | The share of recent requests to the model that may fail, in percent. For example, `20` trips when more than 20% failed. Must be between 0 and 100. |

The baseline is the normal latency of the model in the region that handles your request. The current latency and failure rate are measured by the LLM Gateway on an ongoing basis.

> [!NOTE]
>
> If the gateway has no recent measurements for a model, there is nothing to compare against, so the request is sent to the model as usual. The circuit breaker never blocks a request because of missing data.

To turn the circuit breaker off for a request, set `fallback_config.circuit_breaker` to `false`.

## What happens when it trips

The circuit breaker checks each model before it is called, including each fallback.

1. If the model is healthy, the request is sent to it.
2. If the model trips the breaker and you set a fallback, the gateway moves to the next fallback. The response looks as if you had sent the request to the fallback model, and you are billed only for that model.
3. If the model trips the breaker and there is no fallback left, the request fails with a `500` error.

The circuit breaker only moves to a fallback model that you listed. It doesn't retry the same model. How many fallbacks it can walk through follows `fallback_config.depth`, as described in [Model fallbacks](https://assemblyai.com/docs/llm-gateway/fallback.md).

### Rejected requests

If you don't configure any fallbacks, a tripped circuit breaker fails the request automatically. This is true even though `fallback_config.retry` is on by default. The automatic retry only runs after a request to a model fails, and it would send the request to the same model again, so the circuit breaker skips it. To keep the request alive when the breaker trips, add a fallback.

When there is no fallback to move to, the response is a `500` with a message that names the threshold that was exceeded, and details of what was measured:

```json theme={null}
{
  "request_id": "<REQUEST_ID>",
  "code": 500,
  "message": "Request failed, circuit breaker threshold exceeded percent threshold.",
  "metadata": {
    "errors": [],
    "circuit_breaker": {
      "model": "gemini-2.5-flash-lite",
      "provider": "<PROVIDER>",
      "reason": "percent_threshold",
      "threshold": 100,
      "observed": 250,
      "current_p50_ms": 3500,
      "baseline_p50_ms": 1000
    }
  }
}
```

If the failure threshold tripped, the message ends with `failure threshold`, `reason` is `failure_threshold`, and `failure_rate` is returned in place of the latency fields.

| Key | Type | Description |
| - | - | - |
| `model` | string | The model that tripped the breaker. |
| `provider` | string | The provider the model was routed to. |
| `reason` | string | `percent_threshold` or `failure_threshold`. |
| `threshold` | number | The threshold you set, in percent. |
| `observed` | number | The measurement that exceeded the threshold, in percent. For `percent_threshold`, how far the current p50 latency is above the baseline: `(current - baseline) / baseline * 100`, so `200` means three times the baseline. For `failure_threshold`, the share of recent requests that failed. |
| `current_p50_ms` | number | The model's current median latency in milliseconds. Only returned for `percent_threshold`. |
| `baseline_p50_ms` | number | The model's baseline median latency in milliseconds. Only returned for `percent_threshold`. |
| `failure_rate` | number | The share of recent requests that failed, in percent. Only returned for `failure_threshold`. |

A rejected request is not billed. You can retry it, or catch the `500` and send the request to a different model yourself.

## Choosing thresholds

* **Start loose and tighten.** Latency varies from request to request, so a very low `percent_threshold` such as `10` can trip on normal variation. A value of `100` to `200` catches a model that has clearly slowed down.
* **Use `failure_threshold` for availability.** It protects you from a model or provider that is returning errors, which complements fallbacks that only act after your own request has failed.
* **Always set a fallback for production traffic.** Without one, a tripped breaker returns an error to your user.

## API reference

### Request parameters

| Key | Type | Required? | Description |
| - | - | - | - |
| `fallbacks` | array | No | The models to move to when the circuit breaker trips. See [Model fallbacks](https://assemblyai.com/docs/llm-gateway/fallback.md). |
| `fallback_config` | object | No | Configuration for fallback behavior. |
| `fallback_config.circuit_breaker` | boolean | No | Whether the circuit breaker may run. Defaults to `true`. It still does nothing until you set a threshold. |
| `fallback_config.circuit_breaker_config` | object | No | The thresholds that trip the circuit breaker. |
| `fallback_config.circuit_breaker_config.percent_threshold` | number | No | Percent the current p50 latency may rise above the baseline. Must not be negative. |
| `fallback_config.circuit_breaker_config.failure_threshold` | number | No | Percent of recent requests that may fail. Between 0 and 100. |

## Next steps

* [Model fallbacks](https://assemblyai.com/docs/llm-gateway/fallback.md) - Choose the backup models to use
* [Improve latency](https://assemblyai.com/docs/llm-gateway/improve-latency.md) - Reduce response times for your requests
* [Rate limits](https://assemblyai.com/docs/llm-gateway/rate-limits.md) - Understand the limits on your requests
