Overview
The circuit breaker checks how a model is performing before the LLM Gateway sends your request to it. If the model is slower or failing more than you allow, the gateway skips it. It moves to your next fallback model, or rejects the request if you have none. Fallbacks react to a request that has already failed. The circuit breaker reacts to a model that is degrading, so a slow or failing model costs you nothing extra. The circuit breaker is available on chat completions and decisions requests.Basic usage
Set at least one threshold infallback_config.circuit_breaker_config, and add a fallback to move to when the breaker trips:
- Python - HTTP
- JavaScript
gemini-2.5-flash-lite and uses claude-sonnet-4-6 when either of these is true:
- The model’s current median latency is more than 100% above its normal latency (more than double).
- More than 20% of its recent requests failed.
Thresholds
The circuit breaker only runs when you set at least one of these thresholds. You can set both. The breaker trips when either one is exceeded.
The baseline is the normal latency of the model in the region that handles your request. The current latency and failure rate are measured by the LLM Gateway on an ongoing basis.
If the gateway has no recent measurements for a model, there is nothing to compare against, so the request is sent to the model as usual. The circuit breaker never blocks a request because of missing data.
fallback_config.circuit_breaker to false.
What happens when it trips
The circuit breaker checks each model before it is called, including each fallback.- If the model is healthy, the request is sent to it.
- If the model trips the breaker and you set a fallback, the gateway moves to the next fallback. The response looks as if you had sent the request to the fallback model, and you are billed only for that model.
- If the model trips the breaker and there is no fallback left, the request fails with a
500error.
fallback_config.depth, as described in Model fallbacks.
Rejected requests
If you don’t configure any fallbacks, a tripped circuit breaker fails the request automatically. This is true even thoughfallback_config.retry is on by default. The automatic retry only runs after a request to a model fails, and it would send the request to the same model again, so the circuit breaker skips it. To keep the request alive when the breaker trips, add a fallback.
When there is no fallback to move to, the response is a 500 with a message that names the threshold that was exceeded, and details of what was measured:
failure threshold, reason is failure_threshold, and failure_rate is returned in place of the latency fields.
A rejected request is not billed. You can retry it, or catch the
500 and send the request to a different model yourself.
Choosing thresholds
- Start loose and tighten. Latency varies from request to request, so a very low
percent_thresholdsuch as10can trip on normal variation. A value of100to200catches a model that has clearly slowed down. - Use
failure_thresholdfor availability. It protects you from a model or provider that is returning errors, which complements fallbacks that only act after your own request has failed. - Always set a fallback for production traffic. Without one, a tripped breaker returns an error to your user.
API reference
Request parameters
Next steps
- Model fallbacks - Choose the backup models to use
- Improve latency - Reduce response times for your requests
- Rate limits - Understand the limits on your requests