Skip to main content
LLM Gateway applies two rate limits: requests per minute (RPM) and tokens per minute (TPM). Limits are tracked per account and per model, provider, and region, over a rolling 60-second window. Speech Understanding requests (/v1/understanding) have an RPM limit only, with no token limit.

​
How limits work

Every LLM Gateway chat completion request is checked against both limits. If either one is exceeded, the request is rejected with HTTP 429.
  • Requests per minute (RPM): the number of requests you send.
  • Tokens per minute (TPM): the number of tokens your requests use.

​
Window

Limits apply over a sliding 60-second window, not a fixed one. Usage from the previous minute fades out gradually rather than all at once on the minute boundary.

​
Your limits

Limits are assigned per account in tiers. New accounts start at Tier 0 with conservative limits, and your tier increases automatically as your spend grows.

​
How you move up tiers

  • Tier 1: reached once your account has any spend.
  • Tier 2: reached once your total spend across all AssemblyAI products passes a modest threshold.
  • Tiers 3 to 5: reached through spend on a specific model. Higher spend on a model raises your tier for that model.
Tiers are re-evaluated once per hour, and recent spend can take two to three hours to be reflected. After a spending milestone, your new limits may not apply right away.

​
Requests per minute by tier

​
Tokens per minute by tier

TPM limits vary by model, because they are set relative to each model’s capacity. Each tier grants access to a higher percentage of that capacity, so larger-capacity models get higher TPM limits. To see the specific values for your account, check your dashboard. AssemblyAI can also set custom limits for individual accounts.
Need a higher rate limit?If you need a higher rate limit, contact our support team.

​
How tokens are counted

Token counting is the most common cause of unexpected 429 responses.
  1. On admission: when a request is admitted, the gateway reserves an estimate against your TPM limit: the estimated prompt tokens plus the request’s max_tokens. If you don’t set max_tokens, 8,000 output tokens are reserved.
  2. While it runs: the reservation is held for the whole request.
  3. On completion: the reservation is replaced by the real usage: input tokens + output tokens + cache-write tokens. Cache-read tokens do not count toward TPM.
Because max_tokens is reserved up front, setting it much higher than a request needs uses up TPM headroom while the request runs, even though those tokens are never generated. Concurrent requests with large max_tokens values can hit the TPM limit early. Example: a request with a roughly 2,000-token prompt and max_tokens: 64000 holds about 66,000 tokens of TPM until it finishes, even if the reply is only 300 tokens.
Set max_tokens close to what the response actually needs.

​
Handling 429 responses

When you exceed a limit, the API responds with a 429 status code and a JSON body:
The message tells you which limit was hit: Every 429 response includes a Retry-After header with the number of seconds to wait before retrying. Wait at least that long, and use exponential backoff if retries keep failing.

​
Response headers

These headers are set on rate-limited endpoints, on both successful and 429 responses.

​
Request (RPM) headers

​
Token (TPM) headers

These headers are sent by LLM Gateway only. Speech Understanding doesn’t send them.

​
Retry header

If the response doesn’t include X-RateLimit headers, the endpoint doesn’t have rate limits.

​
Avoid 429 responses

  • Set max_tokens to what you need, so you don’t reserve TPM you won’t use.
  • Watch X-RateLimit-Remaining and X-RateLimit-Remaining-Tokens and slow down as they approach zero.
  • Spread bursts of requests across the window instead of sending them all at once.
  • Retry with backoff, waiting at least the Retry-After interval.
  • Contact support if you need higher limits.