Capacity, Quotas and Resilience

The idea in one minute#

With a hosted model you do not run out of servers. You run out of quota: the number of tokens and requests per minute the provider lets your account send. When you exceed it, requests are refused with a “too many requests” error, and from the user’s side your product is down.

Capacity planning for an AI system is therefore arithmetic on tokens per minute at the peak, compared with the quota you have been granted — which is not the number on the provider’s website, and can take weeks to raise.

Resilience is the second half. The model provider is a dependency you do not control. It will have slow periods and outages, and it will retire the model version you built on. A production design says, for each of those, what the user experiences — a second route, a queue, a simpler answer, a manual path — and proves it by testing.

A picture#

flowchart LR
  APP[":i-server: <b>Applications</b>"] --> GW[":envoyproxy: <b>AI gateway</b><br/><small>limits per team, routing, retries</small>"]
  GW --> PRI[":i-brain: <b>Primary route</b><br/><small>provisioned capacity, in region</small>"]
  GW -->|"spill over at peak"| OD[":i-cloud: <b>On-demand route</b><br/><small>same model, shared capacity</small>"]
  GW -->|"provider failing"| FB[":i-recycle: <b>Fallback route</b><br/><small>second model, evaluated</small>"]
  GW -->|"can wait"| BQ[":apachekafka: <b>Queue or batch</b><br/><small>hours, half price</small>"]
  GW -->|"all else failed"| DM[":i-hand: <b>Degraded mode</b><br/><small>cached answer, manual path</small>"]
  GW -.-> M[":prometheus: <b>Usage and limit telemetry</b>"]
  class APP neutral
  class GW queue
  class PRI,OD compute
  class FB,BQ memory
  class DM warn
  class M io

How it really works#

Quotas, the real limit#

Providers limit each account, per model, on several counters at once.

Three facts surprise new architects. Limits are per model and per region, so moving to an in-region deployment can cut your quota sharply. Limits often scale with spend history, so a new account starts low. And raising them is a request with a lead time — sometimes hours, sometimes weeks for a new model in high demand. Ask for the quota you will need at launch when the pilot starts, not the week before.

Sizing the peak#

Averages are useless here; quotas are enforced by the minute.

peak requests per minute  =  daily requests × share in the busiest hour ÷ 60 × burst factor
peak input tokens / min   =  peak requests per minute × input tokens per request
peak output tokens / min  =  peak requests per minute × output tokens per request

Find the busiest hour from the existing process — claims arrive on Monday mornings, support peaks after a release. A burst factor of 1.5 to 3 covers the uneven minutes inside that hour. Then add headroom: plan to use no more than 60–70% of quota at expected peak, because retries during an incident multiply load exactly when capacity is scarce.

For agents, multiply again: one user task is many model calls, each resending a growing context. Size on tokens per task, not requests per user.

Three kinds of capacity#

A mature design uses all three: provisioned capacity for the steady base of interactive traffic, on-demand to absorb the peak above it, and batch for everything that can wait. The break-even for provisioned capacity depends on how full you keep it — the calculator below shows the shape.

Deployment scope is the fourth choice. Global deployments route to wherever capacity is free: highest limits, lowest price. Regional or data-zone deployments keep processing inside a boundary: required for residency, usually with lower limits and a premium. Decide with Security, Privacy and Residency before sizing.

Designing for the provider failing#

List the failure modes and give each an answer.

Fallback done properly#

A fallback model is only real if four things are true.

1. It has passed the evaluation. A different model gives different answers. If nobody has measured them, you are replacing an outage with an unknown.
2. Its prompts are maintained. Prompts tuned for one model often need changes for another. Two routes mean two sets to keep current.
3. It has capacity. A fallback with a tiny quota fails the moment it is needed. Check its limits too.
4. It has been exercised. Send a small share of real traffic through it every day, or run a scheduled failover test. An untested fallback is a hope.

Two rules keep failover from making things worse. Fail over before the first token has been shown to the user, never mid-answer. And cap retries — an aggressive retry policy during a provider incident is how one outage becomes two.

Degraded modes#

When no model route is available, something should still happen. Decide it in advance.

The manual path matters more than it seems. A team that has forgotten how to work without the tool has turned a degraded service into an outage.

Availability, honestly stated#

Your system cannot be more available than the dependencies it needs in series. If the model route is up 99.5% of the time and your own application 99.9%, the feature is up about 99.4% — roughly four hours of downtime a month — unless a fallback covers the gap. State availability targets with the degraded mode included, and read the provider’s actual service terms rather than assuming one exists for every model and deployment type.

The gateway is where this lives#

Every mechanism on this page — limits per team, routing, spill-over, retries, fallback, usage records — belongs in one place between applications and models. Without a gateway each application reinvents them differently, and nobody can see total demand. The component is described in Model Access and Gateways and the patterns in Reliability Patterns.

Code#

Two calculations an architect brings to the capacity conversation: peak demand against quota, and the utilisation at which provisioned capacity beats on-demand.

// capacity.go — peak token demand against quota, and provisioned vs on-demand.
package main

import "fmt"

func main() {
	// Demand, from discovery.
	perDay := 60_000.0  // requests per day
	busiestHour := 0.18 // share of the day's traffic in the busiest hour
	burst := 2.0        // busiest minute relative to that hour's average
	inTok, outTok := 7_000.0, 450.0

	peakRPM := perDay * busiestHour / 60 * burst
	peakIn, peakOut := peakRPM*inTok, peakRPM*outTok
	avgRPM := perDay / (24 * 60)

	// Quota actually granted for this model and region.
	quotaRPM, quotaIn, quotaOut := 500.0, 2_000_000.0, 400_000.0
	const target = 0.65 // do not plan to exceed this share of quota

	fmt.Printf("average %.0f requests/min, peak %.0f requests/min (%.0f× the average)\n\n",
		avgRPM, peakRPM, peakRPM/avgRPM)
	check := func(name string, need, have float64) {
		use := need / have
		verdict := "ok"
		switch {
		case use > 1:
			verdict = "THROTTLED at peak"
		case use > target:
			verdict = "no headroom — request an increase"
		}
		fmt.Printf("  %-22s need %10.0f  quota %10.0f  %4.0f%%  %s\n", name, need, have, use*100, verdict)
	}
	check("requests per minute", peakRPM, quotaRPM)
	check("input tokens per min", peakIn, quotaIn)
	check("output tokens per min", peakOut, quotaOut)

	// Provisioned vs on-demand: one reserved unit, priced per hour, serving up to a fixed rate.
	inP, outP := 2.0, 10.0 // on-demand dollars per million tokens
	unitPerHour := 55.0    // illustrative price of one provisioned unit
	unitRPM := 60.0        // requests per minute one unit sustains at this request shape
	perRequest := (inTok*inP + outTok*outP) / 1e6

	fmt.Printf("\non-demand cost per request: $%.4f\n", perRequest)
	fmt.Println("one provisioned unit, cost per request at different utilisation:")
	for _, u := range []float64{0.2, 0.4, 0.6, 0.8, 1.0} {
		served := unitRPM * 60 * u
		verdict := "dearer than on-demand"
		if unitPerHour/served < perRequest {
			verdict = "cheaper than on-demand"
		}
		fmt.Printf("  %3.0f%% busy  $%.4f  %s\n", u*100, unitPerHour/served, verdict)
	}
	fmt.Printf("break-even utilisation: %.0f%%\n", 100*unitPerHour/(unitRPM*60*perRequest))
}

The first table usually delivers the surprise: the peak is many times the daily average, and it is the input-token limit, not the request limit, that is hit first. The second shows why provisioned capacity is bought for the steady base and not for the peak.

Remember this#

• Hosted models are limited by quota — requests and tokens per minute — per model and region.
• Size the busiest minute, not the average day. Plan to use about two-thirds of quota.
• Provisioned for the base, on-demand for the peak, batch for what can wait.
• A fallback is real only if it is evaluated, maintained, has capacity and is exercised.
• Decide the degraded mode in advance, and keep the manual path alive.

Try it#

1. Run capacity.go. Which limit fails first? Halve the input tokens per request (better retrieval, trimmed history) and run again. What changed?
2. Set busiestHour to 0.08, a flat day. How does the provisioned-capacity decision change?
3. For a system you know, write one line per failure mode in the table: what does the user see today, and what should they see?

Check yourself#

1. Name four different limits a provider applies to an account.
2. Why is quota requested at the start of the pilot rather than before launch?
3. What four conditions make a fallback model real?
4. Your application is 99.9% available and the model route 99.5%. Roughly how available is the feature, and what changes that?

Sources#

• FinOps Foundation: FinOps for AI overview — provisioned capacity and utilisation
• Claude pricing: data residency and batch — an example of regional multipliers and batch discounts, read 13 October 2026