Rate limiting design: fixed windows, sliding windows, token buckets, and fleet-wide budgets

October 27, 2026 · 9 min read

Why can a 100-per-minute limit admit 200 in a burst? Four tips connect window semantics, token-bucket bounds, atomic decisions, aggregate quotas, and work per request in a reporting API.

Choose rate limiting from the resource it protects

A fictional reporting API needs both user fairness and protection for a shared external service. Per-user quotas alone do not protect one shared API key; a global quota alone does not stop one tenant monopolizing it. Derive identity from authenticated context so callers cannot escape quotas by claiming new keys.

Requirement	Candidate	Qualification
100 in each clock-aligned minute	Fixed window	Allows a short burst across the boundary
At most 100 in the preceding 60 seconds	Exact sliding log	Retain timestamps and make decisions atomic
Separate average rate and burst allowance	Token bucket	Interval allowance depends on capacity plus refill
Approximation acceptable at large scale	Weighted window counters	Not the guarantee of an exact sliding log

Fixed and sliding windows: what passes at a boundary?

With a fixed limit of 100 per minute, 100 requests at 12:00:59 and another 100 at 12:01:00 satisfy each window while passing 200 in a short interval. An exact log of admitted requests over the last 60 seconds still contains the first 100 when the next arrive. Define the interval as (t−60 seconds, t] and distinguish requests sharing timestamps with unique IDs.

Make expiry cleanup, the decision, and recording an admission one atomic operation, so another request cannot interleave between those steps. Budget memory and key retention too. Weighted counters reduce state but approximate within-window distribution, so distinguish them from exact recent counts. Cloudflare’s published implementation illustrates the approximation.

A token bucket separates capacity B and refill rate r

Assume one token per request, capacity 20, and continuous refill of 100 tokens per minute. Starting full, admissions over T seconds are bounded by floor(20 + 100T/60). The floor function rounds down to an integer: at most 23 over two seconds and 120 over 60 seconds. A long-term average of 100 per minute does not cap every 60-second interval at 100. Discrete refill implementations require their own boundary analysis.

Conceptual atomic operation on one authoritative limiter state; not a complete distributed implementation

elapsed = max(0, limiter_now - last_refill)
tokens = min(capacity, tokens + elapsed * refill_rate)
last_refill = max(last_refill, limiter_now)
allowed = tokens >= 1
if allowed:
    tokens -= 1
persist(tokens, last_refill)
return allowed

Use a consistent trusted time source and units, avoiding quota inflation from clock rollback or state resets. Redis Lua can execute atomically on the server, while client-side GET followed by a separate SET still races. Script atomicity does not guarantee prevention of state loss during failover.

Who owns the aggregate budget across instances and regions?

Twenty requests per second per instance allow 200 across ten instances and 800 across forty. Protect a shared 300/s dependency across normal traffic, retries, batches, and overlapping releases. One illustrative budget is 240/s total, split into 200 normal and 40 retry attempts. These are not recommended defaults; align with the dependency’s window and burst rules. Apply the downstream budget immediately before each outbound attempt, counting batch calls and retries individually. One check per incoming request cannot control the many sends it later creates. A bound on send starts is distinct from arrivals that can bunch under network delay; do not claim it guarantees the receiver’s strict arrival window.

Checking shared state per request simplifies coordination but adds a dependency. Regional budgets need disjoint allocations and a redistribution protocol. Define whether limiter unavailability permits or rejects work from business and downstream requirements. An unrestricted fallback undermines a strict aggregate cap even if normal operation is correct.

A single global counter can itself become a hot key. Splitting it into N counters while granting the whole budget to each multiplies the allowance by N. Partition the budget too, explicitly choosing among strictness, utilization, and latency. Redis Cluster still has key placement and asynchronous replication constraints.

Rate, concurrency, and work per request need separate limits

If 100 items each make two external calls, the total is 200 calls. With at most five items active and the two calls inside each item executed serially, at most five external calls are active. Parallelizing inside an item changes that bound. Distinguish total work, start rate, and active operations.

Bound work created by each request

Limit incoming and decompressed bytes → Validate items, page size, and query range → Check user and aggregate quotas → Bound active work and queue deadlines → Reject by contract or offer asynchronous work

An nginx body-size limit bounds incoming bytes, not expansive database work created from a small input. If returning 429, define client retry timing and budgets. RFC 6585 permits Retry-After, but that header alone does not guarantee a safe write retry or eventual success.

Test boundaries, concurrency, and limiter failures

Burst immediately around window boundaries and at identical timestamps; verify admission counts.
Scale from ten to forty instances with retries and batches, measuring totals at the dependency.
Inject limiter timeouts and failover; check unknown admission outcomes and lost-state behavior.
Use oversized bodies, many items, and a slow dependency; observe memory, concurrency, and queue limits.

Primary sources

The official sources below were checked on October 27, 2026. Scenarios and numerical examples are illustrative assumptions. Configuration examples are explanatory; they have not been run against live services or benchmarked.