Concept 3 · Communication and APIs·19 min read
Rate Limiting
Why the algorithm behind a limit decides how forgiving it is, not just what number is printed on it.
Big Idea
Rate limiting caps how often a client can call your system in a given window, and the algorithm you pick decides how forgiving that cap is about bursts, not just what the average rate ends up being. Two limiters advertising “100 requests a minute” can behave completely differently in the second that actually matters.
Why This Exists
Why One Client Can Become Everyone’s Outage
A single misbehaving client, whether a buggy retry loop or a deliberate scraper, can send requests fast enough to consume the capacity that every other client needs, turning one bad actor's mistake into an outage for everyone.
Without a limit, a spike in legitimate traffic, a sale going live or a post going viral, can overwhelm downstream databases and services that were sized for normal load, causing cascading failures well beyond the endpoint that got hit first.
APIs that charge per request or offer tiered plans have no way to enforce those tiers, or to give paying customers a reliable share of capacity, without some mechanism that actually counts and caps usage per client.
Think Of It Like
A Data Plan, And The Night Before It Resets
Rate limiting is like the data allowance on a phone plan. The carrier isn’t trying to stop you using the network, it’s keeping any one phone from eating the capacity of a tower everybody is sharing.
A fixed window plan gives you 10 GB that resets at midnight on the first of the month, which is why someone who checks their usage on the 31st can burn the last 10 GB that night and another 10 GB the next morning: twenty gigabytes in a few hours, out of a plan sized for ten a month. A plan with rollover data behaves like a token bucket instead. The allowance accrues steadily, stops accruing at some ceiling, and a quiet month lets you spend a burst later without ever beating the long-run rate. Same number on the bill, very different night for the tower.
The Shape Of It
A Layered Limiter In Front Of An API
Cheap and coarse at the edge, precise and shared deeper in
Anatomy
The Six Pieces
The ceiling itself: how many requests are allowed. A number with no window attached means nothing.
The span the limit applies over, per second, per minute, per day. The limit and the window are one decision, not two.
What the counter is kept per: an IP, an API key, a user ID, often a pairing of one of those with a specific endpoint.
Where the count or token balance lives. A single process can hold it in memory; a fleet of servers needs a shared store.
Where the check actually runs: at the CDN, at the API gateway, or inside the service handling the request.
What a rejected client is told. A limit the client can't see is a limit it can't cooperate with.
The Concept
Four Algorithms And What Each One Forgives
A bucket holds up to N tokens and refills at a steady rate; each request consumes one token, and requests are rejected once the bucket is empty. It naturally allows short bursts up to the bucket's capacity while enforcing a steady average rate over time, which is why it's the most commonly used algorithm in practice.
Requests enter a queue, the bucket, and are processed, or leak out, at a strictly constant rate regardless of how bursty the incoming traffic is. It smooths bursts into a steady output stream, which protects downstream systems that genuinely can't handle spikes, at the cost of added latency for queued requests during a burst.
Count requests in a fixed time window, like 'this minute', and reset the counter to zero at each window boundary. It's simple to implement and reason about, but has a real burst problem at the edges: a client can send the full limit right at the end of one window and the full limit again at the start of the next, doubling the effective rate for a brief moment.
Instead of resetting sharply at a boundary, the window slides continuously, either logging every request's timestamp or interpolating between the current and previous fixed windows. This removes the boundary-doubling problem at the cost of more memory, for the log, or a small approximation error, for the counter.
The server itself, from being overwhelmed past its capacity, and other clients, from one noisy or abusive client starving everyone else's fair share of a shared resource. A limit set only with the server's CPU in mind can still let one client monopolize the queue ahead of everybody else.
Per-IP limiting is easy but breaks down behind NATs or shared corporate addresses where many real users look like one client. Per-API-key or per-user-account limiting is far more precise, but it requires authentication to already have happened before the limit can apply, which is why the two usually sit in layers rather than competing.
A well-designed limiter returns a clear signal rather than a silent drop: an HTTP 429 status, plus headers like Retry-After or X-RateLimit-Remaining, so a well-behaved client knows it was throttled and exactly when it's safe to try again instead of guessing and retrying blindly.
When the limit has to hold across many API server instances rather than one process, counts or token balances live in a fast shared store like Redis. Each server enforcing its own local limit independently would let a client multiply the effective limit by the number of servers, which is the single most common way a correct-looking limiter turns out not to limit anything.
A coarse limit at the edge, a CDN or API gateway, blocks obvious abuse cheaply, before it reaches application servers. A finer per-endpoint or per-user limit deeper in the stack protects specific expensive operations, like search or file upload, far more tightly than a cheap health-check endpoint needs.
Key Terms
The Vocabulary Of Saying No Politely
An algorithm where a bucket refills tokens at a steady rate and each request consumes one, allowing bursts up to the bucket's capacity while capping the long-run average rate.
An algorithm that queues incoming requests and processes them at a strictly constant output rate, smoothing bursty input into steady downstream load.
An approach that counts requests within a fixed time slice and resets to zero at each boundary, simple but vulnerable to a burst doubling right at the window edge.
An approach that avoids the fixed window's edge-burst problem by continuously sliding the counted window forward, either by logging exact timestamps or interpolating between adjacent windows.
The identity a counter is tracked against. Choosing it is choosing who gets grouped together, and therefore who gets throttled together.
The standard HTTP status returned when a client has exceeded its limit, ideally paired with a Retry-After header telling the client when to try again.
The response header naming the delay, in seconds or as a date, before a retry will be accepted. It's the difference between a client backing off correctly and guessing.
Enforcing a single limit consistently across many server instances by tracking counts in a shared store, instead of each server enforcing its own local, inconsistent limit.
Worked Example
Token Bucket vs Leaky Bucket
The Shape Of It
Token Bucket, One Request At A Time
The refill runs on the clock, not on the request
Failure Mode
The Fixed Window’s Boundary Burst
A limit of 100 per window, and a client that knows where the boundary is
Both lines are enforcing “100 per minute” and both are doing exactly what they were configured to do. The client sends its full allowance at second 59 and its full allowance again at second 60, and the fixed window counter has no memory of the window it just closed, so it permits 200 requests inside a two-second span. If the server was sized for 100 a minute, it just received a minute and a half of traffic in two seconds.
A sliding window log fixes this exactly by keeping every request’s timestamp and counting only those inside the last sixty seconds, which costs one stored timestamp per request. The sliding window counter gets most of the way there far more cheaply by keeping just two counters and weighting the previous window by how much of it is still in view: a quarter of the way into the current window, it counts the current window’s requests plus three quarters of the previous one’s. It approximates, but it never has a boundary to exploit.
Reference
Which Algorithm Fits Which Job
| Algorithm | Allows Bursts | What It Stores | What Defines It | Where You've Met It |
|---|---|---|---|---|
| Token bucket | Yes, up to capacity | A balance and a timestamp per key | The general-purpose default: steady average, forgiving of a short burst | Most APIs you've used: a few calls in a row are fine, hammering it for a minute isn't |
| Leaky bucket | No, smoothed flat | A queue per key | Constant output rate; bursts wait rather than being rejected immediately | Dropping twenty photos into an upload: they all go through, one at a time, at a steady pace |
| Fixed window counter | Yes, at the boundary | One counter per key, per window | Simplest thing that works, and up to 2x the limit around a reset | Your card's daily cash limit: it resets at midnight, so late tonight plus early tomorrow is twice the cap |
| Sliding window log | No | One timestamp per request | Exact, and the most expensive to store at high volume | Three wrong PIN tries and you're locked out, where the count has to be exactly right |
| Sliding window counter | Barely | Two counters per key | Weighted blend of the current and previous window; approximate but cheap | A huge site keeping scrapers out without keeping a record of every single visit |
Failure Mode
One True Limit Across Many Servers
A limiter that works perfectly on one server often stops limiting anything the moment that server becomes five. The load balancer spreads one client’s requests across the fleet, so each instance sees only a fraction of that client’s traffic and happily counts it as well under the limit.
The fix is a shared store, usually Redis, holding the counter or token balance centrally. That introduces two new problems worth deciding about in advance rather than during an incident. The first is that the check now costs a network round trip on every single request, which is why limiters keep the operation to one atomic increment and put the store in the same region as the gateway.
The second is the question nobody enjoys: what happens when the counter store is unreachable? Failing open lets all traffic through and risks the overload the limiter exists to prevent. Failing closed rejects everything and turns a cache outage into a total outage. Most systems fail open with a coarse local fallback limit, so a store outage degrades precision rather than availability, but the point is that it is a decision, not a default.
Counterpoint
Who The Limit Is Actually Counting
Per-IP limiting is the version everyone reaches for first, because it needs nothing from the client and works before authentication. It is also the version that gets both halves of the job wrong. One office, one university, or one mobile carrier gateway can put thousands of unrelated users behind a single address, so the limit throttles them as though they were one client. Meanwhile an attacker with a few hundred addresses stays comfortably under a per-IP limit while sending a few hundred times the traffic.
Per-API-key and per-account limits fix the fairness problem, because the key is the actual identity, but they can only be applied after the request has been authenticated, which means unauthenticated endpoints like login and signup still need an IP-shaped limit in front of them. That is the real reason limits end up layered rather than unified: a coarse IP limit at the edge protects the door, and a precise per-account limit inside protects the expensive rooms.
Seen In The Wild
Limits You’ve Probably Hit
Stripe's API enforces per-account rate limits and returns clear 429 responses with guidance, documented explicitly so integrators can build correct retry logic instead of guessing at the behaviour.
GitHub's REST and GraphQL APIs use a token-bucket-style limit tied to your authentication method, with response headers reporting exactly how many requests remain in the current window.
Cloudflare offers rate limiting as an edge product, letting sites throttle or block abusive traffic patterns before a single request ever reaches the origin server.
Twitter/X has long enforced strict per-endpoint limits on its API, historically a major constraint that shaped how third-party clients and bots were built around it.
Key Points
What To Carry Forward
Token bucket allows controlled bursts up to a cap while enforcing a steady average rate, and is the most common general-purpose choice.
Leaky bucket smooths bursty traffic into a constant output rate by queueing, protecting downstream systems that can't handle spikes at all.
Fixed window counters are simple but allow nearly double the limit at the window boundary; sliding windows fix that for more memory or a small approximation.
Rate limiting protects the server's own capacity and other clients' fair share of it, not just one or the other.
A fleet of servers needs a shared store like Redis to enforce one true limit, or the limit multiplies by the number of instances.
A good limiter communicates: 429 plus retry timing, instead of silently dropping requests and leaving the client to guess.
The rate limit key decides who gets throttled together, which is why per-IP alone punishes shared networks and misses distributed abuse.
Limits belong in layers, cheap and coarse at the edge, tight and specific on the expensive endpoints.
Common Mistakes
Where This Usually Goes Wrong
Implementing a naive fixed window counter without realizing it allows nearly double the intended rate right at the window boundary.
Rate limiting only by IP address, which unfairly throttles many legitimate users sharing a NAT or corporate gateway while barely slowing down a distributed attacker.
Enforcing the limit locally per server instance in a multi-server deployment, letting a client's effective limit multiply by however many servers happen to receive its requests.
Returning a bare rejection with no Retry-After or remaining-quota information, leaving well-behaved clients to retry blindly instead of backing off correctly.
Setting one blanket limit for every endpoint, instead of tighter limits on expensive operations like search or file upload and looser ones on cheap operations like a health check.
Rejecting with a 500 or a connection reset instead of a 429, so clients and their monitoring read a deliberate throttle as a server fault and retry harder.
Treating the limiter itself as infallible, with no decision made about what happens to traffic when the shared counter store is unreachable.
“Three treats a day, my human says. But there are four humans in this house, none of them compare notes, and the paw-on-the-knee routine works on each of them separately, so my real limit is twelve. My harness is getting snug. Nobody has proposed a shared counter, and I am absolutely not going to be the one who brings it up.”
Try It Yourself
Guess The Algorithm Behind A Limit You’ve Hit
Further Reading
Where To Go Deeper
A free, local-first coach for DSA, mock interviews and system design. Zero nudges you toward the answer instead of handing it over, and nothing leaves your machine.
Algorithm by algorithm, with the trade-offs each one makes and the weighting maths behind the sliding window counter.
A production limiter described from the integrator's side: what the limits are, what a 429 looks like, and the retry behaviour expected of clients.
Quiz Review
Check your understanding
Question 1 of 5
Why does a fixed window counter allow more than the stated limit in practice, and how does a sliding window fix it?
- ✓A fixed window resets sharply at each boundary, so a client can send the full limit in the last moment of one window and the full limit again in the first moment of the next, achieving nearly double the intended rate in a short span around the boundary. A sliding window avoids this by never having a hard reset point, either tracking exact timestamps or smoothly weighting between the current and previous window, so no short span of time ever sees more than the intended limit.