October 2021 post-incident analysis
A real network partition: automated failover promoted a cross-country database primary, both sides accepted writes the other never saw, and failing back stopped being safe.
Case Studies
The same trade-offs the concepts cover, met by real teams at real scale. Postmortems where it went wrong, write-ups where it went right. Filter by theme, and follow the concept link to the page behind it.
21 case studies
A real network partition: automated failover promoted a cross-country database primary, both sides accepted writes the other never saw, and failing back stopped being safe.
The paper that made availability the default: eventual consistency, vector clocks, and a shopping cart that must stay writable while the network is broken.
Concurrent edits from many clients converging on one document, and why they chose their own approach over off-the-shelf conflict resolution.
What it takes to retry a payment safely when the first attempt might have succeeded and you cannot tell.
The canonical caching-at-scale paper: leases against stampedes, regional pools, and what happens when a cold cluster takes live traffic.
Choosing a shard key for real data, and the migration it took to get there without downtime.
How far a single primary plus read replicas goes before sharding becomes unavoidable.
Hot partitions, the limits they hit on Cassandra, and what moving the whole message store bought them.
Unique IDs without a central counter, the problem you inherit the moment auto-increment stops working across shards.
Partitioning space instead of rows, so proximity queries stay cheap as the map fills up.
Keeping search indexes, caches, and derived stores in step with the database that owns the truth.
Copying a 28 GB search index from one primary to every replica made replication time grow linearly with the fleet and pinned the primary's network card. Letting the replicas seed each other over BitTorrent decoupled the two.
A layer 4 load balancer designed in the open, including how connections survive a backend coming and going.
Circuit breakers, bulkheads, and fallbacks: how one failing dependency is stopped from taking the product with it.
A permissions change doubled the size of an internally generated config file, it blew past a fixed preallocation, and the proxy panicked. Internal input is still input.
A primary database node failed and the cluster manager declined to promote any replica, because one Pacemaker colocation rule ranked every candidate at negative infinity. A failover budgeted at two minutes ran for nearly two hours.
A network change routed EBS traffic onto a secondary network that could not carry it. When connectivity returned, the isolated nodes all tried to re-mirror at once, and the storm filled one zone and then the region's control plane.
Idempotency keys, and why any API a client will retry needs them before it needs anything else.
Four different limiters, each protecting against a different way callers overwhelm you.
Holding millions of long-lived connections open, and what has to give when every client expects instant delivery.
Millions of trades a day on Postgres, Redis, and Go, as the counterweight to every design that assumes scale demands exotic infrastructure.