Concept 3 · Scaling and Trade-offs·23 min read
CAP Theorem & Trade-offs
When the network fails, you must choose — and there is no middle ground.
Big Idea
CAP isn’t a menu you pick freely from. Network partitions happen whether you plan for them or not, so the real choice every distributed system makes is what happens during one: keep answering and risk being wrong, or refuse to answer until you can be sure.
The CAP theorem states that in the presence of a network partition, a distributed system can only provide either Consistency or Availability — but not both. It defines the constraints every distributed data store must accept by forcing a choice about how the system behaves when its components cannot communicate.
This is the first theorem in distributed systems that architects internalize as a hard constraint, not a preference. Understanding it doesn’t give you a recipe — it gives you a lens for evaluating trade-offs every time you choose a data store or design a failure mode.
Why This Exists
The Decision You Cannot Opt Out Of
The instant a system spans more than one machine connected by a network, that network can fail: a link drops, a data center loses connectivity, a router misbehaves. The system has to decide what to do in that moment, and no distributed system avoids that decision entirely.
Consistency sounds like a single obvious good thing to want, but it comes in graded levels, and the strongest form is directly in tension with staying available and staying fast. Treating it as free or all-or-nothing produces designs that quietly promise more than they can deliver.
CAP is one of the most frequently misquoted ideas in system design. Misreading what partition tolerance means leads to conclusions like “we chose CA”, which is not a real option once a network partition is possible at all.
Think Of It Like
Two Stores, One Inventory Count, One Phone Line
Two people run the same small store in two different towns, sharing one inventory count and staying in sync over a phone line. While the line works this is easy — they call before every sale to confirm current stock.
Now the line goes dead. Each of them has a choice: stop selling that item until the line is back (consistent, but that item is now unavailable in both towns), or keep selling from the last number they had (available, but both towns might sell the last unit and one customer gets disappointed later).
The dead phone line is the partition. What each store does during the outage is the whole CAP theorem in one story.
The Framework
Defining the Three Pillars
CAP as a triangle of trade-offs
The Trade-off
CP vs. AP — Choosing Your Failure Mode
When a network partition splits your nodes into isolated groups, they can no longer coordinate. At that point, you must decide: does the system stop serving requests to avoid returning incorrect data, or does it continue serving requests with whatever data it currently holds?
This is not an engineering oversight — it is a fundamental constraint of distributed computing. The choice you make defines your system’s behavior under failure.
If a node cannot verify its data is current with the other side of the partition, it returns an error or times out rather than risk serving stale information.
If a node is partitioned, it continues accepting writes and serving reads using whatever data it has. Nodes may diverge and must reconcile once the partition heals.
Decision tree during a network partition
Refining the C
The Spectrum of Consistency Models
CP and AP describe a decision made at the moment a partition hits. But consistency itself is not a single property. It is the contract between a data store and its clients about the ordering and visibility of writes, and that contract comes in degrees.
The C in CAP names the strictest point on that gradient. Real systems rarely apply one point across an entire application — they run different guarantees for different workflows, paying for strictness only where the business actually needs it.
The strict end
The strongest single-object model. Every operation appears to take effect atomically at one instant between its invocation and its completion. Once a write completes, any later read — no matter which node serves it — returns that value or a newer one.
Relaxes real time. Operations do not have to line up with a global clock, but every client observes all operations in the same relative order. Cheaper to maintain, and enough for many coordination problems.
The loose end
Eventual consistency guarantees only that if writes stop, all replicas converge on the same value. Until then, concurrent reads sent to different nodes can return stale data, mutations out of order, or values that flatly contradict each other.
Between full linearizability and raw eventual convergence sit the client-centric guarantees. Each is far cheaper than linearizability, and each buys one specific property that users actually notice when it is missing.
A client always sees its own updates. Change your profile photo and a reload shows the new one, even while other users still see the old one.
A client never moves backwards in time. Once it has observed a value, later queries never hand back an older one.
Nobody sees a write without the writes it depends on. A reply never appears before the question it answers.
Replication behavior under network partition
Beyond The Partition
PACELC: The Trade-off That Never Fully Goes Away
CAP only describes behavior during a partition, which is why it is so easy to under-design for: the system looks fine right up until the day the network doesn’t. PACELC adds the other half of the picture.
Keeping replicas strongly consistent means coordinating across them, and coordination costs time. So even with a perfectly healthy network, you are still paying for consistency — just in latency instead of availability.
if Partition → trade Availability against Consistency
else → trade Latency against ConsistencyRefuses during a partition, and pays coordination latency the rest of the time. Spanner, etcd and Zookeeper sit here: every write waits on a quorum even when nothing is broken.
Answers during a partition, and answers from the nearest replica the rest of the time. Cassandra and DynamoDB default here, which is why a read can be stale even on a perfectly healthy day.
Key Terms
The Vocabulary Of The Trade-off
A distributed system experiencing a network partition can guarantee at most two of consistency, availability, and partition tolerance. Since partitions are unavoidable, the live choice is between consistency and availability.
A failure where network communication between nodes is lost or delayed, splitting the system into groups that cannot talk to each other.
Every read reflects the most recent write regardless of which replica answers — the strictest and most expensive model to maintain across replicas.
Replicas converge to the same value over time if writes stop, but a read immediately after a write may temporarily see stale data.
An extension of CAP: even without a Partition, systems trade Latency against Consistency during normal operation, not just Availability against Consistency during a partition.
Under the Hood
Quorum: How a Node Knows to Say No
A partitioned node cannot tell whether the other side is dead or merely unreachable, so it never tries to guess. Consensus protocols such as Raft and Paxos require a strict majority of the cluster, ⌊N/2⌋ + 1 out of N nodes, to agree before a write is committed. A node that cannot reach that many peers refuses to act.
This is what prevents split-brain, the failure mode where two disconnected halves of a cluster both believe they are in charge, both accept writes, and diverge into two irreconcilable versions of the truth. Since only one side of a partition can hold a majority, only one side is ever allowed to make progress.
A 5-node cluster split across two data centers
Quorum is ⌊5/2⌋ + 1 = 3 nodes. The link between the two sites goes down.
Elects a leader and keeps serving reads and writes. This side holds the authoritative state.
Every write and every linearizable read sent to N4 or N5 is rejected or blocked until the partition heals.
Real-World Example
A Banking Application Under Partition
A user has $100 in their account. The system replicates this balance across two data centers. A network partition cuts off communication between them. The user attempts a withdrawal.
The AP Side
Conflict Resolution When Replicas Diverge
Choosing AP is not the end of a decision — it is the start of a second one. Node B accepted the write while it was isolated, so once the link comes back there are two versions of the same record and no node that witnessed both.
Every highly available store therefore ships a reconciliation strategy. Which one it picks decides whether divergence costs you data or merely costs you code.
Approach 1 — Last-Write-Wins
Approach 2 — Vector Clocks and Version Vectors
// One version descends from the other
V1 = [ a:1, b:1 ] V2 = [ a:2, b:1 ]
every entry in V1 is ≤ V2, and one is strictly less
→ V1 happened before V2, so V2 wins and V1 is safe to drop
// Neither descends from the other
V1 = [ a:2, b:1 ] V2 = [ a:1, b:2 ]
neither vector dominates the other
→ concurrent write, unresolvable without domain knowledge
→ both versions surface to the application as siblingsApproach 3 — Conflict-Free Replicated Data Types
Arrival order does not change the result.
Grouping of merges does not change the result.
Applying the same update twice changes nothing.
A counter supporting increments and decrements, by keeping one grow-only tally per node for each direction and summing them.
A set supporting adds and removes, where each element carries a timestamp that settles membership conflicts.
An observed-removed set, where each add is tagged with a unique id so a concurrent add always beats a concurrent delete.
Seen In The Wild
Which Side Real Systems Picked
DynamoDB and Cassandra are classic AP-leaning systems. They default to eventual consistency and stay available during network trouble, which fits shopping carts and activity feeds.
Relational databases configured with synchronous replication, and systems like Google Spanner (which uses tightly synchronized clocks to approximate global strong consistency), lean CP — correctness first, at some availability and latency cost.
Zookeeper and etcd, used for distributed configuration and leader election in systems like Kubernetes, are explicitly CP: they would rather stop answering than risk two nodes both believing they are the leader.
Amazon’s original Dynamo paper, the ancestor of DynamoDB and Cassandra, was built to keep the shopping cart available during partitions — preferring “the cart might briefly show slightly stale items” over “the customer cannot shop right now”.
Key Points
What To Carry Forward
CAP says a distributed system can guarantee at most two of consistency, availability, and partition tolerance, but partition tolerance is not optional — so the live trade-off is CP versus AP during an actual partition.
CP systems refuse or delay answers during a partition to avoid returning wrong data; AP systems keep answering with possibly stale data and reconcile afterwards.
Strong consistency is the easiest model to reason about and the most expensive to provide across distant replicas; eventual consistency is weaker, far cheaper, and often sufficient.
The trade-off only bites during an actual partition. Under a healthy network most systems offer both consistency and availability, which is exactly why under-designing for it stays invisible until the one day it does not.
Different data in the same system deserves different points on the spectrum — a balance and a like count should not share a consistency model.
PACELC extends the idea: even with no partition, there is a latency-versus-consistency trade-off during ordinary operation.
Common Mistakes
Where This Usually Goes Wrong
Claiming a system “chooses CA”, treating partition tolerance as optional, when any real distributed system eventually faces a partition and needs defined behavior for it.
Assuming eventual consistency is “basically strong consistency, just a little slower”, when the actual guarantee — convergence only if writes stop — allows meaningfully stale or conflicting reads under continuous load.
Applying one consistency model to an entire system, when a financial balance and a like count clearly belong at different points on the spectrum.
Using CAP to argue about normal-operation latency trade-offs. CAP is about partition-time behavior; the latency argument is PACELC.
“Both of my humans feed me dinner. The weekend they stopped comparing notes, I got fed twice a night. Each of them stayed available, neither of them was wrong, and the divergence was only detected at the vet.”
Try It Yourself
Two Features, Two Failure Modes
Summary
Choose The Failure You Can Live With
Quiz Review
Check your understanding
Question 1 of 16
Why is it inaccurate to say a system 'chooses CA' in the CAP theorem?
- ✓Because partition tolerance isn't a feature you can opt out of — network partitions are a real, unavoidable failure mode in any system with more than one node communicating over a network. 'Choosing CA' would mean assuming partitions never happen. The actual choice a system makes is what to do when one does: stay consistent by refusing some requests (CP), or stay available by answering with possibly stale data (AP).