Lesson 2 · Scaling and Trade-offs
Latency, Throughput & Availability
The three pillars of system observability — if you can't measure these, you can't operate.
Before you can reason about scaling, consistency, or fault tolerance, you need a shared language for measuring system health. These three metrics are that language. Latency tells you how fast the system responds. Throughput tells you how much work it can handle. Availability tells you how often it's actually reachable.
They are related but not interchangeable — and optimizing for one often creates pressure on another. Understanding their definitions, how they're measured, and where they conflict is the prerequisite to every architectural decision that follows.
When an architecture buckles under load or degrades in production, it is almost never because of an obscure syntax bug. It fails because someone miscalculated the relationship between how fast an operation completes, how many operations can run concurrently, and what proportion of them succeed over time. Treat these three as mathematical constraints that push against one another, not as isolated target numbers on a dashboard.
Metric 01
LatencyTime to complete a request
Latency is the time it takes for a single request to complete. In a distributed architecture, this is not a single number — it's a distribution. The most common mistake is reporting latency as an arithmetic mean.
If 99% of requests take 10ms but 1% take 5 seconds, your average looks healthy while a real slice of users hits timeouts. The mean hides outliers entirely.
Latency = Total Round-Trip Time
Latency = time from client sends request → client receives response
Latency is often confused with response time. Response time is the total elapsed time the client experiences. Latency is the portion of it introduced by network transit, queuing, and compute along the path. Those three pieces are additive:
Total Latency = Network Transit + Queue Wait + Execution
Knowing which term dominates tells you which fix is worth attempting. Adding a CDN does nothing for a request stuck in a connection pool, and a faster query does nothing for a client three continents away.
Bounded by physics. Light travels roughly 200 km per millisecond through fiber, and every routing hop adds more. You cannot optimize this away, you can only move the data closer.
Time spent sitting in thread pools, TCP buffers, and database connection pools waiting for a free worker. The only dynamic term, and it dominates as the system approaches saturation.
Raw CPU and I/O: parsing payloads, running business logic, reading from storage engines. This is what profilers measure and what most engineers instinctively try to fix first.
Metric 02
ThroughputRate of work processed
Throughput is the rate at which your system processes requests — measured in Requests Per Second (RPS) or Transactions Per Second (TPS).
Units follow the workload. Web services report RPS or QPS (queries per second), while data pipelines report records per second or MB/s. The unit changes, the reasoning does not.
High throughput does not imply low latency. A system can process 10,000 RPS while taking 2 seconds to respond to each. This happens when a system is heavily queued — work is being accepted, but users wait in line before processing begins.
Common mistake
Engineers often assume throughput is simply the inverse of latency. That identity holds for exactly one case: a strictly synchronous, single-threaded worker handling one request at a time. At 50ms per request:
1 request ÷ 0.05s = 20 RPS
Real architectures break that ceiling with concurrency: multiple threads, event loops, CPU cores, and distributed nodes. A 50ms endpoint on 64 concurrent workers serves 1,280 RPS without getting a single millisecond faster.
Is the system CPU-bound (compute exhausted), memory-bound (heap pressure / GC), or I/O-bound (disk or network saturated)? Each bottleneck type demands a different fix.
The point at which throughput plateaus. Beyond it, adding more load causes exponential latency spikes rather than more processed requests. This is your effective capacity ceiling.
Metric 03
AvailabilityPercentage of time the system is reachable
Availability is the percentage of time a system is functional and reachable, expressed in "nines". It is calculated as:
Availability = Uptime ÷ (Uptime + Downtime)
In distributed systems, availability is rarely binary. A system can be "up" while returning 500 errors to 5% of users. The inverse metric is Error Rate — Failed Requests ÷ Total Requests.
A request that must touch five services, each at 99.9%, succeeds only 99.5% of the time. Every hard dependency you add subtracts uptime, which is why deep synchronous call chains are so expensive.
Two redundant 99% replicas fail together only 0.01% of the time. Redundancy is the only structural move that buys nines, and it only works if the replicas do not share a failure domain.
Putting it together
The Tension Between Metrics
These metrics don't exist in isolation — optimizing for one creates pressure on the others.
To improve latency, you might cache results — increasing memory usage and cache consistency complexity. To improve availability, you add redundant nodes — which increases consistency complexity. To improve throughput, you scale horizontally — which introduces distributed state challenges covered in the previous lesson.
Grouping 1,000 individual inserts into one bulk write collapses connection and disk sync overhead, so database throughput climbs sharply. Each individual record pays for it by waiting in a memory buffer until the batch window closes.
Writing synchronously to two regions means a total loss of Region A costs zero data. It also adds a cross-region round trip of 50ms to 150ms to every single write, forever, including the 99.99% of days nothing fails.
Aggressive client retries mask transient blips and improve perceived availability. Under sustained saturation those same retries become a thundering herd that multiplies load against an already struggling system, turning a partial slowdown into a full outage.
Quiz Review
Check your understanding
Question 1 of 13
Why should you never use the arithmetic mean to report latency?
- ✓The mean hides outliers — a small number of very slow requests can be masked by the fast majority, making the system appear healthier than it is.