Lesson 1 · Scaling and Trade-offs

Scalability: Vertical vs Horizontal Scaling

Understanding how systems grow — and the trade-offs each approach demands.

Big Idea

There are only two ways to serve more load: give one machine more power (vertical scaling), or give the work to more machines (horizontal scaling), and the second one only works if you first design your system so no machine has to remember anything the others don’t.

Why This Exists

Why Capacity Is A Design Decision

▸

A system built for a thousand users doesn't automatically handle a million, at some point a single server runs out of CPU, memory, or disk I/O no matter how well it's tuned, and something has to change structurally, not just get faster code.

▸

The two ways to add capacity, a bigger machine or more machines, have very different costs, limits, and failure characteristics, and picking the wrong one for a given system leads to either wasted spend or an architecture that can't actually grow further.

▸

Adding more machines only helps if requests can actually be spread across them, which is not automatic: a server holding state in local memory (like an in-process session) breaks the moment a request can land on a different machine than the one before it.

Every system eventually faces a growth problem: traffic increases, latency climbs, and the infrastructure that worked yesterday starts to buckle. The fundamental question becomes — how do you give the system more capacity?

There are two directions you can go. You can make the existing machine bigger, or you can bring in more machines. The first is vertical scaling (scaling up); the second is horizontal scaling (scaling out). Both solve the same problem, but they do so with very different architectures, cost curves, and failure modes.

Most real-world systems don't pick one and ignore the other — they start vertical for simplicity, and shift horizontal as demand outgrows what a single box can handle. Understanding why that transition happens, and what it costs, is what this lesson is about.

Think Of It Like

One Fast Cashier, Or Four Registers

Imagine a single cashier at a small shop who gets faster and faster at ringing people up, that’s vertical scaling: same one person, more capability. Now imagine the line is too long even for the fastest possible cashier, so you open three more registers, that’s horizontal scaling: more people doing the same job in parallel.

But opening more registers only works if any cashier can help any customer. If each cashier only remembers their own customers and refuses to help someone who started at a different register, adding registers doesn’t actually shorten the line, it just creates three separate confused lines.

VerticalScale Up

Upgrade the hardware of an existing server — more CPU cores, more RAM, faster storage.

Your application architecture stays unchanged. A single node handles everything.

+Zero overhead — no inter-node communication, no load balancers, no distributed state.
−Hard ceiling — hardware has physical limits. Cost grows non-linearly at high tiers.
−Single point of failure — if the machine goes down, everything goes down.

HorizontalScale Out

Add more machines to your resource pool. A load balancer distributes traffic across nodes.

State can't live in memory locally — you must go stateless.

+Theoretically unlimited — add nodes as demand grows.
+Built-in redundancy — one node fails, others keep serving.
−Distributed complexity — state coordination, consistency, and inter-node communication.

Key Terms

The Vocabulary Of Growth

Vertical scaling

Increasing the capacity of a single machine by adding more CPU, memory, or faster storage, without changing how many machines are running.

Horizontal scaling

Increasing total system capacity by adding more machines and distributing load across them, typically via a load balancer.

Statelessness

A server design where no request-specific data is kept in that server's local memory between requests, so any server in the fleet can handle any request equally.

Elasticity

The ability to automatically add or remove capacity in response to real-time demand, practical mainly with horizontal scaling of stateless components.

Single point of failure

A component whose failure takes down the whole system, a risk that vertical scaling never removes since it's still one machine.

The Shape Of It

Stateless Servers Behind A Load Balancer

The three app servers are interchangeable, which is the whole point. None of them remembers you, so the load balancer is free to send your next request anywhere. What makes that possible is the session store on the right: the state moved out of the servers and into one place they all share.

Databases are usually the hardest part of a system to scale horizontally, which is why they often get scaled vertically first, or handled with replication and sharding as a separate, more involved technique.

Feature	Vertical	Horizontal
Complexity	Low — single node	High — distributed coordination
Availability	Low — single point of failure	High — redundancy by design
Cost	Expensive at high tiers	Cost-effective (commodity hardware)
Growth Limit	Physical hardware ceiling	Theoretically unlimited
When to use	Early stages, simplicity first	When throughput exceeds a single node

Seen In The Wild

How Real Systems Pick A Direction

▪

Stack Overflow famously ran on a relatively small number of powerful, vertically scaled servers for years, a case where vertical scaling was a deliberate, working choice, not a failure to modernize.

▪

Netflix's application tier runs as thousands of stateless horizontally scaled instances behind load balancers, auto-scaling up and down with viewing demand across time zones.

▪

AWS RDS offers both vertical scaling (resizing your database instance to a bigger type) and read replicas for a form of horizontal scaling on the read path, reflecting how databases resist horizontal writes far more than app servers do.

▪

Kubernetes' Horizontal Pod Autoscaler exists specifically to automate horizontal scaling for stateless services, adding and removing container instances based on CPU or custom metrics.

Common Mistakes

Where This Usually Goes Wrong

Assuming horizontal scaling is always the right answer, when a well-tuned vertical scale is simpler, cheaper, and sufficient for many real systems.

Adding more servers behind a load balancer without first removing local server state, which causes subtle bugs (like a user randomly getting logged out) instead of the intended capacity increase.

Treating 'add more servers' as free, ignoring the real added complexity: network calls where there used to be function calls, coordination, and a load balancer that itself needs to be highly available.

Forgetting that horizontal scaling for a database is a fundamentally harder problem than for a stateless app server, and reaching for sharding before exhausting simpler options like read replicas or vertical scaling.

Try It Yourself

Find The State That Has To Move

Think of an app you use that clearly has millions of users, like a messaging app. Ask yourself: could this possibly be running on one giant machine? What would have to be true about how it stores your logged-in state for it to spread your requests across many servers safely? Write down one thing you think has to live outside any single server for that to work.

Quiz Review

Check your understanding

Question 1 of 12

Why can't you just add more servers behind a load balancer to scale a system that stores user sessions in server memory?

✓Because a session stored in one server's local memory only exists on that server. If the load balancer routes a later request from the same user to a different server, that server has no idea who the user is. Horizontal scaling requires moving that state somewhere shared, like a database or a distributed cache, so any server can serve any request.