Lesson 4 · Scaling and Trade-offs

Core Concepts of Load Balancing

Distribute traffic, eliminate single points of failure, and scale without downtime.

Load balancing is the process of distributing incoming network traffic across a group of backend servers to ensure no single server bears too much demand. By acting as a reverse proxy, the load balancer prevents bottlenecks, eliminates single points of failure, and allows for the seamless addition or removal of resources — directly supporting the horizontal scaling patterns discussed in lesson one.

The choice of where the load balancer sits in the OSI model and how it decides which server to use next are the two decisions that determine the trade-offs you accept.

The Big Picture

The Basic Shape of Every Web App

Clients make requests. Servers handle them. Put a load balancer in front and that traffic spreads across many identical servers, so one box going down doesn't take the whole app down with it.

Pretty much every design question you'll ever see — from a chat app like WhatsApp to a live streaming platform like Twitch — starts with some version of this pattern. The more detail you layer on top, the more specialized the design gets, but the skeleton stays the same.

The skeleton every design starts from

Client
Single address
↓
Load Balancer
Routes each request
↓
Server 1
Server 2
Server 3
The client doesn't care how many servers exist. It talks to a single address and trusts the load balancer to route things. That abstraction is the entire point.

Distribution Logic

AlgorithmsTraffic Distribution Strategies

The algorithm determines how the load balancer selects the "next" server for each incoming request. The right choice depends on whether your servers are homogeneous, whether connections are short- or long-lived, and whether clients need to consistently land on the same server.

Round RobinSimple

Requests are distributed sequentially across available servers. Works well when all servers have identical hardware and request processing times are uniform. Breaks down when requests have variable processing costs.

Least ConnectionsAdaptive

The balancer tracks active connections and sends new requests to the server with the fewest. Superior for long-lived connections (WebSockets, streaming) or scenarios where request processing times vary significantly.

IP HashSticky

The client's IP address is hashed to deterministically map it to a specific server. Ensures session stickiness — useful if the server caches user-specific data in memory. Can cause uneven load if a small number of IPs drive disproportionate traffic.

WeightedCapacity-aware

Each server is assigned a weight proportional to its capacity. A server with twice the RAM and CPU might receive weight 2, getting twice the traffic. Use when your server fleet is heterogeneous.

Load balancer as reverse proxy

Client A
Client B
Client C
→
→
→
Load Balancer
Algorithm routes each request
→
→
→
Server 1
Server 2
Server 3

OSI Model

Layer 4 vs. Layer 7 Balancing

Load balancers operate at different layers of the OSI model, which changes how routing decisions are made. The lower the layer, the less information the balancer has — and the faster it operates. The higher the layer, the more intelligent the routing — at the cost of inspection overhead.

Layer 4 · Transport

Routes traffic based on network-level data: IP addresses and TCP/UDP ports. The balancer does not inspect packet contents. Minimal processing means extremely high throughput and low latency.

Example: Route all traffic on port 80 to a cluster of web servers — regardless of URL path or HTTP headers.
Layer 7 · Application

Inspects the actual content of the request: HTTP headers, cookies, URL paths, or even the request body. Enables content-based routing to different backend pools.

Example: Route /images to a media server cluster, while /billing routes to a PCI-compliant payment service.
Layer 7 introduces higher latency because the load balancer must fully parse the HTTP request before making a routing decision. The trade-off is worth it for microservices architectures where different services require specialized infrastructure.

Fault Tolerance

Health Checking and Failover

A load balancer is only useful if it knows which servers are actually alive. It performs continuous health checks — periodically sending a probe request (typically an HTTP HEAD or GET to a /health endpoint) to every backend server.

If a server fails to respond or returns a 5xx error, the balancer marks it as unhealthy and stops routing traffic to it. Once it passes health checks again, it's automatically reinstated. This mechanism is the bedrock of system availability.

If your design requires high availability, you must assume servers will fail. The load balancer's job is to hide that failure from the end-user by redirecting traffic in real-time — before the user notices anything is wrong.
Health checks also enable zero-downtime deployments: drain a server, deploy, wait for it to pass health checks, then bring it back into rotation. No traffic is ever sent to a server that isn't ready.

Reference

Comparison of Balancing Strategies

StrategyBest Use CaseDownside
Round RobinUniform, stateless requests on identical serversIgnores actual server load
Least ConnectionsVariable request duration, WebSocketsSlightly higher balancer CPU overhead
IP HashSession-dependent applicationsUneven load from heavy-traffic IPs
WeightedHeterogeneous server fleetsRequires manual weight tuning
Layer 4Maximum throughput, static traffic routingNo content-based routing
Layer 7Microservices, content-based routingHigher latency due to packet inspection

Quiz Review

Check your understanding

Question 1 of 8

What is the primary purpose of a load balancer in a distributed system?

  • ✓To distribute incoming traffic across multiple backend servers, preventing any single server from being overwhelmed, eliminating single points of failure, and enabling seamless horizontal scaling.