Scaling Strategies: Replicas, Load Balancing and Parallelism

'Scaling an LLM' hides two distinct problems, and conflating them leads to bad architecture. The first is: the model is too big for one GPU. The second is: one GPU can't handle the request volume. They have different solutions.

When the MODEL doesn't fit, you use model parallelism. Tensor parallelism splits each layer's weight matrices across multiple GPUs so they compute one forward pass together (great within a single node with fast NVLink interconnects; vLLM exposes this as tensor_parallel_size). Pipeline parallelism instead puts different layers on different GPUs and passes activations down the line (better across nodes, but can leave GPUs idle waiting for the pipeline to fill). These let a 70B+ model run when no single GPU has enough VRAM — but note they raise complexity and inter-GPU communication cost, so you use them only when a model genuinely can't fit (after quantization) on one device.

When the TRAFFIC is too high, you use replicas: run multiple independent copies of the model server, each on its own GPU(s), behind a load balancer that spreads requests across them. This is horizontal scaling and it's how you handle growth — double the replicas, roughly double the throughput. The load balancer's routing matters for LLMs: naive round-robin can send a request to a replica that's already saturated, so 'least-outstanding-requests' or queue-depth-aware routing works better, and session/prefix affinity (routing requests that share a prompt prefix to the same replica) can boost prefix-cache hit rates.

Autoscaling ties it together: add replicas when load rises and remove them when it falls, to control cost. The subtlety is picking the right signal. CPU utilization — the default for generic web services — is meaningless for GPU inference. Scale on GPU-relevant metrics instead: queue depth / number of pending requests, GPU utilization, or latency (TTFT creeping up means you're saturated). Also budget for cold starts: loading a multi-GB model onto a fresh GPU takes tens of seconds, so reactive autoscaling can lag demand — keep warm headroom or pre-scale for known traffic peaks.

The rule of thumb: parallelize only as much as needed to make the model fit, then scale out with replicas for throughput, route load-aware, and autoscale on queue depth or latency rather than CPU.

# A load balancer that routes to the LEAST-BUSY replica (better than
# round-robin for LLMs), plus a queue-depth-based autoscaling rule.
from dataclasses import dataclass, field

@dataclass
class Replica:
    name: str
    outstanding: int = 0   # in-flight requests right now

def route_least_busy(replicas: list[Replica]) -> Replica:
    """Send the request to whichever replica has the fewest in-flight
    requests — avoids piling onto an already-saturated GPU."""
    target = min(replicas, key=lambda r: r.outstanding)
    target.outstanding += 1
    return target

def desired_replicas(pending_requests: int, per_replica_capacity: int,
                     current: int, min_r: int = 1, max_r: int = 20) -> int:
    """Autoscale on QUEUE DEPTH, not CPU — the metric that reflects
    real LLM saturation."""
    import math
    needed = math.ceil(pending_requests / per_replica_capacity)
    return max(min_r, min(max_r, max(needed, current if pending_requests else min_r)))

pool = [Replica("gpu-0", 3), Replica("gpu-1", 1), Replica("gpu-2", 5)]
print("routed to:", route_least_busy(pool).name)          # gpu-1 (least busy)
print("scale to:", desired_replicas(120, 16, current=4))  # -> 8 replicas