Scaling Strategies: Replicas, Load Balancing and Parallelism
~13 min read
Scaling LLMs means two different things: fitting a big model across GPUs (model parallelism) and handling more traffic (replicas + load balancing). Autoscaling on the right metric keeps cost and latency in check.
Scaling Strategies: Replicas, Load Balancing and Parallelism is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Separate two problems: model parallelism (fit a big model across GPUs) vs replicas (handle more traffic) — they have different solutions
- •Tensor parallelism splits layers across GPUs within a node (fast NVLink); pipeline parallelism spreads layers across nodes — use only when the model won't fit
- •Horizontal scaling runs multiple model replicas behind a load balancer; doubling replicas roughly doubles throughput
- •Route load-aware (least-outstanding-requests, queue-depth, or prefix affinity) rather than naive round-robin, which can hit saturated replicas
- •Autoscale on GPU-relevant signals (queue depth, GPU util, TTFT) not CPU; budget for slow cold starts by keeping warm headroom