Batching Strategies: Static, Dynamic and Continuous
~13 min read
Batching runs multiple requests through the GPU together to keep it busy. Continuous batching (used by vLLM) is the key advance: it swaps finished requests out and new ones in at every step instead of waiting for the whole batch to finish.
Batching Strategies: Static, Dynamic and Continuous is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Batching keeps the GPU busy by processing many requests together — critical because one request barely uses a large GPU
- •Static batching returns all results together, so the whole batch stalls on the single longest-generating request
- •Dynamic batching forms batches within a short time window (bounded latency) but still treats a batch as all-in/all-out
- •Continuous (in-flight) batching schedules per generation step: finished requests are evicted and waiting ones admitted immediately
- •You rarely hand-roll this — pick a continuous-batching engine (vLLM, TGI), fire concurrent requests, and tune concurrency/memory limits