Batching Strategies: Static, Dynamic and Continuous

A GPU is happiest when it's doing a lot of work at once. Running a single request through a large LLM leaves most of the GPU idle — like running a giant industrial oven to bake one cookie. Batching means processing several requests simultaneously so the oven is full. But HOW you form and manage the batch matters enormously for LLMs, because different requests finish at different times.

Static batching is the simplest: collect N requests, run them all together, and return all results when the batch is done. The problem is that LLM outputs have wildly different lengths. If request A generates 10 tokens and request B generates 500, the whole batch is held hostage until B finishes — A's GPU slot sits idle for hundreds of steps, and no new request can start. It's like a bus that won't leave until every passenger has reached their stop.

Dynamic batching (common in traditional model serving, e.g. Triton) improves the FORMING of batches: instead of waiting for a fixed count, the server waits a short time window (say 5-50 ms) and batches whatever requests arrived, so latency stays bounded under light load. This helps a lot for fixed-shape models, but it still treats a batch as an all-in, all-out unit — so it doesn't solve the ragged-finish problem for autoregressive generation.

Continuous batching (a.k.a. in-flight or iteration-level batching, popularized by vLLM and Orca) is the LLM-specific breakthrough. The scheduler works at the granularity of a single generation STEP, not a whole request. After every token step, it checks: has any request finished? If so, evict it and admit a waiting request into that freed slot immediately. The batch's membership changes continuously. No request waits for the slowest one to finish, and the GPU stays saturated. Combined with PagedAttention's flexible memory, this is what lets vLLM achieve many times the throughput of static batching at the same latency.

The practical upshot: you rarely implement this yourself. You choose a serving engine that does continuous batching (vLLM, TGI) and simply fire many concurrent requests at it; the scheduler packs them efficiently. Your job is mostly to tune batch/memory limits and concurrency, not to hand-roll the batching loop.

# Simulating the core difference: static batching stalls on the
# longest request; continuous batching frees slots every step.

def static_batch_gpu_slot_utilization(gen_lengths):
    """Everyone waits for the longest request to finish."""
    longest = max(gen_lengths)
    useful = sum(gen_lengths)            # token-steps actually generating
    reserved = longest * len(gen_lengths)  # slots held until batch ends
    return useful / reserved

def continuous_batch_gpu_slot_utilization(gen_lengths, waiting):
    """Finished slots are refilled from the waiting queue each step,
    so slots keep doing useful work instead of idling."""
    # With a full waiting queue, freed slots are immediately reused,
    # approaching ~100% useful utilization.
    return 1.0 if waiting >= len(gen_lengths) else 0.9

lengths = [10, 25, 500, 40]  # ragged output lengths
print(f"static utilization:     {static_batch_gpu_slot_utilization(lengths):.0%}")
print(f"continuous utilization: {continuous_batch_gpu_slot_utilization(lengths, waiting=20):.0%}")

# In practice you just point concurrency at a continuous-batching engine:
#   from vllm import LLM
#   llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.2")
#   llm.generate(list_of_many_prompts, params)  # scheduler batches them