Model Deployment: vLLM & LitServe
Learn deployment requirements, PagedAttention, Continuous Batching, and serve models via vLLM and LitServe.
GPU Queue Strategy:
Client request queue initialization
Requests from multiple users enter the scheduler queue.
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Explain how PagedAttention eliminates memory fragmentation
- •Compare static batching and continuous (iteration-level) batching
- •Deploy models using vLLM and configure LitServe architectures
What is it?
LLM deployment is the discipline of turning a trained model into a fast, reliable, cost-effective API — a fundamentally different problem from training, dominated by purpose-built serving engines (vLLM, TGI, Triton, LitServe) that solve continuous batching, paged KV-cache memory, and multi-GPU scheduling far better than a naive model.generate() loop ever could. Getting this right is what separates a model that works in a notebook from one that can serve real production traffic at acceptable latency and cost.
Why it exists
Standard web frameworks (like Flask) do not support iteration-level scheduling, causing massive latency and low throughput under multi-user loads.
Problem it solves
Solves VRAM waste, GPU under-utilization, request queue delays, and memory fragmentation.
Intuition
When thousands of users ask an AI questions at the same time, the server can crash. We use "vLLM" to deploy models. vLLM has two superpowers: "PagedAttention" (which organizes memory like a computer OS does, preventing wasted space) and "Continuous Batching" (which groups incoming requests immediately, so the GPU never sits idle).
Analogy
Think of standard batching like a bus that only leaves when all seats are full, forcing early passengers to wait. Continuous batching is like an escalator: people step on and off dynamically without stopping the system.
Technical explanation
PagedAttention manages the Key-Value cache by storing keys and values in non-contiguous physical memory blocks (pages), mapping them via a page table. This eliminates external fragmentation and reduces wasted VRAM from pre-allocated max-length buffers. Continuous Batching schedules generation step-by-step: newly arrived requests are injected into the active batch at the next token step, and completed requests are dropped immediately, maximizing GPU utilization.
Architecture
Serving node wrapping model weights in vLLM execution instances, connected to request queue managers and network endpoints.
Workflow
- Queue incoming request -> 2. Continuous batching scheduler maps request block -> 3. Execute step -> 4. Stream token -> 5. Free block.
Example
from vllm import LLM, SamplingParams llm = LLM(model="facebook/opt-125m") outputs = llm.generate(["Japan capital?"], SamplingParams(temperature=0.0))
Real-world usage
Deploying high-concurrency microservices to handle thousands of prompt completions per minute.
Trade-offs
Continuous batching reduces individual request latency variance but increases overall GPU power consumption.
Visual explanation
PagedAttention Memory Mapping: [Logical KV Cache Blocks] (Block 0, Block 1, Block 2) │ ▼ (Mapped via Page Table) [Physical Block Manager] (Non-contiguous GPU memory pages) ┌──────────────┬──────────────┬──────────────┐ │ Page 1243 │ Page 0054 │ Page 9871 │ │ (Allocated) │ (Allocated) │ (Allocated) │ └──────────────┴──────────────┴──────────────┘
Advantages
- —
Up to 20x higher throughput compared to Hugging Face transformers
- —
Zero external memory fragmentation
Disadvantages
- —
High hardware VRAM requirements for model loading
Common mistakes
- —
Deploying serving engines without limiting max_model_len (causes GPU OOMs)
- —
Using static request-level batching wrappers in production scale
🎤 Interview questions
Explain how Continuous Batching schedules requests at the iteration level compared to standard request-level batching.
How does GPU tensor parallelism differ from pipeline parallelism when serving large models across multiple GPUs?
📂 Subtopics
Serving Frameworks: vLLM vs TGI vs Triton (and LitServe)
You almost never serve an LLM with a raw model.generate() loop. Purpose-built engines — vLLM, TGI, Triton, LitServe — add continuous batching, paged KV memory, and production plumbing. Which to pick depends on your constraints.
~14 min
API Design for LLMs: Streaming, Async and Timeouts
LLM endpoints behave unlike normal APIs: responses take seconds and arrive token-by-token. Good design means streaming, async concurrency, and generous-but-bounded timeouts.
~13 min
Scaling Strategies: Replicas, Load Balancing and Parallelism
Scaling LLMs means two different things: fitting a big model across GPUs (model parallelism) and handling more traffic (replicas + load balancing). Autoscaling on the right metric keeps cost and latency in check.
~13 min
Cost Optimization: Token Budgets, Caching and Model Routing
LLM cost scales with tokens and model size. The three biggest levers are budgeting tokens, caching repeated work, and routing each request to the cheapest model that can handle it.
~13 min