Serving Frameworks: vLLM vs TGI vs Triton (and LitServe)
~14 min read
You almost never serve an LLM with a raw model.generate() loop. Purpose-built engines — vLLM, TGI, Triton, LitServe — add continuous batching, paged KV memory, and production plumbing. Which to pick depends on your constraints.
Serving Frameworks: vLLM vs TGI vs Triton (and LitServe) is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Never serve LLMs with a raw generate() loop — purpose-built engines add continuous batching, paged KV memory, streaming, and concurrency
- •vLLM: throughput-focused pure-LLM serving (PagedAttention + continuous batching), OpenAI-compatible API — the common default
- •TGI: Hugging Face's equivalent with a similar feature set and tight HF-ecosystem integration
- •Triton: the generalist — serves any model type / multiple backends (incl. TensorRT-LLM) behind one server, at higher config cost
- •LitServe: lightweight FastAPI-based framework for wrapping any model or custom logic with batching, when you want flexibility over a fixed engine