Serving Frameworks: vLLM vs TGI vs Triton (and LitServe)

Serving an LLM in production is a different problem from running it in a notebook. A naive Flask endpoint calling model.generate() handles one request at a time, wastes the GPU, and falls over under load. Purpose-built serving frameworks solve the hard parts — batching, memory, streaming, concurrency — so you don't reinvent them. The main choices differ in what they optimize for.

vLLM is the throughput-focused default for pure LLM serving. Its signature features are PagedAttention (OS-style paging of the KV cache to eliminate memory fragmentation) and continuous batching (swapping requests in and out at every generation step). It exposes an OpenAI-compatible API, supports quantized models (AWQ/GPTQ), tensor parallelism across GPUs, and prefix caching. Reach for vLLM when you're serving one or a few large models and want maximum tokens-per-second per GPU.

TGI (Hugging Face Text Generation Inference) is a close competitor with a similar feature set — continuous batching, paged attention, quantization, streaming — and tight integration with the Hugging Face ecosystem and Hub. It's a strong pick when you're already deep in the HF stack and want production serving with minimal glue.

NVIDIA Triton Inference Server is the generalist. It serves ANY model type (LLMs, vision, tabular, custom ensembles) behind one server, supports multiple backends (including a TensorRT-LLM backend for maximum NVIDIA-hardware performance), model ensembles, and dynamic batching. Choose Triton when you're running a heterogeneous fleet of models and want one unified, hardware-optimized serving layer, and you can absorb its heavier configuration overhead.

LitServe (from Lightning AI) sits at a different point on the spectrum: it's a lightweight, flexible serving framework built on FastAPI that makes it easy to wrap ANY model or custom inference logic (not just LLMs) in a scalable API with batching and GPU management, without the full weight of Triton. It's a good fit when you need custom pre/post-processing, multi-model pipelines, or you want more control than a fixed LLM engine gives but less ceremony than Triton.

The decision, boiled down: maximum single-model LLM throughput -> vLLM or TGI; a diverse fleet of models on NVIDIA hardware with one serving layer -> Triton; custom logic / flexibility with less overhead -> LitServe. In all cases you're buying the same thing: someone else's battle-tested batching and memory management instead of your own.

# vLLM exposes an OpenAI-compatible server, so you run one command
# and talk to it with the standard OpenAI client — no bespoke API.
#
#   python -m vllm.entrypoints.openai.api_server \
#       --model mistralai/Mistral-7B-Instruct-v0.2 \
#       --quantization awq --max-model-len 4096

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.2",
    messages=[{"role": "user", "content": "Say hi in one word."}],
)
print(resp.choices[0].message.content)

# A LitServe example — wrap ANY custom inference logic in a scalable API:
#   import litserve as ls
#   class MyLLM(ls.LitAPI):
#       def setup(self, device): self.model = load_model().to(device)
#       def decode_request(self, req): return req["prompt"]
#       def predict(self, prompt): return self.model.generate(prompt)
#       def encode_response(self, out): return {"text": out}
#   ls.LitServer(MyLLM(), accelerator="gpu", max_batch_size=8).run(port=8000)