vLLM: Fast, OpenAI-Compatible Serving

vLLM is described in this course as a fast and easy-to-use library for LLM inference and serving — a meaningfully different tool from Ollama and LM Studio, both of which are oriented around individual, casual local usage. vLLM is built for actually SERVING models efficiently, whether that's for local development or genuine production deployment (this course notes it's covered in more detail in the LLM deployment section of the curriculum, reflecting that vLLM is as much a production-serving tool as it is a 'run locally' option).

With just a few lines of code, you can locally run LLMs — including models like DeepSeek — in an OpenAI-compatible format. This OpenAI-compatible piece matters enormously in practice: it means any code already written against the OpenAI SDK (or any tool expecting an OpenAI-shaped API) can point at a locally-running vLLM server with essentially no code changes — the same pattern shown for LM Studio's local server, but backed by vLLM's considerably more optimized serving engine underneath.

vLLM's core differentiator from Ollama and LM Studio is throughput: it implements serving-specific optimizations (efficient memory management for the KV cache, continuous batching of concurrent requests, and more, covered in depth in this curriculum's LLM deployment topic) that are specifically about serving many concurrent requests efficiently — closer to what you'd actually run behind a production API endpoint than what you'd run for one person's interactive chat session.

This makes vLLM the right choice among the 4 tools specifically when you need to actually SERVE a locally-hosted model to multiple clients or at meaningful request volume — testing a production-shaped serving setup locally before deploying, or running a local model behind an API your own application code calls — rather than just chatting with a model interactively, which is squarely Ollama or LM Studio's territory instead.

# Starting a local vLLM OpenAI-compatible server (run as a separate
# process, typically): vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B

from openai import OpenAI

# Same OpenAI SDK, pointed at vLLM's local, OpenAI-compatible endpoint —
# existing application code needs essentially zero changes
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    messages=[{"role": "user", "content": "Explain what a vector database is, briefly."}],
)
print(response.choices[0].message.content)

# vLLM's serving optimizations (continuous batching, efficient KV cache
# management) mean this same server can handle many CONCURRENT requests
# efficiently — the scenario Ollama/LM Studio aren't built to optimize for