API Design for LLMs: Streaming, Async and Timeouts
~13 min read
LLM endpoints behave unlike normal APIs: responses take seconds and arrive token-by-token. Good design means streaming, async concurrency, and generous-but-bounded timeouts.
API Design for LLMs: Streaming, Async and Timeouts is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •LLM responses take seconds and arrive token-by-token, so don't design the API like a fast atomic request/response
- •Stream tokens (SSE / chunked HTTP) so text appears immediately — the key metric becomes time-to-first-token (TTFT), not total time
- •Use async endpoints: LLM requests are long and I/O-bound, so a thread-per-request server exhausts its workers under concurrency
- •Bound every request: cap max output tokens, set realistic timeouts, and cancel generation on client disconnect to free the GPU slot
- •For very long jobs, prefer a submit-then-poll/webhook async job pattern over holding one HTTP connection open