API Design for LLMs: Streaming, Async and Timeouts

A typical REST endpoint returns in milliseconds. An LLM endpoint might take 5-30 seconds to finish a long generation, and the output is produced incrementally, one token at a time. Designing the API as if it were a fast, atomic request/response leads to terrible UX and fragile infrastructure. Three patterns make LLM APIs work well.

Streaming is the biggest UX lever. Instead of making the user stare at a spinner for 20 seconds and then dumping the whole answer, you stream tokens as they're generated so text appears immediately and flows in — the experience everyone knows from ChatGPT. The relevant latency metric shifts from total time to 'time to first token' (TTFT): as long as the first token appears quickly, the wait feels responsive even if the full answer takes many seconds. Streaming is usually implemented with Server-Sent Events (SSE) or chunked HTTP; the serving engine emits partial results and the API forwards each chunk to the client as it arrives.

Async handling is essential because LLM requests are long and I/O-bound from the server's perspective (the server is mostly waiting on the GPU/engine). A synchronous, thread-per-request server exhausts its worker pool almost immediately under concurrent load — a handful of 20-second requests block everything. An async framework (FastAPI with async endpoints, or async calls to the serving engine) lets a single worker juggle many in-flight requests while they wait, which is exactly the concurrency pattern that continuous-batching engines are built to be fed.

Timeouts need rethinking. A default 30-second HTTP timeout that's generous for a normal API can be too tight for a long generation, yet you also can't let requests hang forever and pile up. The right approach is layered: set a max output-token limit so generations are bounded by construction, set request timeouts that account for realistic worst-case generation time, and implement cancellation so that if a client disconnects, you stop generating and free the GPU slot rather than wasting compute on an answer nobody will read. For very long jobs, consider an async job pattern (submit -> poll/webhook) instead of holding one HTTP connection open.

Together these turn an LLM from an awkward fit for HTTP into a responsive service: stream for perceived speed, go async for concurrency, and bound every request so one slow generation can't take down the service.

# A streaming, async LLM endpoint with FastAPI + SSE, a bounded
# token limit, and client-disconnect cancellation.
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import asyncio

app = FastAPI()

async def token_stream(prompt: str, request: Request, max_tokens: int = 512):
    count = 0
    async for token in call_llm_engine(prompt):   # async generator of tokens
        # Stop early if the client hung up — free the GPU slot
        if await request.is_disconnected():
            break
        count += 1
        if count > max_tokens:                     # bound the generation
            break
        yield f"data: {token}\n\n"               # SSE chunk
    yield "data: [DONE]\n\n"

@app.post("/generate")
async def generate(request: Request):
    body = await request.json()
    return StreamingResponse(
        token_stream(body["prompt"], request),
        media_type="text/event-stream",
    )

async def call_llm_engine(prompt):   # stand-in for a real streaming engine call
    for word in ["Hello", " ", "world", "!"]:
        await asyncio.sleep(0.05)
        yield word