Calling LLM APIs: An OpenAI Example End-to-End, Including Streaming

Every idea from this unit — request/response, HTTP methods, status codes, JSON, authentication — comes together in one concrete, extremely common task: calling an LLM API from code. Walking through this end to end ties the whole unit together.

The basic (non-streaming) flow is a single POST request. You send an HTTP POST to the provider's endpoint (e.g. OpenAI's /v1/chat/completions), with your Bearer token API key in the Authorization header (authentication), and a JSON body containing your prompt/messages and settings like which model to use and how creative the output should be (temperature). The provider's server does the actual generation work (invisible to you, per the restaurant-menu abstraction from the first subtopic) and sends back a JSON response containing the generated text, plus metadata like how many tokens were used (relevant to the probability-basics and cost-optimization material elsewhere in this curriculum). You check the status code first — a 200 means you can trust the response body; a 4xx or 5xx means something went wrong and you should read the error message in the JSON body instead of the (missing) generated text.

Streaming responses solve a real UX problem: a long generation might take many seconds, and waiting for the ENTIRE response before showing anything feels sluggish (this connects directly to the 'time to first token' concept from the LLM deployment/observability material elsewhere in this curriculum). Instead of one request returning one complete JSON blob, a streaming request keeps the connection open and sends back small CHUNKS of the response as they're generated — you set stream: true in your request, and instead of getting one JSON response, you receive a sequence of small events (commonly formatted as Server-Sent Events), each containing just the next few tokens of generated text. Your code reads these chunks one at a time and can display each one immediately — this is exactly what produces the familiar 'typing' effect you see in ChatGPT and similar interfaces, rather than a long pause followed by the whole answer appearing at once.

Error handling matters especially for LLM APIs specifically because of two very common failure modes: 429 (rate limit exceeded — you're calling too fast for your account tier) and requests that simply take a long time and might time out. Production code typically wraps API calls with automatic retry logic (often with 'exponential backoff' — waiting progressively longer between retries) specifically to handle 429s gracefully, rather than failing the user's request outright the first time a rate limit is hit.

# The real (non-streaming) OpenAI call, then a streaming version --
# both require `pip install openai` and a real API key to actually run.

# --- Non-streaming: one request, one complete JSON response ---
# from openai import OpenAI
# client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])  # Bearer auth handled internally
#
# response = client.chat.completions.create(
#     model="gpt-4o-mini",
#     messages=[{"role": "user", "content": "Say hello in one sentence."}],
#     temperature=0.7,
# )
# print(response.choices[0].message.content)
# print(f"tokens used: {response.usage.total_tokens}")

# --- Streaming: many small chunks instead of one blob ---
# stream = client.chat.completions.create(
#     model="gpt-4o-mini",
#     messages=[{"role": "user", "content": "Count to 5."}],
#     stream=True,
# )
# for chunk in stream:
#     token = chunk.choices[0].delta.content
#     if token:
#         print(token, end="", flush=True)  # print each piece as it arrives

# A runnable simulation of the streaming loop, no API key needed:
import time

def fake_streamed_response(full_text: str):
    """Yields the response one small chunk at a time -- what a real
    streaming API does over the network, simulated locally."""
    for word in full_text.split():
        yield word + " "
        time.sleep(0.02)   # stand-in for network/generation latency

print("Streaming response: ", end="")
for chunk in fake_streamed_response("One two three four five"):
    print(chunk, end="", flush=True)   # display immediately, don't wait for the end
print()