Using VLMs in Practice: API Usage and Prompt Design for Images

The previous three subtopics covered VLM theory and task categories; this subtopic covers the practical mechanics of actually calling a vision-capable model through an API, building directly on this curriculum's api-basics prerequisite material.

Both OpenAI's and Anthropic's vision-capable APIs follow the same general request shape as their text-only chat APIs (from api-basics) — a list of messages, where each message's content can now include an image alongside text, rather than text alone. Images are typically provided one of two ways: as a publicly accessible URL (the API fetches the image itself), or as base64-encoded image DATA embedded directly in the request body (useful for local files, or images that shouldn't be hosted publicly). The response comes back as ordinary generated text, exactly like any other chat completion — the 'vision' part only affects what goes INTO the request, not the shape of what comes out.

Prompt design for images has specific practices worth knowing, distinct from pure-text prompting. Be specific about what to look at — a vague 'what's in this image?' invites a generic caption-style answer, while 'what is the value in the top-right cell of this table?' targets the VQA-style precision from the previous subtopic's task breakdown. For multiple images in one request, explicitly reference which image a question concerns ('in the SECOND image, is the door open or closed?') rather than assuming the model will correctly track which image you're asking about implicitly. For document/OCR-style tasks specifically, higher image RESOLUTION genuinely matters for accuracy — a heavily compressed or low-resolution screenshot can make fine text or small table cells illegible to the model in exactly the way it would be harder for a human to read too, so it's worth checking a provider's documented resolution/tiling behavior rather than assuming any image quality works equally well.

Cost is worth flagging explicitly, connecting to the cost-optimization material in this curriculum's llm-deployment topic: images are typically converted into a meaningful number of 'tokens' for billing purposes (often scaling with resolution — a higher-resolution image costs more tokens than a smaller one), so a request with several large images can cost substantially more than an equivalent text-only request, and this is worth factoring into cost estimates for any production feature that sends images routinely rather than occasionally.

The practical takeaway: using a VLM through an API is structurally the SAME request/response pattern as any other LLM call (per api-basics), with images as an additional input type — the real skill is in the task-specific prompt precision (from the previous subtopic's reliability differences) and awareness of the resolution/cost tradeoffs that don't come up at all with text-only requests.

# The shape of a real vision-API request (OpenAI-style), plus a
# comparison of vague vs. specific prompting for a VQA-style task.

import base64

def build_vision_request(text_prompt: str, image_url: str = None,
                         image_base64: str = None) -> dict:
    """Same message-list shape as a text-only chat request (api-basics),
    with an image included in the 'content' list alongside text."""
    content = [{"type": "text", "text": text_prompt}]
    if image_url:
        content.append({"type": "image_url", "image_url": {"url": image_url}})
    elif image_base64:
        content.append({"type": "image_url",
                        "image_url": {"url": f"data:image/png;base64,{image_base64}"}})
    return {"model": "gpt-4o", "messages": [{"role": "user", "content": content}]}

# Path 1: reference a publicly hosted image by URL
request_via_url = build_vision_request(
    "What is the value in the top-right cell of this table?",
    image_url="https://example.com/quarterly_report_table.png",
)
print("Request (via URL):", request_via_url)

# Path 2: embed a local file directly as base64
fake_image_bytes = b"not a real PNG, just illustrating the encoding step"
encoded = base64.b64encode(fake_image_bytes).decode("utf-8")
request_via_base64 = build_vision_request(
    "Transcribe the handwritten note in this image.", image_base64=encoded,
)
print("\nRequest (via base64, truncated):",
      {**request_via_base64, "messages": "[...]"})

# Prompt specificity comparison
vague_prompt = "What's in this image?"
specific_prompt = "In the SECOND image, is the office door open or closed?"
print(f"\nVague prompt (invites generic caption): {vague_prompt!r}")
print(f"Specific prompt (targets VQA-style precision): {specific_prompt!r}")