Common Tasks: Image Captioning, Visual QA, Document Understanding, and OCR

The previous two subtopics covered HOW vision-language models work architecturally; this subtopic covers WHAT you actually ask them to do in practice — four task categories that account for the large majority of real-world VLM usage.

Image captioning is the most basic task: given an image, generate a natural-language description of what's in it. This is the task CLIP-adjacent research originally targeted, and it's a genuinely useful building block (accessibility alt-text generation, content indexing), but it's also the LEAST demanding of the four — a caption just needs to be broadly accurate, not precisely reasoned about.

Visual Question Answering (VQA) raises the bar: instead of an open-ended description, the model answers a SPECIFIC question about the image ('how many people are wearing hats?', 'what color is the car in the background?'). This requires genuinely attending to the particular detail the question asks about, rather than producing a generically-plausible-sounding description — a model can produce a perfectly fluent caption that completely misses the specific detail a VQA question targets, which is exactly why VQA is treated as a meaningfully harder benchmark than captioning.

Document understanding extends VQA to structured, information-dense inputs specifically — reading a table and answering a question about a specific cell, interpreting a chart's trend, or extracting a specific field from a scanned form or invoice. This is meaningfully harder than natural-photo VQA because documents pack dense, precisely-positioned information (a table's row/column structure, a chart's axis labels) that the model must parse accurately, not just perceive in a general sense — this is directly the capability the multimodal-rag-pipeline companion topic depends on for handling PDFs with charts and tables.

OCR (Optical Character Recognition) — reading the literal TEXT present within an image (a photographed sign, a screenshot of a document, handwritten notes) — is worth distinguishing from document UNDERSTANDING specifically: OCR is about accurately transcribing what characters are present, while document understanding is about interpreting the STRUCTURE and MEANING of that information once transcribed. A model can nail OCR (perfectly transcribe every word on an invoice) while still failing document understanding (misidentifying which transcribed number is the total versus the tax line) — they're related but genuinely distinct capabilities, and modern VLMs vary in how reliably they handle each independently.

Reliability generally decreases roughly in the order presented above — captioning is the most reliable, OCR and document understanding on messy real-world documents (skewed scans, handwriting, complex multi-column layouts) remain the most failure-prone in practice, which is exactly why production systems handling these harder tasks often still combine a VLM with dedicated OCR tooling or verification steps rather than trusting a single VLM call end to end.

# Modeling the four task categories as distinct request TYPES with
# different expected reliability -- illustrating why a production
# system might route them differently.

TASK_RELIABILITY_PROFILE = {
    "captioning": {"difficulty": "low", "typical_accuracy": 0.95,
                    "example": "Describe what's in this image."},
    "visual_qa": {"difficulty": "medium", "typical_accuracy": 0.85,
                  "example": "How many people are wearing hats in this photo?"},
    "document_understanding": {"difficulty": "high", "typical_accuracy": 0.78,
                               "example": "What was Q3 revenue according to this chart?"},
    "ocr": {"difficulty": "variable", "typical_accuracy": 0.90,
            "example": "Transcribe all text visible in this scanned page."},
}

def should_add_verification_step(task_type: str, accuracy_threshold: float = 0.9) -> bool:
    """Tasks below the reliability threshold get an extra verification
    step (e.g. a dedicated OCR tool cross-check) rather than trusting
    a single VLM call end-to-end."""
    profile = TASK_RELIABILITY_PROFILE[task_type]
    return profile["typical_accuracy"] < accuracy_threshold

for task, profile in TASK_RELIABILITY_PROFILE.items():
    needs_verification = should_add_verification_step(task)
    print(f"{task:22s} difficulty={profile['difficulty']:9s} "
          f"accuracy~{profile['typical_accuracy']:.0%}  "
          f"needs_extra_verification={needs_verification}")
    print(f"  example: {profile['example']!r}")