Handling Different Modalities: Charts, Slides, Screenshots, and Handwriting

The previous subtopic's pipeline architecture treats 'images' somewhat uniformly, but the previous topic's task-reliability discussion (vision-language-overview) already established that different visual content types have genuinely different difficulty profiles. This subtopic covers four common document-content types specifically, and the practical handling considerations each raises.

Charts embedded in PDFs (bar charts, line graphs, pie charts) require the retrieval/generation VLM to correctly read axis labels, legend mappings, and visual trends — getting the GIST right ('revenue is trending up') is usually reliable, but extracting a PRECISE specific value from a chart (per the document-understanding task category) is meaningfully less reliable, and a pipeline handling financial or scientific charts specifically often benefits from ALSO extracting the underlying data as structured numbers where possible (e.g. from an accompanying data table, if one exists in the source), rather than relying purely on visual chart-reading for precision-critical values.

Slide decks present a distinct challenge: information is often spread across TEXT, embedded images, AND spatial LAYOUT simultaneously (a slide's title, a bullet list, and a supporting diagram all convey related meaning through their position relative to each other, not just their individual content). Treating each slide as ONE combined image (rather than trying to separately extract 'the text part' and 'the image part') often preserves this spatial relationship better than a pipeline that aggressively splits every element out individually, at the cost of coarser retrieval granularity (retrieving a whole slide rather than one specific bullet point).

Screenshots (of software UIs, websites, or documents) combine rendered text with visual layout/styling — OCR (from vision-language-overview) generally handles cleanly-rendered digital screenshot text quite reliably, since the text itself is crisp and unambiguous, unlike a photograph of physical text. The harder part of screenshots is often interpreting UI STRUCTURE (which button does this label belong to, what state is this toggle in) rather than reading the text itself.

Handwritten notes are the most failure-prone category by a meaningful margin: handwriting has vastly more visual variation than any rendered font, no two people's handwriting looks alike, and quality varies enormously (a neat, careful note vs. a rushed scrawl). Production pipelines handling handwritten content typically expect meaningfully lower accuracy and often add explicit confidence scoring or human-review fallback specifically for this content type, rather than trusting automated transcription at the same confidence level as cleanly rendered digital text.

The practical design implication across all four: a production multimodal RAG pipeline benefits from detecting or classifying WHICH content type a given visual element is (chart vs. slide vs. screenshot vs. handwriting) and applying type-appropriate handling — structured-data extraction alongside chart images, whole-slide treatment for decks, confidence-aware handling for handwriting — rather than running one uniform 'extract and embed the image' step across everything.

# A content-type classifier + type-specific handling dispatcher --
# illustrating why 'treat every image the same' underperforms a
# type-aware pipeline.

CONTENT_TYPE_PROFILES = {
    "chart": {"typical_accuracy": 0.80, "needs_structured_data_backup": True},
    "slide": {"typical_accuracy": 0.85, "needs_whole_element_treatment": True},
    "screenshot": {"typical_accuracy": 0.92, "needs_ui_structure_parsing": True},
    "handwriting": {"typical_accuracy": 0.65, "needs_confidence_flagging": True},
}

def classify_content_type(element_metadata: dict) -> str:
    """Stand-in for a real classifier -- in practice this might be a
    small trained model, or heuristics based on document structure."""
    return element_metadata.get("detected_type", "chart")

def handle_visual_element(element: dict) -> dict:
    """Type-aware handling: apply the right extra step for each
    content type, rather than one uniform process for everything."""
    content_type = classify_content_type(element)
    profile = CONTENT_TYPE_PROFILES[content_type]
    result = {"content_type": content_type, "base_accuracy": profile["typical_accuracy"]}

    if profile.get("needs_structured_data_backup"):
        result["action"] = "also extract underlying structured data table if available"
    elif profile.get("needs_whole_element_treatment"):
        result["action"] = "embed the whole slide as one unit, preserving spatial layout"
    elif profile.get("needs_ui_structure_parsing"):
        result["action"] = "parse UI structure (buttons/labels/state), not just OCR text"
    elif profile.get("needs_confidence_flagging"):
        result["action"] = "flag for human review if confidence is below threshold"
    return result

elements = [
    {"detected_type": "chart"},
    {"detected_type": "slide"},
    {"detected_type": "handwriting"},
]
for el in elements:
    print(handle_visual_element(el))