Why Multimodal RAG: The Limits of Text-Only RAG for Real Documents

The standard RAG pipeline (covered extensively elsewhere in this curriculum's RAG & Vectors material) chunks documents into TEXT, embeds that text, and retrieves relevant chunks at query time. This works well for documents that are genuinely mostly prose. It works considerably worse for the kind of real-world documents that are extremely common in practice: financial reports full of tables, slide decks built around charts and diagrams, technical manuals with annotated figures, and scanned forms.

The core problem is that a standard text-extraction step (pulling raw text out of a PDF, say) either drops non-text content ENTIRELY (a chart becomes nothing — no text representation exists for it at all in the extracted output) or mangles structured content badly (a table's rows and columns often get flattened into a confusing run-on sequence of numbers and labels, losing the row/column relationships that gave the numbers their actual meaning — '2023 Revenue $4.2M 2024 Revenue $5.1M' extracted as a flat string loses the structure that made clear WHICH number belongs to WHICH year). A chunk that contains 'see Figure 3 below' with no representation of what Figure 3 actually shows is nearly useless as retrieved context — the reference is there, but the actual information the text is pointing at is gone.

This isn't a niche edge case — for many real document types (earnings reports, scientific papers, product spec sheets), a significant fraction of the document's ACTUAL informational content lives specifically in charts, tables, and figures, not in the surrounding prose. A text-only RAG pipeline applied to these documents is systematically blind to exactly the content a user is often asking about ('what was our revenue growth rate shown in the chart on page 12?').

Multimodal RAG addresses this directly by extending the standard pipeline to also retrieve and reason over the VISUAL content itself — treating page images, extracted figures, and tables as first-class retrievable objects alongside text chunks, rather than discarding them during text extraction. This connects directly to the vision-language-overview companion topic's document-understanding capability (VLMs that can read a chart's trend or a table's specific cell) — multimodal RAG is essentially the RAG pipeline pattern (retrieve relevant content, then generate an answer grounded in it) extended to use a VLM's visual understanding as part of both the retrieval and generation steps, rather than relying exclusively on a text embedding model that never sees the actual pixels at all.

# Illustrating the CORE failure mode: naive text extraction destroys
# a table's structure, losing the information a user actually needs.

def naive_text_extraction(page_content: dict) -> str:
    """Simulates flattening a structured table into raw text --
    exactly what many basic PDF text extractors do."""
    flattened_parts = []
    for row in page_content["table_rows"]:
        flattened_parts.extend(str(cell) for cell in row)
    return " ".join(flattened_parts)

page = {
    "table_rows": [
        ["Year", "Revenue", "Growth"],
        ["2023", "$4.2M", "--"],
        ["2024", "$5.1M", "21%"],
    ],
    "chart_description": None,   # a naive extractor has NOTHING for the chart
}

flattened = naive_text_extraction(page)
print("Naive extracted text (structure lost):")
print(f"  {flattened!r}")
print("  -> Which number is the 2024 revenue? Which is the growth rate?")
print("     The row/column relationship that answered this is GONE.\n")

def multimodal_representation(page_content: dict) -> dict:
    """A multimodal approach keeps the table's STRUCTURE (not just a
    flattened string) and treats the chart as a retrievable object
    with its own visual understanding, rather than discarding it."""
    return {
        "structured_table": page_content["table_rows"],   # rows/columns preserved
        "chart_understood_via_vlm": "Chart shows revenue growing from $4.2M to $5.1M, a 21% increase",
    }

mm_repr = multimodal_representation(page)
print("Multimodal representation (structure preserved):")
print(f"  structured_table: {mm_repr['structured_table']}")
print(f"  chart understanding: {mm_repr['chart_understood_via_vlm']!r}")