Tools and Frameworks: LlamaIndex Multimodal, ColPali, and an End-to-End Example

The previous three subtopics covered multimodal RAG conceptually and architecturally; this subtopic covers concrete tools that implement it, plus a worked example tying the whole topic together.

LlamaIndex (a widely-used RAG orchestration framework) provides purpose-built multimodal RAG support that directly implements the dual-track pipeline from this topic's architecture subtopic: dedicated document parsers that extract both text AND images from PDFs/slides, multimodal-capable vector store integrations that can hold both text and image embeddings together, and query engines that automatically route retrieved images to a VLM (rather than a text-only LLM) for the final generation step. Its practical value is exactly what a good framework should provide: it implements the dual-track extraction, per-modality embedding, and combined retrieval/generation plumbing from the architecture subtopic, so you assemble a working pipeline from pre-built components rather than writing every stage from scratch.

ColPali represents a genuinely different architectural bet, worth understanding as an alternative rather than just 'another implementation of the same idea.' Instead of extracting text and images SEPARATELY and embedding them through different pipelines (this topic's dominant pattern so far), ColPali embeds each ENTIRE PAGE as a single image directly, using a vision-language model to produce the embedding — completely skipping text extraction, chunking, and OCR as separate steps. The argument behind this: text extraction is exactly where the earlier subtopic's failure modes happen (tables mangled, charts dropped, layout lost) — so ColPali's bet is that a sufficiently capable vision-language embedding model can represent a WHOLE page's combined text-plus-layout-plus-visual content directly and more faithfully than a pipeline that first tries to (imperfectly) separate a page into 'the text part' and 'the image part.' This trades away some of the dual-track pipeline's flexibility (like retrieving individually at sub-page granularity) for avoiding the earlier subtopic's extraction failure modes entirely.

A minimal end-to-end shape, tying the whole topic together: a user uploads a financial report PDF; the pipeline extracts text paragraphs and page images (or, in ColPali's approach, treats whole pages as the unit); each gets embedded appropriately; a user asks 'what was our revenue growth shown in the Q3 chart?'; retrieval surfaces the relevant page/chunk based on embedding similarity; that retrieved content (chart image included) gets passed to a VLM; the VLM reads the chart directly and generates a grounded answer, citing the specific page — the same fundamental RAG shape (retrieve, then generate grounded in what was retrieved) from this curriculum's RAG & Vectors material, just extended at every stage to keep visual content in play rather than discarding it during text extraction, which is the central idea this whole topic has built toward.

# A minimal end-to-end multimodal RAG flow, contrasting the
# dual-track approach (this topic's default pattern) against
# ColPali's whole-page-as-image alternative.

def dual_track_pipeline(document_pages: list[dict], query: str) -> dict:
    """This topic's default pattern: extract text + images SEPARATELY,
    embed each, retrieve across both (LlamaIndex-style)."""
    retrieved = []
    for page in document_pages:
        if "revenue" in page["text"].lower() or "revenue" in page.get("chart_caption", "").lower():
            retrieved.append({"page": page["page_num"], "text": page["text"],
                              "chart": page.get("chart_image")})
    return {"approach": "dual_track", "retrieved_pages": [r["page"] for r in retrieved],
            "passed_to_vlm": retrieved}

def colpali_style_pipeline(document_pages: list[dict], query: str) -> dict:
    """ColPali's approach: embed each WHOLE PAGE as one image, no
    separate text extraction step at all."""
    retrieved_whole_pages = [
        page["page_num"] for page in document_pages
        if "revenue" in page["whole_page_image_understood_content"].lower()
    ]
    return {"approach": "colpali_whole_page", "retrieved_pages": retrieved_whole_pages}

document_pages = [
    {"page_num": 12, "text": "Our revenue grew this quarter.",
     "chart_caption": "Q3 revenue chart", "chart_image": "q3_chart.png",
     "whole_page_image_understood_content": "page showing text about revenue and a Q3 chart"},
    {"page_num": 5, "text": "Company overview and mission statement.",
     "chart_caption": "", "chart_image": None,
     "whole_page_image_understood_content": "page showing company mission statement"},
]

query = "What was our revenue growth shown in the Q3 chart?"
print("Dual-track (LlamaIndex-style):", dual_track_pipeline(document_pages, query))
print("ColPali-style (whole page as image):", colpali_style_pipeline(document_pages, query))
print("\n-> Both retrieve page 12; the retrieved content (chart included)")
print("   is then passed to a VLM to generate a grounded, cited answer")