Pipeline Architecture: Extract, Embed Separately, Retrieve, and Combine

The previous subtopic established WHY multimodal RAG is needed; this subtopic covers the actual pipeline shape that implements it, extending the standard RAG workflow (covered elsewhere in this curriculum) at each of its stages.

Extraction, dual-tracked: rather than one text-extraction step, a multimodal pipeline runs TWO parallel extraction processes on the same source document. The text track pulls out prose paragraphs as usual (via the standard chunking strategies covered elsewhere in this curriculum). The image track identifies and extracts visual elements specifically — full page renders (useful when layout itself carries meaning), cropped figures/charts/tables as separate images, or both, depending on the pipeline's design.

Embedding, per modality: text chunks get embedded with an ordinary text embedding model (from embeddings-basics). Images get embedded with an IMAGE-capable embedding approach — either a CLIP-style image encoder (from the vision-language-overview companion topic, producing embeddings in a shared image/text space) or, increasingly, by having a VLM first GENERATE a detailed text description of the image's content, then embedding THAT generated description with an ordinary text embedding model (a workaround that reuses standard text-embedding infrastructure rather than requiring a genuinely multimodal embedding model). Both text chunks and image representations typically get stored in the same vector database (or a coordinated pair of them), tagged with metadata indicating their source page and modality.

Retrieval, across both modalities: an incoming query gets embedded (as text, typically), and retrieval searches across BOTH the text-chunk embeddings and the image representation embeddings, returning a combined ranked list — a query about 'the revenue trend' might surface both a relevant prose paragraph AND the specific chart image showing that trend, since both were embedded into comparable spaces.

Generation, combining everything: whatever mix of text chunks and images the retrieval step surfaces gets passed to a VLM (not a text-only LLM) for the final generation step, since the VLM needs to actually SEE the retrieved images directly — not just a text description of them — to reason precisely about their content (per the document-understanding task category from vision-language-overview). This is the key architectural difference from standard RAG's final step: the assembled context passed to generation is genuinely multimodal (text AND images together), not text-only, and the model doing the generating must itself be vision-capable to make use of that.

The overall shape closely mirrors standard RAG's extract-chunk-embed-retrieve-generate pipeline — the genuinely new engineering work is specifically in handling TWO parallel tracks (text and image) through extraction and embedding, then reconverging them at retrieval and generation.

# Implementing the dual-track pipeline shape: separate extraction and
# embedding for text vs images, unified retrieval across both, VLM
# generation over the combined multimodal context.

def extract_text_chunks(document: dict) -> list[dict]:
    return [{"type": "text", "content": p, "page": document["page"]}
            for p in document["paragraphs"]]

def extract_images(document: dict) -> list[dict]:
    return [{"type": "image", "content": img, "page": document["page"]}
            for img in document["figures"]]

def embed_text(text: str) -> list[float]:
    """Ordinary text embedding model (embeddings-basics)."""
    return [len(text) % 10 / 10, text.count("revenue") / 5]   # toy stand-in

def embed_image_via_vlm_caption(image_placeholder: str) -> list[float]:
    """Common workaround: have a VLM caption the image, then embed
    the CAPTION with a normal text embedder -- reuses text-embedding
    infrastructure instead of needing a native image embedding model."""
    caption = f"Chart showing data from {image_placeholder}"   # simulated VLM caption
    return embed_text(caption)

document = {
    "page": 12,
    "paragraphs": ["Our revenue grew significantly this year."],
    "figures": ["revenue_chart.png"],
}

text_items = extract_text_chunks(document)
image_items = extract_images(document)

for item in text_items:
    item["embedding"] = embed_text(item["content"])
for item in image_items:
    item["embedding"] = embed_image_via_vlm_caption(item["content"])

unified_store = text_items + image_items   # both modalities, same store

def retrieve_across_modalities(query_embedding: list[float], store: list[dict], top_k: int = 2) -> list[dict]:
    def score(item):
        return sum(a * b for a, b in zip(item["embedding"], query_embedding))
    return sorted(store, key=score, reverse=True)[:top_k]

query_embedding = embed_text("what was our revenue trend?")
results = retrieve_across_modalities(query_embedding, unified_store)
for r in results:
    print(f"[{r['type']}] page {r['page']}: {r['content']}")
print("\n-> Both a text chunk AND an image can surface for the same")
print("   query -- both get passed to a VLM (not text-only LLM) for generation")