Pipeline Architecture: Extract, Embed Separately, Retrieve, and Combine
~14 min read
A multimodal RAG pipeline extracts text AND images separately, embeds each with an appropriate model, retrieves across both modalities, and combines everything into one generation step.
Pipeline Architecture: Extract, Embed Separately, Retrieve, and Combine is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Extraction runs two parallel tracks: text (prose paragraphs, standard chunking) and images (full pages or cropped figures/charts/tables)
- •Text gets embedded with an ordinary text embedding model; images get embedded via a CLIP-style image encoder or by captioning-then-embedding with a VLM
- •Both text and image representations typically live in the same (or a coordinated) vector store, tagged with source page and modality metadata
- •Retrieval searches across both modalities together, so a query can surface both a relevant text chunk and a relevant image for the same request
- •Generation must use a VLM, not a text-only LLM, since retrieved images need to be seen directly by the model to reason precisely about their content