intermediate~2h

ETL Pipeline for RAG Ingestion

Module 09 assumed documents were already in the vector store. This module is how they get there — from a raw PDF or JSON file to searchable, chunked, metadata-enriched vectors.

Learning objectives

Beginner: Explain the three stages of a RAG ingestion ETL pipeline (extract, transform, load).
Intermediate: Configure a document reader and chunking transformer to prepare a raw PDF for embedding.
Advanced: Design a re-ingestion strategy for updated source documents that avoids duplicating stale chunks in the vector store.

Spring AI models document ingestion as a classic Extract-Transform-Load pipeline, built from three composable interfaces:

Stage	Interface	Job
Extract	DocumentReader	Reads a raw source (PDF, JSON, Markdown, a database) into a list of Document objects.
Transform	DocumentTransformer	Chunks (splits), enriches with metadata, or otherwise modifies Documents before storage.
Load	DocumentWriter	Writes the final Documents (as embeddings) into a VectorStore.

A Document is the common currency across all three stages: text content plus a metadata map — the exact same type Module 08's VectorStore.add() already consumes.

// PDF
DocumentReader pdfReader = new PagePdfDocumentReader("classpath:/docs/handbook.pdf");

// JSON — extract specific fields into the Document's content
DocumentReader jsonReader = new JsonReader(new ClassPathResource("faqs.json"), "question", "answer");

// Markdown
DocumentReader mdReader = new MarkdownDocumentReader("classpath:/docs/README.md");

List<Document> documents = pdfReader.get();

💻 Code example

// PDF
DocumentReader pdfReader = new PagePdfDocumentReader("classpath:/docs/handbook.pdf");

// JSON — extract specific fields into the Document's content
DocumentReader jsonReader = new JsonReader(new ClassPathResource("faqs.json"), "question", "answer");

// Markdown
DocumentReader mdReader = new MarkdownDocumentReader("classpath:/docs/README.md");

List<Document> documents = pdfReader.get();

TokenTextSplitter splitter = new TokenTextSplitter(800, 100, 5, 10000, true);
// (chunkSize, minChunkSizeChars, minChunkLengthToEmbed, maxNumChunks, keepSeparator)

List<Document> chunks = splitter.apply(documents);

// enrich every chunk with a consistent source tag before it's embedded
chunks.forEach(doc -> doc.getMetadata().put("source", "employee-handbook-v3"));

vectorStore.add(chunks);

◆ Under the hood — why enrich metadata at ingestion time, not query time

Metadata attached during ETL (source document, version, department, publish date) becomes exactly the field set Module 08 §5's FilterExpression can query against later. Getting this right at ingestion is much cheaper than trying to reconstruct "which document did this chunk come from, and when was it published" after the fact from chunked, decontextualized text alone.

▲ Pitfall

Re-running an ETL pipeline against an updated source document without a strategy for removing/replacing the old chunks leaves stale vectors in the store indefinitely — RAG will happily retrieve and ground answers in outdated content sitting right alongside the new version, with no built-in signal that it's stale. Version or tag chunks by source revision, and delete superseded chunks explicitly as part of your ingestion pipeline.

💻 Code example

TokenTextSplitter splitter = new TokenTextSplitter(800, 100, 5, 10000, true);
// (chunkSize, minChunkSizeChars, minChunkLengthToEmbed, maxNumChunks, keepSeparator)

List<Document> chunks = splitter.apply(documents);

// enrich every chunk with a consistent source tag before it's embedded
chunks.forEach(doc -> doc.getMetadata().put("source", "employee-handbook-v3"));

vectorStore.add(chunks);

vectorStore.add(chunks) is itself the "Load" stage — the VectorStore interface from Module 08 doubles as the ETL pipeline's DocumentWriter, so there's no separate writer type to learn once you already know the store's API.

✓ Quick recap

What common type flows through every stage of the ETL pipeline? Document — text content plus a metadata map, produced by readers, modified by transformers, and consumed by the vector store on load. Why is metadata enrichment done at ingestion time rather than query time? Chunked, decontextualized text is hard to re-attribute later; attaching source/version/date during ETL makes it directly queryable via metadata filters afterward.

Want a visual for this concept?

Generate a diagram tailored to “ETL Pipeline for RAG Ingestion” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.

Practice quiz

Next Step

Continue to Tool Calling & Function Calling →← Back to all Spring AI chapters