5 Chunking Strategies for RAG
Fixed-size, Semantic, Recursive, Document-structure-based, and LLM-based chunking — five distinct ways to split documents for RAG, each with different tradeoffs.
Model Context Protocol creates a unified standard for AI models to query external databases. When combined with local fi
s. When combined with local filesystems, this allows developers to configure custom tools instantly.
instantly.
5 Chunking Strategies for RAG
How you split source documents directly determines retrieval quality — a chunk boundary that cuts through meaning is a chunk that will retrieve poorly.
| How it splits | Best for | |
|---|---|---|
| Fixed-size | Equal-length blocks with a small overlap between adjacent chunks | Simple, predictable pipelines; not sensitive to cutting mid-sentence |
| Semantic | Boundaries placed exactly where cosine-similarity between sentences drops sharply (topic shifts) | Content where topical coherence matters more than uniform size |
| Recursive | Two-level cut — paragraph breaks first, then a second cut inside any paragraph still too large | General-purpose documents with mixed structure |
| Document-structure | Splits strictly along the document's own heading/section markers | Well-formatted docs (manuals, reports) where structure = meaning |
| LLM-based | An LLM judges where a 'semantically isolated and meaningful' unit ends | Highest-quality boundaries when the extra LLM call cost is acceptable |
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Implement and compare fixed-size, semantic, recursive, document-structure-based, and LLM-based chunking
- •Explain why chunk overlap matters for fixed-size chunking
- •Choose the right chunking strategy given content type, embedding model, and compute budget
- •Identify the specific failure mode each chunking strategy is prone to
What is it?
Chunking is the first step of any RAG pipeline: splitting a large document into smaller, manageable pieces that fit within an embedding model's input size. There are 5 distinct chunking strategies, each trading off implementation simplicity, computational cost, and how well chunk boundaries preserve semantic meaning: Fixed-size chunking (uniform character/word/token splits), Semantic chunking (cosine-similarity-based grouping of meaningful units), Recursive chunking (separator-based splitting with a size-limit fallback), Document-structure-based chunking (using headings/sections as boundaries), and LLM-based chunking (prompting an LLM to generate semantically isolated chunks directly).
Why it exists
Chunking exists because embedding models have a limited input size, and a large document — a 50-page PDF, a long transcript — cannot be embedded as a single vector without losing most of its specificity. But how you split matters enormously: split badly, and you sever ideas mid-sentence, scattering the information needed to answer a question across multiple, individually-incomplete chunks. Different chunking strategies exist because there's no single approach that's both cheap and always semantically faithful — each strategy makes a different tradeoff between implementation simplicity, computational cost, and boundary quality.
Problem it solves
Chunking solves the fundamental RAG problem of fitting large documents into an embedding model's context window while keeping each resulting chunk retrievable and meaningful. Poor chunking directly causes poor retrieval: if a sentence or idea is split across two chunks by an arbitrary character-count cutoff, neither chunk alone may contain enough context to be a good semantic match for a relevant query, and the answer effectively becomes unreachable even though the source document technically contains it.
Intuition
Think of chunking like deciding how to cut a cake for a group photo of the pieces. Fixed-size chunking is cutting exactly equal wedges regardless of where the decorations are — fast and uniform, but you might slice right through one. Semantic chunking is cutting along the natural decoration boundaries — slower to figure out, but every piece looks complete. Recursive chunking cuts along the cake's pre-scored layer lines first, then further divides any layer that's still too big. Document-structure-based chunking uses the cake's actual tiers as natural boundaries. LLM-based chunking is like asking a professional cake decorator to look at the whole cake and personally decide the most sensible way to divide it — the most thoughtful cut, but also the slowest and most expensive.
Analogy
Fixed-size chunking is like highlighting a book every 200 words regardless of sentence boundaries — fast but might cut off mid-thought. Semantic chunking is like a careful reader who highlights complete ideas, stopping exactly when the topic shifts. Recursive chunking is like first splitting a book by chapter, then splitting any chapter that's still too long by section. Document-structure chunking is like using the book's own table of contents as your highlighting boundaries. LLM-based chunking is like hiring an editor to read the whole book and manually decide the most sensible place to break every excerpt.
Technical explanation
(1) Fixed-size chunking splits text into uniform segments based on a predefined number of characters, words, or tokens, maintaining some overlap between consecutive chunks (since a direct split can disrupt semantic flow). It's simple to implement and, because all chunks are equal size, simplifies batch processing — but it usually breaks sentences or ideas mid-stream, scattering important information across chunk boundaries.
(2) Semantic chunking segments the document into meaningful units (sentences, paragraphs, thematic sections), embeds each, and merges consecutive segments into the same chunk as long as their embeddings have high cosine similarity — starting a new chunk the moment similarity drops significantly. This maintains the natural flow of language and preserves complete ideas, producing richer chunks that improve retrieval accuracy and downstream answer coherence; its main drawback is dependence on a similarity-drop threshold that can vary from document to document.
(3) Recursive chunking first splits based on inherent separators (paragraphs, sections), then further splits any resulting chunk that still exceeds a predefined size limit — chunks that already fit the limit aren't split further. Like semantic chunking, this preserves natural language flow and complete ideas, at the cost of extra implementation and computational overhead versus fixed-size.
(4) Document-structure-based chunking uses a document's inherent structure — headings, sections, paragraphs — to define chunk boundaries, maintaining structural integrity by aligning with the document's own logical sections. It assumes the document actually has a clear structure (which may not hold), and chunks may vary in length, potentially exceeding model token limits — often mitigated by merging this approach with recursive splitting.
(5) LLM-based chunking prompts an LLM directly to generate semantically isolated and meaningful chunks, achieving the highest semantic accuracy since the LLM understands context and meaning beyond the simple heuristics the other four approaches rely on — but it is the most computationally demanding of all five techniques, and needs care around the LLM's own limited context window when processing very long documents.
Architecture
A chunking pipeline sits at the very front of RAG indexing: [Raw Document] → [Chunker (one of the 5 strategies)] → [Chunks] → [Embedding Model] → [Vector Store]. The choice of chunker doesn't change anything downstream structurally, but it dramatically affects the quality of what flows into the embedding step — garbage chunk boundaries produce garbage embeddings regardless of how good the embedding model itself is.
Workflow
- Assess your content type: highly structured documents (docs with clear headings) favor document-structure-based chunking; unstructured prose favors semantic or recursive chunking; mixed or unpredictable content may need LLM-based chunking for best quality.
- Start with the simplest viable option — fixed-size chunking with 10-15% overlap — as a fast baseline to validate the rest of your RAG pipeline before over-investing in chunking sophistication.
- If retrieval quality is poor and you suspect chunk boundaries are the cause (ideas split across chunks), upgrade to semantic or recursive chunking.
- For structured content (technical docs, legal contracts with numbered sections), try document-structure-based chunking, falling back to recursive splitting for any oversized section.
- Reserve LLM-based chunking for cases where retrieval quality is critical and the extra compute cost is justified — it's the most accurate but also the most expensive per document.
- Test empirically: semantic chunking works well in many cases, but the right choice ultimately depends on your specific content, embedding model, and compute budget — always validate against your own retrieval metrics rather than assuming one strategy is universally best.
Example
import re
1) Fixed-size chunking with overlap
def fixed_size_chunk(text: str, size: int = 400, overlap: int = 50) -> list[str]: words = text.split() chunks, i = [], 0 while i < len(words): chunks.append(' '.join(words[i:i + size])) i += size - overlap return chunks
2) Semantic chunking via cosine-similarity threshold
def semantic_chunk(sentences: list[str], embed_fn, threshold: float = 0.75) -> list[str]: embeddings = [embed_fn(s) for s in sentences] chunks, current = [], [sentences[0]] for i in range(1, len(sentences)): sim = cosine_similarity(embeddings[i - 1], embeddings[i]) if sim >= threshold: current.append(sentences[i]) else: chunks.append(' '.join(current)) current = [sentences[i]] chunks.append(' '.join(current)) return chunks
3) Recursive chunking: split by separators, then by size limit
def recursive_chunk(text: str, max_size: int = 500) -> list[str]: paragraphs = text.split('\n\n') result = [] for p in paragraphs: if len(p) <= max_size: result.append(p) else: result.extend(fixed_size_chunk(p, size=max_size, overlap=0)) return result
4) Document-structure-based chunking via markdown headings
def structure_chunk(markdown_text: str) -> list[str]: return re.split(r'\n(?=#{1,3} )', markdown_text)
5) LLM-based chunking
def llm_chunk(text: str, llm_complete) -> list[str]: prompt = f'Split this text into semantically isolated, meaningful chunks. Return each chunk separated by ---.\n\n{text}' return llm_complete(prompt).split('---')
Real-world usage
LangChain's RecursiveCharacterTextSplitter is the most widely used off-the-shelf implementation of recursive chunking, defaulting to splitting on paragraph breaks, then sentences, then words, as needed to hit a target chunk size. LlamaIndex's SemanticSplitterNodeParser implements semantic chunking using embedding-based similarity thresholds. Technical documentation platforms (Notion AI, GitBook-powered docs search) commonly use document-structure-based chunking since their content is already heavily headinged/sectioned, making structural boundaries a natural and cheap choice. Companies processing highly variable, unstructured input (legal discovery documents, scanned/OCR'd documents with unreliable formatting) increasingly turn to LLM-based chunking despite the cost, since heuristic-based methods perform poorly on genuinely unstructured or low-quality source text.
Trade-offs
The 5 strategies form a rough simplicity-vs-quality spectrum: fixed-size is cheapest and simplest but produces the worst boundaries; semantic and recursive chunking cost more compute (an embedding pass or separator-parsing logic) for meaningfully better boundaries; document-structure-based chunking is cheap when the document genuinely has clean structure but fails silently when it doesn't; LLM-based chunking produces the best boundaries but is the most expensive and slowest, potentially prohibitively so for large corpora. There's no universally best choice — the right strategy depends on content type, available compute, and how much retrieval quality actually matters for the specific application.
Visual explanation
Five side-by-side diagrams of the same source paragraph being chunked differently.
Fixed-size: a ruler cutting the text into equal-length blocks with a small overlapping band between adjacent blocks, with one cut visibly falling mid-sentence.
Semantic: the same text divided into blocks whose boundaries fall exactly at topic-shift points, based on a cosine-similarity curve dropping sharply at each boundary.
Recursive: a two-level cut — first at paragraph breaks (level 1), then, for any paragraph exceeding the size limit, a second cut inside it (level 2).
Document-structure: the text divided strictly along heading/section markers pulled from the document's own formatting.
LLM-based: the text with boundary markers placed by an LLM's own judgment of where a 'semantically isolated and meaningful' unit ends.
Advantages
- —
Provides 5 distinct, well-understood tradeoff points to choose from rather than a single one-size-fits-all approach
- —
Semantic and recursive chunking meaningfully improve retrieval quality over naive fixed-size splitting by preserving complete ideas
- —
Document-structure-based chunking is nearly free when a document already has clean formatting to exploit
- —
LLM-based chunking provides a ceiling on achievable chunk quality when retrieval accuracy is critical enough to justify the cost
Disadvantages
- —
Fixed-size chunking reliably breaks sentences and ideas mid-stream, scattering information across chunk boundaries
- —
Semantic chunking's similarity-drop threshold is document-dependent and requires tuning
- —
Document-structure-based chunking assumes clean document structure that may not actually exist, and produces variable chunk lengths that can exceed model limits
- —
LLM-based chunking is the most computationally expensive of all 5 approaches and is constrained by the chunking LLM's own context window on very long documents
Common mistakes
- —
Defaulting to fixed-size chunking for all content types without testing whether a smarter strategy would meaningfully improve retrieval for that specific corpus
- —
Not including any overlap in fixed-size chunking, making semantic breaks even more damaging by ensuring zero context bridges across the cut
- —
Assuming document-structure-based chunking will work without verifying the source documents actually have consistent, parseable structure
- —
Using LLM-based chunking on very large documents without accounting for the chunking LLM's own context window limits
- —
Picking one chunking strategy once and never re-evaluating it empirically against actual retrieval quality metrics as the corpus or use case evolves
📂 Subtopics
Fixed-Size Chunking
Split text into uniform segments based on a pre-defined number of characters, words, or tokens, with overlap between consecutive chunks to reduce information loss at the boundaries.
~10 min
Semantic Chunking
Group sentences into a chunk as long as consecutive segments stay semantically similar (via cosine similarity of their embeddings), starting a new chunk exactly where the similarity drops significantly.
~15 min
Recursive Chunking
Chunk first using natural separators like paragraphs or sections, then recursively split any chunk that still exceeds the size limit — preserving natural structure while still respecting a hard size cap.
~10 min
Document Structure-Based Chunking
Use a document's own structure — headings, sections, paragraphs — to define chunk boundaries directly, maintaining structural integrity with the document's logical organization.
~10 min
LLM-Based Chunking
Prompt an LLM to directly generate semantically isolated and meaningful chunks. Highest semantic accuracy of all five strategies, but also the most computationally demanding.
~10 min