Fixed-Size Chunking
~10 min read
Split text into uniform segments based on a pre-defined number of characters, words, or tokens, with overlap between consecutive chunks to reduce information loss at the boundaries.
Fixed-Size Chunking is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Splits text into uniform segments of a fixed size (characters, words, or tokens)
- •Overlap between consecutive chunks (e.g. 50 tokens repeated) softens — but doesn't eliminate — the boundary-cutting problem
- •Simplest strategy to implement, with no embeddings or LLM calls needed to determine boundaries
- •Uniform chunk sizes simplify batch processing and give predictable downstream compute/memory needs
- •Main weakness: routinely breaks sentences and ideas mid-thought, scattering one concept across two chunks