Fixed-Size Chunking

Fixed-size chunking is the simplest of the five strategies: split the text into uniform segments based on a pre-defined number of characters, words, or tokens — say, every 500 tokens.

Because a hard, direct split at exactly N tokens can disrupt the semantic flow (cutting a sentence, or even a word, right in half), it's standard practice to maintain some overlap between two consecutive chunks — for example, the last 50 tokens of chunk 1 are repeated as the first 50 tokens of chunk 2. This overlap doesn't eliminate the boundary problem, but it substantially reduces how often an important idea gets fully severed between two chunks with no surrounding context in either one.

The appeal of fixed-size chunking is its simplicity: it's trivial to implement, requires no embeddings or LLM calls to determine chunk boundaries, and since every chunk is (approximately) the same size, it simplifies batch processing and gives predictable memory/compute requirements downstream.

The trade-off is exactly what you'd expect from an approach that ignores meaning entirely: it usually breaks sentences or ideas right in the middle, scattering a single coherent piece of information across two separate chunks. When that happens, a similarity search against either chunk alone may retrieve only half the relevant context — which is the core motivation for the more meaning-aware strategies (semantic, recursive, document-structure-based, LLM-based) covered elsewhere in this topic.

def fixed_size_chunks(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
    """Split text into overlapping fixed-size chunks (character-based here;
    swap `len(text)` / slicing for a tokenizer's token counts in production)."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start = end - overlap  # step back by `overlap` before the next chunk
    return chunks

doc = "..."  # a long document
chunks = fixed_size_chunks(doc, chunk_size=500, overlap=50)
print(f"{len(chunks)} chunks, each ~500 chars with 50-char overlap")