Semantic Chunking

Semantic chunking segments a document based on meaningful units — sentences, paragraphs, or thematic sections — rather than a fixed character count, and uses embedding similarity to decide where chunk boundaries actually belong.

The process: first, create an embedding for each segment (typically sentence-level). Start with the first segment and its embedding. If that segment's embedding has high cosine similarity with the next segment's embedding, the two segments belong in the same chunk — they're talking about the same thing. This continues, absorbing more segments into the current chunk, until the cosine similarity between consecutive segments drops significantly. The moment it does, that's the signal a topic shift has happened — a new chunk starts there, and the process repeats.

Unlike fixed-size chunks, this maintains the natural flow of language and preserves complete ideas — a chunk boundary lands where the topic actually changes, not wherever the character count happened to hit a round number. Because each resulting chunk is semantically richer and more coherent, this generally improves retrieval accuracy: a query's embedding is more likely to closely match a chunk that's entirely about one coherent topic than a chunk that's an arbitrary half-and-half mix of two different ideas — which in turn produces more coherent, relevant responses from the LLM.

The one real complication: the similarity-drop detection depends on a threshold that determines how much of a cosine-similarity drop counts as 'significant enough' to start a new chunk. That threshold isn't universal — it can vary meaningfully from document to document (a technical manual and a narrative essay have very different natural 'topic drift' patterns), so it typically needs to be tuned rather than hardcoded once and reused everywhere.

import numpy as np

def cosine_sim(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

def semantic_chunks(
    sentences: list[str], embeddings: list[np.ndarray], threshold: float = 0.75,
) -> list[str]:
    """Group consecutive sentences into a chunk while similarity stays high;
    start a new chunk the moment similarity drops below `threshold`."""
    chunks, current = [], [sentences[0]]
    for i in range(1, len(sentences)):
        sim = cosine_sim(embeddings[i - 1], embeddings[i])
        if sim >= threshold:
            current.append(sentences[i])       # same topic — extend the chunk
        else:
            chunks.append(" ".join(current))   # topic shift — close the chunk
            current = [sentences[i]]
    chunks.append(" ".join(current))
    return chunks

# sentences + embeddings computed upstream via your embedding model
# (e.g. OpenAI text-embedding-3-small), one embedding per sentence