LLM-Based Chunking

LLM-based chunking takes a fundamentally different approach from the other four strategies: instead of relying on character counts, embedding similarity, or document structure, it prompts an LLM directly to generate semantically isolated and meaningful chunks from the source text.

This method ensures the highest semantic accuracy of all five techniques discussed, precisely because the LLM can understand context and meaning beyond the simple heuristics the other four approaches rely on (character counts, similarity thresholds, or structural markers). An LLM reading a document can recognize that two sentences separated by a paragraph break are actually part of the same argument, or that a single long paragraph actually contains two genuinely distinct ideas — distinctions that fixed-size, semantic-similarity, or structure-based chunking can miss.

The cost of that accuracy is real: this is the most computationally demanding chunking technique of all five, since it requires an actual LLM inference call (or several) per document, rather than a cheap embedding lookup or a string-splitting operation. There's also a practical constraint worth noting — since LLMs themselves have a limited context window, chunking a very large document with this method may itself require first breaking the document into large pieces the LLM can actually process, adding another layer of complexity.

In practice, semantic chunking tends to work well enough for most use cases at a fraction of the cost, which is why LLM-based chunking is typically reserved for scenarios where retrieval accuracy is worth the extra expense — high-stakes domains, or corpora where the other four strategies have been tried and empirically underperform. As with all five strategies, the right choice ultimately depends on testing against your specific content, embedding model, and computational budget.

import json
from openai import OpenAI

client = OpenAI()

def llm_based_chunks(text: str) -> list[str]:
    prompt = (
        "Split the following text into semantically isolated and "
        "meaningful chunks. Each chunk should represent one complete, "
        "self-contained idea. Return a JSON object with a 'chunks' key "
        "containing a list of chunk strings.\n\n"
        f"TEXT:\n{text}"
    )
    resp = client.chat.completions.create(
        model="gpt-4.1",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)["chunks"]

# For documents longer than the LLM's context window, first split
# into large sections (e.g. via document_structure_chunks) and run
# llm_based_chunks() on each section separately.