How to Choose a RAG Chunking Strategy and Test It

Most advice on a RAG chunking strategy is folklore with a number attached: "use 512", "1000 with 200 overlap", "whatever the tutorial used". By the end of this guide you'll have one Python script that chunks your documents three ways, scores each setup on your own questions, and prints a retrieval hit rate table. Hit rate at k is the share of questions where at least one of the top k retrieved chunks contains the answer.

You need Python 3.10 or newer and Markdown files in docs/. Convert PDFs to Markdown first, or the heading-aware chunker has nothing to split on.

1. Install the splitters and a local embedding model

pip install -qU langchain-text-splitters sentence-transformers numpy

Embeddings run locally, so the experiment is free and repeatable. Swap in your production embedding model later.

2. Write a gold set before you pick a RAG chunking strategy

A gold set is a list of questions, each paired with a short phrase from your docs that answers it. Save one JSON object per line in gold.jsonl:

{"q": "How long do refunds take to reach the customer's card?", "evidence": "within 5 to 7 business days"}
{"q": "Which role can rotate the API signing key?", "evidence": "only the Security Admin role can rotate"}

Three rules keep the test honest:

• Write at least 30 questions, in the words users actually use (support tickets and search logs are the best source), not in the wording of the document.
• Keep each evidence phrase short and unique across the corpus.
• Write the set before you look at any chunks, or you'll unconsciously write questions your favorite chunker passes.

3. Chunk the same documents three ways

We compare three chunkers: fixed-size slices that ignore sentences; LangChain's recursive text splitter, which tries paragraph breaks first, then line breaks, then spaces, then single characters; and a heading-aware chunker that splits at Markdown headings with MarkdownHeaderTextSplitter, then caps each section with the recursive splitter.

Put this at the top of chunk_eval.py:

import json
from pathlib import Path

import numpy as np
from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter
from sentence_transformers import SentenceTransformer

DOCS = {p.name: p.read_text(encoding="utf-8") for p in Path("docs").glob("*.md")}
GOLD = [json.loads(line) for line in Path("gold.jsonl").read_text(encoding="utf-8").splitlines() if line.strip()]


def fixed(text, size, overlap):
    step = size - overlap
    return [text[i:i + size] for i in range(0, max(len(text) - overlap, 1), step)]


def recursive(text, size, overlap):
    splitter = RecursiveCharacterTextSplitter(chunk_size=size, chunk_overlap=overlap)
    return splitter.split_text(text)


def heading_aware(text, size, overlap):
    headers = [("#", "h1"), ("##", "h2"), ("###", "h3")]
    sections = MarkdownHeaderTextSplitter(headers_to_split_on=headers).split_text(text)
    splitter = RecursiveCharacterTextSplitter(chunk_size=size, chunk_overlap=overlap)
    chunks = []
    for doc in splitter.split_documents(sections):
        path = " > ".join(doc.metadata[k] for k in ("h1", "h2", "h3") if k in doc.metadata)
        chunks.append(path + "\n" + doc.page_content)
    return chunks

Sizes here are characters, because the splitter counts with len, not tokens. The heading-aware version prepends the heading path to every chunk, so a fragment from the middle of a long section still knows where it came from.

4. Measure retrieval hit rate at k

Embed every chunk and every question, rank chunks by cosine similarity, and count a hit when the evidence phrase appears in any of the top k:

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
print("embedder max_seq_length:", model.max_seq_length)


def embed(texts):
    return model.encode(texts, normalize_embeddings=True, batch_size=32)


def norm(s):
    return " ".join(s.split())


def evaluate(chunker, size, overlap, ks=(1, 2, 4)):
    chunks = [c for text in DOCS.values() for c in chunker(text, size, overlap)]
    ranked = np.argsort(-(embed([g["q"] for g in GOLD]) @ embed(chunks).T), axis=1)
    hits = dict.fromkeys(ks, 0)
    for g, order in zip(GOLD, ranked):
        evidence = norm(g["evidence"])
        for k in ks:
            if any(evidence in norm(chunks[i]) for i in order[:k]):
                hits[k] += 1
    return len(chunks), {k: hits[k] / len(GOLD) for k in ks}

With normalized vectors, the dot product is cosine similarity. Whitespace is normalized before matching because splitters trim and rejoin it. An answer cut in half by a chunk boundary counts as a miss, which is exactly the failure you're testing for.

The max_seq_length print matters. BGE-small reads 512 tokens, and the sentence-transformers reference says longer inputs are truncated. As a rough rule English runs about 4 characters per token, so 1600-character chunks fit and 4000-character chunks lose their tails.

5. Sweep chunk size and chunk overlap

Run every strategy at three chunk sizes, with zero overlap and with 10 percent overlap:

STRATEGIES = {"fixed": fixed, "recursive": recursive, "heading": heading_aware}

print(f"{'strategy':<10}{'size':>6}{'overlap':>8}{'chunks':>8}{'hit@1':>8}{'hit@2':>8}{'hit@4':>8}")
for name, chunker in STRATEGIES.items():
    for size in (400, 800, 1600):
        for overlap in (0, size // 10):
            n, hit = evaluate(chunker, size, overlap)
            print(f"{name:<10}{size:>6}{overlap:>8}{n:>8}{hit[1]:>8.2f}{hit[2]:>8.2f}{hit[4]:>8.2f}")

Run it with python chunk_eval.py. It re-embeds the corpus for every setting, so try a slice of your docs first if it's slow.

6. Read the table and choose

• Compare at equal context. Big chunks win hit rate by sheer volume of text. Hit@4 at 400 characters, hit@2 at 800 and hit@1 at 1600 all hand the model about 1,600 characters, so compare those three cells against each other.
• Ignore small gaps. With 30 questions, one question is 3.3 points.
• Keep overlap only if it pays. If 10 percent overlap doesn't beat zero, drop it and store fewer chunks.
• Read the gap between hit@1 and hit@4. High at 4 but low at 1 is a reranking problem, not a chunking one.
• Pick the best score at the k you will actually serve, then the cheapest setting within the noise.

Gotchas

• LLM-generated questions are fine to bootstrap a gold set, but review them by hand: a question generated from a chunk echoes its wording and flatters whichever chunker produced it.
• Re-run the script when the embedding model, document format or text extraction step changes.

Chunking decides what your model gets to read, so a hit rate table you can re-run after every document or embedding change beats a tuning argument in production.