Chunking strategies for RAG: how to split documents

Chunking is the step in a RAG pipeline that gets the least attention and causes the most bad answers. Before you can embed and retrieve your documents, you have to split them into pieces, and the chunking strategy you pick decides what a single search hit can contain. Split badly and the right answer is spread across three chunks, none of which ranks first. This post compares the main chunking strategies for RAG and how I'd choose between them.

Why chunk size matters

A chunk is the unit of retrieval. When a user asks a question, you embed it and fetch the chunks whose vectors are closest. That creates a tension:

• Small chunks are specific. Their embedding represents one idea, so they match precise questions well. But a small chunk may lack the context needed to answer ("it supports up to 10 users" without saying what "it" is).
• Large chunks carry context, but their embedding is an average of many ideas, so they match everything a little and nothing strongly. They also eat your context window, which limits how many you can pass to the model.

There is no universal right size. A few hundred words is a reasonable starting point for prose documentation, and you tune from there with real questions. If you haven't built the rest of the pipeline yet, start with RAG in Node.js and come back to this once retrieval is running.

Strategy 1: fixed size with overlap

The simplest approach: cut the text every N characters or tokens, and repeat the last M characters at the start of the next chunk so a sentence on the boundary appears whole in at least one of them.

It is fast, predictable and easy to reason about. It is also blind to structure: it will cut a table in half, split a code sample, or separate a heading from the paragraph it introduces. For unstructured text (transcripts, chat logs, plain notes) it works fine. For documentation it is a baseline to beat.

Strategy 2: structure-aware splitting

Most real documents already tell you where the boundaries are. Markdown has headings, HTML has sections, legal documents have numbered clauses, code has functions. Structure-aware chunking splits on those boundaries first and only falls back to size limits when a section is too long.

This is my default for documentation and knowledge bases. A practical version:

1. Split on the highest-level headings.
2. If a section is still over the size limit, split on paragraphs.
3. If a single paragraph is still too big, split on sentences.
4. Prepend the heading path ("Billing > Refunds > Partial refunds") to each chunk's text before embedding.

That last step is cheap and helps a lot. A chunk that says "Refunds are processed within 5 business days" retrieves much better when its embedded text also contains the words "Billing" and "Refunds".

Here is a compact paragraph-aware chunker in TypeScript. It packs paragraphs up to a size limit and carries an overlap tail into the next chunk.

export function chunkText(
  text: string,
  maxChars = 1500,
  overlapChars = 200
): string[] {
  const paragraphs = text
    .split(/\n\s*\n/)
    .map((p) => p.trim())
    .filter(Boolean);

  const chunks: string[] = [];
  let current = "";

  for (const p of paragraphs) {
    const candidate = current ? `${current}\n\n${p}` : p;
    if (current && candidate.length > maxChars) {
      chunks.push(current);
      const tail = current.slice(-overlapChars);
      current = `${tail.slice(tail.indexOf(" ") + 1)}\n\n${p}`;
    } else {
      current = candidate;
    }
  }
  if (current) chunks.push(current);
  return chunks;
}

Two deliberate limits: a single paragraph longer than maxChars stays one oversized chunk (add a sentence-level split if your content has those), and size is measured in characters. Characters are a fine proxy for tokens while you tune; switch to your provider's tokenizer when you need exact budgets.

Strategy 3: semantic chunking

Semantic chunking uses embeddings to find the boundaries. You embed each sentence, compare neighbouring sentences, and start a new chunk where similarity drops sharply, on the theory that the topic changed there.

It can produce cleaner chunks for long unstructured text where headings don't exist. The costs are real, though: you embed every sentence at ingestion time, the thresholds need tuning per corpus, and the results are harder to debug. When a user complains about a bad answer, "the similarity dropped below 0.72 between sentences 14 and 15" is a hard thing to reason about.

I'd only reach for semantic chunking after structure-aware splitting has been measured and found lacking on a specific kind of content.

Details that matter more than the strategy

Whichever strategy you pick, these habits usually move retrieval quality more than switching between them:

• Keep tables and code blocks whole. Detect them and treat each as an atomic unit, even if it is over the size limit.
• Store metadata with every chunk. Source URL, heading path, document date, tenant id. You need it for citations, filtering and debugging.
• Keep a pointer to neighbours. Store each chunk's position in the document. At query time you can fetch the chunk before and after a strong hit and give the model more context without making every chunk bigger.
• Clean before you split. Navigation menus, cookie banners, repeated footers and page numbers from PDFs pollute embeddings. Strip them first.
• Version your chunker. When you change chunking logic, every chunk changes. Record the chunker version so you can re-ingest cleanly and compare old against new.

How to tune it

Chunking is an empirical problem, so treat it like one. Collect 30 to 50 real questions, and for each one note which document section actually contains the answer. Then for each chunking configuration, run retrieval and check whether that section appears in the top results. That single number (how often the right chunk is in the top five) tells you more than any amount of reading about strategies.

Change one variable at a time: size, overlap, then strategy. Re-run the same questions each time. The approach is the same one I'd use for any LLM feature, covered in more depth in evaluating an LLM feature before you ship it.

If exact terms like product codes or error messages keep getting missed regardless of chunking, the problem is probably vector search itself. Hybrid search adds keyword matching to cover those cases.

Where I'd start

Use structure-aware splitting with a size limit, a small overlap and the heading path prepended to each chunk. Keep tables and code whole, store rich metadata, and measure retrieval hits against real questions before trying anything fancier. Semantic chunking is a tool for specific content, and most corpora never need it.