Token and Chunking in RAG — Complete Guide for Developers 2026

Tokenization and chunking are the quiet foundation of every good RAG pipeline. You can have the perfect vector database and the best model, but if documents are split badly, retrieval quality falls apart. This guide explains what tokenization and chunking actually are, why the way you split documents changes the answers your chatbot gives, and how to choose a strategy for your content.

Tokenization: how text becomes numbers

Before any model can process text, the text is broken into tokens — small units that are roughly word fragments. A token might be a whole word, part of a word, or a piece of punctuation. LLMs have a token limit (context window), so every chunk you retrieve and every prompt you build consumes tokens. Understanding token count is how you budget both cost and context space.

A practical rule of thumb is that 100 English tokens are roughly 75 words. Languages differ, but for budgeting purposes this estimate is close enough to plan your chunk sizes and your model spend.

Chunking: how documents become retrievable units

Chunking splits a long document into pieces that get embedded and stored. Each chunk should be large enough to contain a self-contained idea, but small enough to keep the embedding focused. If a chunk is too large, its embedding averages out the meaning and retrieval becomes fuzzy; too small, and context is lost.

Fixed-size chunking

Fixed-size chunking splits text every N tokens with an optional overlap. It's the simplest to implement and predictable to operate, which makes it a fine baseline. The downside is that it ignores document structure — a sentence can be cut in half, and related ideas can be separated.

• Typical sizes: 300–500 tokens with a 50–100 token overlap.
• Best for: homogeneous content like support articles of similar length.
• Watch out for: cutting tables, code blocks, and sentences mid-way.

Semantic chunking

Semantic chunking groups text by meaning. It detects natural boundaries — where the topic shifts — and splits there, so each chunk is coherent. This produces better retrieval quality but is slower to compute because it runs the text through a model to find boundaries.

• Best for: long, varied documents where topic boundaries matter, like research reports and product documentation.
• Trade-off: higher embedding cost and latency during indexing.

Recursive / structure-aware chunking

Recursive chunking tries separators in order — paragraphs, then sentences, then words — so splits land on natural boundaries whenever possible. Structure-aware chunking goes further by respecting headings, tables, and list boundaries, which is what you want for HTML and Markdown documents.

• Best for: web docs, Markdown, and any content with headings and lists.
• Why it wins: chunks stay readable, and a heading can be carried into the chunk as context.

Chunk overlap and metadata

A small overlap between chunks prevents the model from losing context at boundaries. Equally important is attaching metadata to each chunk — the source document, section heading, page number, or URL. When retrieval returns a chunk, that metadata is what lets you show citations and let the reader jump to the original.

Practical tuning steps

Start with fixed-size chunking at around 400 tokens with a 20% overlap, measure answer quality on a fixed set of test questions, then try semantic or structure-aware chunking and compare. Tune one variable at a time and keep a small evaluation set so improvements are measurable rather than vibes.