---
title: Chunking
description: Page on Vedang Vatsa's site: https://veda.ng/glossary/chunking
canonical: https://veda.ng/glossary/chunking
last_updated: 2026-10-03
type: text/markdown
---
# Chunking

Source: https://veda.ng/glossary/chunking
Author: Vedang Vatsa (https://veda.ng/about)

Chunking splits documents into smaller segments for embedding and retrieval in RAG systems. The split is a design choice that changes both retrieval quality and generation accuracy.

Smaller chunks are more precise: you retrieve the relevant sentence, not a page of extras. They can also lack the context needed to understand that sentence. Larger chunks keep context and can dilute scores or confuse the generator with off-topic sentences. Fixed-size chunking cuts at character or token counts, for example 500 tokens with 50-token overlap. It is simple and can split a sentence mid-thought. Sentence-based chunking respects linguistic boundaries and yields uneven lengths. Semantic chunking uses embeddings to find topic shifts and keep a coherent idea together. Hierarchical chunking stores several granularities: paragraphs for broad retrieval, sentences for precise hits. Structure-aware chunking follows headings, sections, and formatting so tables and lists stay intact.

Overlap keeps facts that straddle a boundary retrievable. The right split depends on document type, query patterns, and the embedding model. Dense manuals often need smaller chunks. Narrative text often wants larger ones. Bad chunks cause retrieval misses that generation cannot fix.

A 50-token overlap is a simple insurance policy when a definition straddles a 500-token cut. Semantic chunking recomputes embeddings and splits where similarity drops, which keeps a procedure together. Hierarchical indexes let a first query hit a section, then a second hit a sentence. Tables chopped through the middle become unreadable to both the embedder and the generator. Technical manuals with dense facts often want small chunks. Stories with setup-and-payoff often want larger ones. Retrieval bugs that start here look like model failures later. RAG systems split documents into chunks before embedding. Chunk size is a retrieval hyperparameter: too big and the hit is noisy, too small and context dies.

Glossary index: https://veda.ng/glossary