RAG
Chunking
Chunking divides source material into units that can be indexed and retrieved.
How it works
Chunk boundaries control how much context stays together and how precisely retrieval can target it. Overlap can preserve cross-boundary meaning at extra storage and retrieval cost.
A concrete example
Illustrative: split 10,000 tokens into 20 chunks of 512 tokens with 12-token overlap adjustments.
Do not confuse it with Tokenization
tokenization creates model units; chunking groups content for retrieval.