Retrieval-Augmented Generation (RAG)

An LLM's knowledge is frozen at training time and limited to what fit in its weights. RAG fixes both problems by retrieving relevant information at query time and feeding it into the prompt as grounding context.

The RAG Pipeline, End to End

Nine steps, in the order they actually run. Everything else on this page is one of these steps covered in real depth — worth seeing the whole shape once before diving into any single piece:

1. Ingestion: pull in your raw source documents — PDFs, wikis, tickets, whatever your knowledge base actually is — and get them into clean, parseable text.
2. Chunking: split that text into retrievable units. The size and strategy chosen here shapes the quality of every step downstream.
3. Embedding: turn each chunk into a vector using an embedding model (see Foundation Model Internals).
4. Indexing: store those vectors in a structure built for fast similarity search at scale (see RAG Optimization for how that index is actually tuned in production, and Databases for the storage layer itself).
5. Query Processing: transform the user's raw question before searching with it — decompose it, rephrase it, or generate a hypothetical answer to search with instead.
6. Retrieval: search the index — dense, sparse, or hybrid — using the (possibly transformed) query, and pull back the chunks most likely to be relevant.
7. Reranking: re-score the retrieved candidates with a more precise, more expensive method before committing to a final set.
8. Generation: feed the reranked chunks into the LLM's context alongside the query (see Prompt Engineering for how that context actually gets structured into a prompt), and let it generate an answer grounded in what it just read.
9. Evaluation & Monitoring: measure whether the whole pipeline actually produces faithful, relevant answers — both before shipping and continuously once it's live.

Click any node below for what it does, then type a real question and watch retrieval actually run — real chunking, real cosine-similarity ranking, a real (if simplified) reranking pass:

Chunking Strategies

Pipeline step 2. Once documents are ingested, this is the first real decision point — every later step operates on whatever units get produced here.

• Fixed-size chunking: split every N tokens — simple, but can cut sentences/ideas in half.
• Semantic chunking: split at natural boundaries (paragraphs, topic shifts) so each chunk is a coherent unit of meaning.
• Recursive chunking: try splitting on large boundaries first (sections), fall back to smaller ones (sentences) only if a chunk is still too large.
• Parent-child chunking: retrieve using small, precise child chunks, but pass the larger parent chunk (with more surrounding context) to the LLM — balances retrieval precision with generation context.
• Late chunking: embed the entire document first with a long-context embedding model, then pool the resulting token embeddings into chunks only afterward — the reverse order of every strategy above. Because each chunk's embedding is derived from token representations that already attended to the full document, it carries whole-document context even for a chunk that, read in isolation, wouldn't obviously connect to it (a pronoun resolved only by an earlier paragraph, for instance). This solves a different problem than parent-child chunking: parent-child pairs a small retrieval unit with a larger separate context chunk passed to the LLM; late chunking bakes document-wide context directly into the retrieval embedding itself, before chunking ever happens.

Recursive chunking in code, via LangChain's RecursiveCharacterTextSplitter — the standard implementation of the "try large boundaries first, fall back to smaller ones" strategy described above:

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Tries splitting on paragraph breaks first, then sentences, then words --
# only falls back to a smaller separator when a chunk is still too large.
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_text(document_text)

chunk_overlap repeats a small amount of text between consecutive chunks, so a sentence or idea split across a chunk boundary still appears in full in at least one of the two chunks.

Choosing an Embedding Model

Pipeline step 3. Whatever chunks step 2 produced now need to become vectors. Different embedding models trade off quality, speed, cost, and domain fit (a model tuned on general web text may underperform on legal or medical documents). Dimensionality matters too — higher-dimensional embeddings usually capture more nuance but cost more to store and search at scale.

In code, via the standard sentence-transformers library (see Retrieval & Reranking Architectures for the bi-encoder architecture this implements):

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# Every chunk from the chunking step, embedded once and indexed --
# this is the vector data that step 4 (indexing) stores.
chunk_embeddings = model.encode(chunks)

What retrieval above is actually doing under the hood — comparing vectors by cosine similarity in a learned space where meaning clusters geometrically — is exactly what the Embedding Space Explorer makes clickable:

Query Transformation

Pipeline step 5. Before searching at all, the raw user question can itself be transformed into something more retrievable:

• HyDE (Hypothetical Document Embeddings): have the LLM first generate a hypothetical answer to the query, then embed and search using that instead of the raw query — often more similar to real relevant documents than the terse original question.
• Query decomposition: break a complex multi-part question into sub-questions, retrieve for each separately.
• Step-back prompting: ask a more general question first to retrieve broader context, before answering the specific one.
• Multi-query retrieval: have the LLM generate several rephrasings of the same query (different word choices, different framings), retrieve for each independently, and union the results — hedges against any single phrasing missing a relevant chunk purely due to vocabulary mismatch.
• RAG-Fusion: multi-query retrieval's natural extension — generate multiple query variations, retrieve for each, then combine the ranked result lists via Reciprocal Rank Fusion (the same technique the Retrieval section below uses for hybrid search) rather than a simple union, so chunks ranked highly across multiple query variations surface to the top.

Whichever version of the query comes out of this step — rephrased, decomposed, or hypothetical — is what actually gets searched next.

Dense vs. Sparse Retrieval, and BM25

Pipeline step 6. Two structurally different ways to find relevant chunks — worth distinguishing precisely rather than lumping together as "search," because they fail in genuinely different, complementary ways.

Dense Retrieval

What is it? Finding chunks by meaning, using the same embedding-space idea the rest of this page already relies on for chunk vectors.

How does it work? Embed both the query and every chunk into the same continuous vector space (see Foundation Model Internals), and retrieve by vector similarity — the same cosine-similarity comparison the Embedding step above computes.

Why is it useful? It captures semantic similarity: a query about "canines" can retrieve a chunk about "dogs" even with zero shared words, because the embedding space clusters them near each other regardless of exact wording.

Limitation: Because it's comparing meaning, not exact text, dense retrieval can miss exact matches that matter — a specific product code, an exact legal term, a precise error message — where the literal string is the point, not the general topic.

Dense retrieval's blind spot — missing exact term matches — is precisely what sparse retrieval, the classical information-retrieval approach, was built to guarantee.

Sparse Retrieval and BM25

What is it? Finding chunks by exact term overlap with the query, the pre-embeddings approach to search that's still highly effective today.

How does it work? Represent text as a high-dimensional, mostly-zero vector over the vocabulary — which exact terms appear, and how often — then score query-document overlap with BM25, the standard modern scoring function. BM25 refines raw term-frequency matching with two corrections: it saturates the benefit of a term appearing many times in a document (the 5th occurrence of a word matters far less than the 1st), and it downweights matches on terms that are common across the whole corpus (matching "the" tells you nothing; matching a rare technical term tells you a lot) while upweighting matches on documents that are otherwise concise (a short document containing your query terms is more likely to be specifically about them than a long document that happens to mention them once).

Why is it useful? It guarantees exact term matches aren't missed — the opposite failure mode from dense retrieval — which is exactly why it still matters even in an embeddings-heavy pipeline.

Limitation: Sparse retrieval has no concept of meaning — a query about "canines" simply won't match a chunk that only ever says "dogs," no matter how relevant that chunk actually is, because there's zero literal term overlap for BM25 to score.

In code, via rank_bm25, the standard Python implementation — it deliberately does no text preprocessing itself, so tokenizing (and lowercasing, stemming, etc. if wanted) is the caller's job:

from rank_bm25 import BM25Okapi

tokenized_corpus = [chunk.lower().split() for chunk in chunks]
bm25 = BM25Okapi(tokenized_corpus)

tokenized_query = query.lower().split()
top_chunks = bm25.get_top_n(tokenized_query, chunks, n=5)

Dense misses exact matches; sparse misses meaning — two failures with no overlap, which is exactly what makes running both together, rather than picking one, worth the extra cost.

Hybrid Search

What is it? Running dense and sparse retrieval independently, then merging their two ranked result sets into one — the practical default in production RAG, not a niche option.

How does it work? Merge the two ranked lists via Reciprocal Rank Fusion, which combines rankings based on each result's position in each list rather than trying to normalize and compare two differently-scaled similarity scores directly.

Why is it useful? It gets both failure-avoidance properties at once — a chunk can surface either because it's semantically close (dense) or because it shares exact terms with the query (sparse), covering each method's blind spot with the other.

Re-ranking with Cross-Encoders

Pipeline step 7. The retrieval step above (dense, sparse, or hybrid) has to be fast enough to search potentially millions of chunks — it uses a bi-encoder architecture, embedding the query and each document independently so document embeddings can be precomputed and indexed once (see Recommender Systems — Two-Tower Architecture for the identical architectural pattern). A cross-encoder re-ranker instead feeds the query and a candidate document into the model together, letting it directly attend across both — far more accurate at judging true relevance, but far too slow to run against the entire corpus, since nothing about it can be precomputed per-document. The standard pattern is exactly this two-stage shape: cheap bi-encoder retrieval narrows millions of chunks to a few dozen candidates, then an expensive cross-encoder re-ranks just those few dozen precisely — the same retrieval-then-rank funnel as Recommender Systems and Learning-to-Rank.

A third architecture sits between these two: late interaction (ColBERT) keeps a separate embedding per token instead of pooling to one vector per document, and scores query-document pairs via MaxSim — for each query token, the single best-matching document token, summed. Document token embeddings stay precomputable and indexable, like a bi-encoder's, but nothing gets collapsed away, so it recovers much of a cross-encoder's precision without the joint forward pass. See Retrieval & Reranking Architectures for the real MaxSim formula, a fully worked numeric example, and a three-way comparison of when production systems reach for each of the three.

Advanced RAG Architectures

Beyond the single retrieve-then-generate pass above, several architectures address specific failure modes by adding a control loop or changing what's actually retrieved:

• Agentic RAG: instead of one fixed retrieve→generate pass, an agent (see Agent Architectures) decides whether to retrieve, what to retrieve, and how many times — it can issue multiple retrieval calls, reformulate its own queries based on what came back, or decide the retrieved context is insufficient and search again, rather than being locked into exactly one retrieval step per query.
• Corrective RAG (CRAG): adds an explicit evaluation step after retrieval — a lightweight grader assesses whether the retrieved chunks are actually relevant/sufficient, and if not, triggers a fallback (query rewriting, a broader search, or falling back to general web search) before generation proceeds, rather than generating from possibly-irrelevant context regardless.
• Self-RAG: trains the LLM itself to emit special reflection tokens during generation — deciding when retrieval is needed, critiquing whether retrieved passages are actually relevant, and assessing whether its own generated output is properly supported by them — folding the "should I retrieve, was this useful, is my answer grounded" judgment directly into the generating model's own token stream instead of a separate pipeline stage.
• Multimodal RAG: retrieves and grounds generation in non-text content — image embeddings (via CLIP-style models, see Computer Vision — Modern Vision & Multimodal) alongside or instead of text chunks, for questions that need to reference charts, diagrams, or photos in the source material.
• SQL RAG: instead of retrieving unstructured text chunks, translates a natural-language question into a SQL query against a structured database (see Databases — Relational), executes it, and feeds the query result to the LLM as grounding context — the right approach when the answer genuinely lives in structured tables (aggregates, exact counts) rather than prose a chunk-based retriever would need to happen to contain.
• Code RAG: retrieval over a codebase specifically — chunking strategies built around code structure (functions, classes) rather than fixed token windows, and embeddings from code-aware models, used for AI coding assistants that need to ground suggestions in a specific repository's actual code rather than the model's general programming knowledge.

GraphRAG

Standard RAG retrieves isolated chunks, which struggles with questions that require connecting facts across multiple documents (multi-hop reasoning). GraphRAG instead builds a knowledge graph (see Databases — Graph) from the source documents, and retrieval can traverse relationships between entities directly, rather than relying purely on chunk-level similarity.

RAG Optimization

Everything above this section establishes correctness — this section is about making the same pipeline fast and cheap enough to actually run at real production query volume, without rebuilding any of it.

Index optimization. Production vector search almost universally runs on HNSW (Hierarchical Navigable Small World) graphs rather than exact nearest-neighbor search, which stops scaling past a small corpus. Two parameters do most of the work:

• ef_construction: how thoroughly the index explores candidate connections while it's built — higher values produce a higher-quality graph (better recall later) at the cost of slower, more expensive indexing, paid once.
• ef_search: the equivalent knob at query time — how many candidates the search explores before returning results. This is the parameter actually tuned in production, live: push it up for better recall at higher per-query latency, pull it down for faster, cheaper queries at some recall cost. It's the single biggest recall/latency lever in a deployed vector index.

Quantization. Storing every embedding at full float32 precision gets expensive at real corpus scale. Scalar quantization (e.g. float32 → int8) shrinks storage with modest accuracy loss; Product Quantization (PQ) goes further — splitting each vector into sub-vectors and replacing each sub-vector with the nearest entry in a small learned codebook, trading some retrieval accuracy for a much larger compression ratio than scalar quantization alone. The right choice is a direct function of how much accuracy loss the application can tolerate versus how much the index costs to store and search at the corpus's real size.

Semantic caching. Real production query traffic repeats — not usually word-for-word, but semantically: many users ask close variants of the same handful of questions. Semantic caching embeds each incoming query and checks it against previously served queries by similarity (not exact string match); a close-enough hit reuses the cached retrieval (or even the cached final answer) instead of re-running the full pipeline. This is a genuine, increasingly standard production technique (tools like GPTCache implement it directly) — the tradeoff is the similarity threshold itself: too loose and semantically different queries get served a wrong cached answer; too tight and the cache rarely hits at all.

Embedding fine-tuning on domain data. A general-purpose embedding model is trained on broad web text, not your specific domain's vocabulary and relevance judgments — fine-tuning it on domain query-document pairs (contrastive/triplet-loss training, pulling truly relevant pairs closer and irrelevant ones apart) can meaningfully close the gap between "generically similar" and "actually relevant for this specific corpus," at the cost of needing labeled or weakly-labeled domain relevance data to train on.

Evaluating RAG

Pipeline step 9, and the last one — none of the steps above matter if there's no way to tell whether the whole pipeline is actually working:

• Faithfulness / groundedness: does the generated answer actually match what the retrieved context says (or does it hallucinate beyond it) — is every claim in the answer traceable back to the retrieved chunks?
• Answer relevance: does the generated answer actually address the user's question (a faithful-to-context but off-topic answer still fails the user).
• Context precision/recall: of the chunks retrieved, how many were relevant (precision); of all relevant chunks that exist, how many were retrieved (recall) — see Model Evaluation & Metrics for the general precision/recall definitions this specializes.
• MRR and NDCG: the same ranking-quality metrics from Learning-to-Rank apply directly to retrieval quality — did the most relevant chunk rank first, not just appear somewhere in the top-k.

See LLM Evaluation & RAGOps for the full evaluation tooling (Ragas, DeepEval) and production RAG monitoring.

Common Failure Modes

• Hallucination despite correct context → the model ignores or misreads retrieved content; mitigated with stronger grounding instructions and citation requirements
• "Lost in the middle" on long retrieved context → reorder chunks to put the most relevant near the start/end of the prompt
• Multi-hop questions failing → GraphRAG, or explicit query decomposition
• Stale knowledge base → versioned, incremental re-indexing pipelines rather than full rebuilds
• PDF/table parsing issues → dedicated document-layout parsers instead of naive text extraction
• Per-user access control → metadata filtering at retrieval time, scoped to what each user is authorized to see

Next: Evaluation & Serving — measuring whether any of this actually works, and running it in production.