Retrieval-Augmented Generation (RAG)

An LLM's knowledge is frozen at training time and limited to what fit in its weights. RAG fixes both problems by retrieving relevant information at query time and feeding it into the prompt as grounding context.

RAG isn't the only way to get a model to know things it wasn't trained on — fine-tuning is the other lever, and it's worth being clear about which one actually fixes which problem, because reaching for the wrong one wastes real engineering time. Fine-tuning changes the model's weights, which makes it genuinely good at adopting a tone, a format, or a narrow skill — but it's a slow, expensive way to teach a model facts, and those facts freeze the moment training ends. RAG does the opposite: it leaves the weights untouched and instead hands the model fresh, specific facts at query time, which makes it the right tool whenever the knowledge itself needs to stay current or come from documents too numerous or too sensitive to bake into weights. In practice, production systems often use both — RAG for the facts, a lightly fine-tuned model for the voice they're delivered in — rather than treating it as an either-or choice.

The RAG Pipeline, End to End

Nine steps, in the order they actually run. Everything else on this page is one of these steps covered in real depth — worth seeing the whole shape once before diving into any single piece. One operating principle is worth internalizing before any of them: retrieval quality is the rate-limiting factor for the entire system. A perfect generation model fed the wrong chunks still produces a wrong answer — so when a RAG system is giving bad answers, the fix is almost always upstream of generation, in how documents were chunked, embedded, or retrieved, not in prompting the LLM harder.

1. Ingestion: pull in your raw source documents — PDFs, wikis, tickets, whatever your knowledge base actually is — and get them into clean, parseable text. This step is more failure-prone than it sounds: a PDF with multi-column layouts or embedded tables will mangle badly under naive text extraction, which is why production ingestion usually runs through a dedicated document-layout parser (tools like LlamaParse exist specifically for this) rather than a plain text-extraction call, and why it's worth capturing structural metadata — section headings, page numbers, source filenames — at this stage, since it's far cheaper to keep than to reconstruct later.
2. Chunking: split that text into retrievable units. The size and strategy chosen here shapes the quality of every step downstream.
3. Embedding: turn each chunk into a vector using an embedding model (see Foundation Model Internals).
4. Indexing: store those vectors in a structure built for fast similarity search at scale (see RAG Optimization for how that index is actually tuned in production, and Databases for the storage layer itself).
5. Query Processing: transform the user's raw question before searching with it — decompose it, rephrase it, or generate a hypothetical answer to search with instead.
6. Retrieval: search the index — dense, sparse, or hybrid — using the (possibly transformed) query, and pull back the chunks most likely to be relevant.
7. Reranking: re-score the retrieved candidates with a more precise, more expensive method before committing to a final set.
8. Generation: feed the reranked chunks into the LLM's context alongside the query (see Prompt Engineering for how that context actually gets structured into a prompt), and let it generate an answer grounded in what it just read. When there are several chunks to combine rather than one obvious answer, how they're combined matters too — compact just concatenates chunks into a single prompt, refine answers from the first chunk and then revises that answer against each subsequent chunk in turn, and tree_summarize summarizes chunks in parallel and then summarizes those summaries, trading cost and latency against how well the final answer actually synthesizes everything retrieved rather than leaning on whichever chunk happened to come first.
9. Evaluation & Monitoring: measure whether the whole pipeline actually produces faithful, relevant answers — both before shipping and continuously once it's live.

Click any node below for what it does, then type a real question and watch retrieval actually run — real chunking, real cosine-similarity ranking, a real (if simplified) reranking pass:

Chunking Strategies

Pipeline step 2. Once documents are ingested, this is the first real decision point — every later step operates on whatever units get produced here.

Say you're building search over a pile of internal documents. The simplest thing you can do is slice every document into pieces of, say, 500 tokens and move on — this is fixed-size chunking, and it's exactly as crude as it sounds. It doesn't know or care where a sentence ends, so it will happily cut a paragraph in half mid-thought, leaving two chunks that each make less sense alone than the original made together.

The obvious fix is to respect the document's own structure instead of chopping it blindly. Semantic chunking splits at paragraph breaks, section headers, or topic shifts, so each chunk is a complete, coherent unit of meaning rather than an arbitrary slice. The version of this you'll actually reach for in practice is recursive chunking: try splitting on the largest natural boundary first (a section), and only fall back to a smaller one (a sentence, then a word) if a piece is still too big. It's semantic chunking with a size limit, and it's the default most teams start with.

Even a clean, well-formed chunk has a problem, though — it was sized for retrieval, where small and specific is what lets it match a query precisely, but that same smallness often starves the LLM of surrounding context once the chunk is actually handed to it for generation. Parent-child chunking solves this by decoupling the two jobs: retrieve using small, precise child chunks, but once one is found, pass its larger parent chunk — the fuller paragraph or section it came from — to the LLM instead. You get precise retrieval and generous context from two different sizes of the same text.

Late chunking solves a different, subtler problem, and it does so by working backwards from everything above. Every strategy so far chunks first and embeds after, which means each chunk gets embedded in isolation, blind to anything outside its own boundaries. If a chunk reads "it reduced costs by 40%" with no idea what "it" refers to, its embedding has no way to encode that either — the context that would resolve it lives in an earlier chunk the embedding model never saw. Late chunking flips the order: embed the entire document first with a long-context embedding model, so its attention mechanism can resolve those cross-references while it can still see the whole document, and only pool the resulting token embeddings into chunks afterward. Each chunk's embedding ends up carrying whole-document context, even though the chunk itself, read alone, wouldn't obviously connect to it.

A closely related technique, contextual retrieval, attacks the exact same isolation problem from the opposite direction: instead of changing when embedding happens, it changes the chunk's text before embedding it at all — an LLM reads each chunk alongside the full document and prepends a short, generated blurb of context ("this chunk is from the Q3 earnings call, discussing the cloud division's margin") directly onto the chunk before it gets embedded or indexed. Anthropic's own cookbook popularized this specific version, making it cheap in practice by caching the shared document context across the many chunks generated from it, rather than re-processing the whole document from scratch for each one.

A couple of numbers are worth having on hand rather than guessing at. Parent-child setups typically run child chunks around 100–200 tokens with parents in the 500–1000 token range; recursive splitters are usually built on sentence boundaries detected by a real sentence tokenizer (spaCy or NLTK), not naive period-splitting, which breaks on abbreviations and decimals. And one failure worth watching for regardless of strategy: any chunk that exceeds the embedding model's max input length doesn't error out — it just gets silently truncated, so the back half of an oversized chunk is quietly dropped from the index with no warning anywhere in the pipeline.

A small habit pays for itself across all of the strategies above: prepend the source document's title and the enclosing section heading to a chunk's text before embedding it. A chunk that just says "set the timeout to 30 seconds" is ambiguous on its own; the same text prefixed with "Database Connection Pooling — Advanced Settings:" is not.

Recursive chunking in code, via LangChain's RecursiveCharacterTextSplitter — the standard implementation of the "try large boundaries first, fall back to smaller ones" strategy described above:

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Tries splitting on paragraph breaks first, then sentences, then words --
# only falls back to a smaller separator when a chunk is still too large.
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_text(document_text)

chunk_overlap repeats a small amount of text between consecutive chunks, so a sentence or idea split across a chunk boundary still appears in full in at least one of the two chunks.

Choosing an Embedding Model

Pipeline step 3. Whatever chunks step 2 produced now need to become vectors, and the model you pick to do that is a real decision with real consequences, not a formality. Embedding models trade off against each other on four axes that matter in practice: how good the embeddings actually are at your kind of text, how fast the model runs, what it costs per million tokens, and how well it fits your domain — a model tuned mostly on general web text (like text-embedding-3-small) can genuinely underperform on legal contracts or medical notes, where the vocabulary and the very notion of "similar" differ from the open web. The MTEB benchmark is the standard public way to compare candidates before committing to one, though it's still worth spot-checking a shortlist against your own documents rather than trusting a leaderboard rank alone.

Dimensionality is part of the same trade-off, and current models span a wide range — commonly 768 to 3072 dimensions. A higher-dimensional embedding usually captures more nuance, but it also costs more to store and search once you're indexing millions of chunks, so the right choice depends on your corpus's real size, not just which number looks best on a benchmark. Matryoshka embeddings offer a practical way to avoid committing to one point on that trade-off up front: they're trained so that a truncated prefix of the full vector is still a valid, usable embedding, letting you store the full-size vector but search with a shorter, cheaper slice of it when speed matters more than top accuracy.

Two gotchas are worth knowing before they bite you in production. Some models (E5 is the well-known example) are trained with asymmetric "query: " and "passage: " text prefixes, and skipping them measurably hurts retrieval quality — always check a model's docs for this before embedding anything. And because different embedding models place meaning in geometrically unrelated vector spaces, never mix embeddings from two different models in the same index — a query embedded by one model has no meaningful distance to a chunk embedded by another, even if both models are individually good.

In code, via the standard sentence-transformers library (see Retrieval & Reranking Architectures for the bi-encoder architecture this implements):

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# Every chunk from the chunking step, embedded once and indexed --
# this is the vector data that step 4 (indexing) stores.
chunk_embeddings = model.encode(chunks)

What retrieval above is actually doing under the hood — comparing vectors by cosine similarity in a learned space where meaning clusters geometrically — is exactly what the Embedding Space Explorer makes clickable:

Query Transformation

Pipeline step 5. The query a user actually types is rarely the query you'd want to search with. A terse, three-word question is a poor embedding-space match for the long, well-formed paragraph in your documents that actually answers it — so before any retrieval happens, it's often worth transforming the query first.

One effective trick looks backwards: instead of searching with the question itself, ask the LLM to write a hypothetical answer to it, then embed and search with that fabricated answer in place of the real query. This is HyDE (Hypothetical Document Embeddings), and it works because a plausible-looking answer, even a wrong one, tends to sit much closer in embedding space to a real relevant document than the terse original question does — it's playing the same game the documents are playing, instead of asking about it from the outside.

Other transformations attack different problems. If a question is really several questions stitched together ("how does X compare to Y, and which one works better for Z"), query decomposition breaks it into its separate sub-questions and retrieves for each independently, rather than hoping one search satisfies all three at once. Step-back prompting goes the opposite direction — it asks a broader, more general version of the question first, pulling in context a narrowly-scoped original question would have missed entirely.

A third family hedges against a far more mundane failure: vocabulary mismatch. Two people can ask the exact same thing in different words, and a retriever that only ever searches with the literal phrasing a user happened to type can miss a chunk that would have matched perfectly under slightly different wording. Multi-query retrieval has the LLM generate several rephrasings of the same question, searches for each independently, and takes the union of whatever comes back. RAG-Fusion is the more careful version of the same idea: instead of simply unioning the result lists, it combines their rankings with Reciprocal Rank Fusion (the same technique the hybrid-search section below uses), so a chunk that surfaces near the top across several rephrasings rightly outranks one that only showed up once.

One more layer worth knowing by name, even though it isn't really a query transformation: Adaptive RAG uses a lightweight classifier on the incoming query to route it — a simple factual lookup might skip retrieval entirely, a straightforward question gets a single retrieval pass, and a genuinely complex one gets routed into decomposition or a multi-hop strategy. It's the same instinct behind everything above, pushed one level up: don't just transform the query, decide how much transformation it actually needs.

Whichever version of the query comes out of this step — rephrased, decomposed, or hypothetical — is what actually gets searched next.

Dense vs. Sparse Retrieval, and BM25

Pipeline step 6. Two structurally different ways to find relevant chunks — worth distinguishing precisely rather than lumping together as "search," because they fail in genuinely different, complementary ways.

Dense Retrieval

Dense retrieval finds chunks by meaning, using the same embedding-space idea this page has relied on since the chunking step: embed the query and every chunk into the same continuous vector space (see Foundation Model Internals), and rank candidates by how close they land to the query — the same cosine-similarity comparison the embedding step computes. The payoff is real: a query about "canines" can surface a chunk about "dogs" with zero words in common, because the embedding space clusters the two concepts near each other regardless of exact wording. (The architecture behind this — a bi-encoder trained with contrastive loss to pull relevant query-document pairs together — traces back to Dense Passage Retrieval, DPR, the paper that established this approach as the default.)

But that same strength is exactly where it breaks. Because dense retrieval compares meaning rather than literal text, it can walk right past an exact match that actually matters — a specific product code, an exact legal clause, a precise error string — where the literal characters are the point, not the general topic they happen to belong to.

Dense retrieval's blind spot — missing exact term matches — is precisely what sparse retrieval, the classical information-retrieval approach, was built to guarantee.

Sparse Retrieval and BM25

Sparse retrieval takes the opposite bet: forget meaning, match exact words. It represents each chunk as a vector over the vocabulary — which terms appear, and how often — and scores a chunk against a query with BM25, the scoring function that's survived largely unchanged since long before embeddings existed. The naive version of this idea would just count shared words, but BM25 adds two corrections that make it actually work well: it saturates the value of repetition, so a word's fifth occurrence in a document barely moves the score beyond what the first occurrence already contributed, and it weighs a match by how informative the matched term actually is — matching "the" tells you nothing, because every document has it, while matching a rare technical term tells you a great deal, and a match in an otherwise short, focused document counts for more than the identical match buried in a long document that only mentions it in passing.

Sparse retrieval's whole value proposition is that it guarantees literal matches don't get missed — exactly the case dense retrieval struggles with. But it pays for that guarantee with the opposite blind spot: it has no concept of meaning at all, so a query about "canines" will not match a chunk that only ever says "dogs," no matter how relevant that chunk obviously is to a human reading it, because there isn't a single shared term for BM25 to count.

In code, via rank_bm25, the standard Python implementation — it deliberately does no text preprocessing itself, so tokenizing (and lowercasing, stemming, etc. if wanted) is the caller's job:

from rank_bm25 import BM25Okapi

tokenized_corpus = [chunk.lower().split() for chunk in chunks]
bm25 = BM25Okapi(tokenized_corpus)

tokenized_query = query.lower().split()
top_chunks = bm25.get_top_n(tokenized_query, chunks, n=5)

Dense misses exact matches; sparse misses meaning — two failures with no overlap, which is exactly what makes running both together, rather than picking one, worth the extra cost.

Hybrid Search

Given that dense and sparse retrieval fail in opposite, non-overlapping ways, the obvious move in production isn't to pick one — it's to run both and combine what they find. That's hybrid search: dense and sparse retrieval execute independently against the same query, producing two separate ranked lists, which then get merged into one. The result is a chunk that can surface for either reason — because it's semantically close (dense) or because it shares exact terms with the query (sparse) — covering each method's blind spot with the other.

The merging itself is the interesting part, and Reciprocal Rank Fusion is the standard way to do it: you can't just average a cosine similarity and a BM25 score, because they live on completely different, incompatible scales, so RRF throws the raw scores away and combines the lists using only each result's position in its own ranking (the full mechanics are in the Go Deeper box below). A simpler, more ad-hoc alternative some teams use instead is alpha-weighting — picking a fixed blend weight and linearly combining normalized dense and sparse scores directly — which is easier to reason about but more sensitive to how you normalize scores in the first place; RRF's rank-only approach sidesteps that fragility entirely, which is why it's the more common production default. For corpora up to a few hundred thousand documents, the sparse side of this is usually just the same rank_bm25 library from the BM25 example above — no separate infrastructure needed until the corpus outgrows it.

Re-ranking with Cross-Encoders

Pipeline step 7. The retrieval step above (dense, sparse, or hybrid) has to be fast enough to search potentially millions of chunks — it uses a bi-encoder architecture, embedding the query and each document independently so document embeddings can be precomputed and indexed once (see Recommender Systems — Two-Tower Architecture for the identical architectural pattern). A cross-encoder re-ranker instead feeds the query and a candidate document into the model together, letting it directly attend across both — far more accurate at judging true relevance, but far too slow to run against the entire corpus, since nothing about it can be precomputed per-document. The standard pattern is exactly this two-stage shape: cheap bi-encoder retrieval narrows millions of chunks to a few dozen candidates, then an expensive cross-encoder re-ranks just those few dozen precisely — the same retrieval-then-rank funnel as Recommender Systems and Learning-to-Rank.

In practice, most teams don't train their own cross-encoder — they call a hosted reranking API, Cohere Rerank, Jina Reranker, and Mixedbread's rerank models being the common choices, each adding roughly 100–400ms of latency per call in exchange for not having to host the model yourself. A simpler option that needs no dedicated reranker model at all is to just ask the LLM itself to score or reorder the candidates — slower and more expensive per call, but it needs no extra infrastructure beyond the LLM you already have. Whichever reranker you use, it's worth deliberately over-retrieving going in — pulling back something like 3–5x as many candidates from the first stage as you actually intend to keep after reranking — since a candidate the first stage ranked outside its usual top cutoff can still turn out to be the most relevant once the more careful reranker actually looks at it.

A third architecture sits between these two: late interaction (ColBERT) keeps a separate embedding per token instead of pooling to one vector per document, and scores query-document pairs via MaxSim — for each query token, the single best-matching document token, summed. Document token embeddings stay precomputable and indexable, like a bi-encoder's, but nothing gets collapsed away, so it recovers much of a cross-encoder's precision without the joint forward pass. See Retrieval & Reranking Architectures for the real MaxSim formula, a fully worked numeric example, and a three-way comparison of when production systems reach for each of the three.

Advanced RAG Architectures

Everything described so far is a single, fixed pass: retrieve once, then generate. A handful of named architectures exist specifically because that fixed shape breaks down for certain classes of question, and each one fixes a different part of it.

The most general fix adds a control loop around retrieval itself. In Agentic RAG, an agent (see Agent Architectures) decides whether to retrieve at all, what to search for, and how many times — it can issue a second retrieval call after seeing the first one come back thin, reformulate its own query, or decide outright that it already has enough and skip retrieval entirely, instead of being locked into exactly one retrieval step no matter what the question actually needs. Corrective RAG (CRAG) is a narrower, cheaper version of the same instinct: after retrieval, a lightweight grader checks whether what came back is actually relevant and sufficient, and if it isn't, falls back to rewriting the query, searching more broadly, or even going to general web search — rather than generating an answer from context that was never going to support one. FLARE adds the same correction at a finer grain, mid-sentence rather than mid-pipeline: it watches the model's own confidence while it generates, and triggers a fresh retrieval call the moment confidence drops below a threshold — useful for long answers where the information needed partway through is different from what the opening sentence needed. Self-RAG pushes the judgment one level deeper still, training the LLM to emit special reflection tokens while it generates — deciding on its own whether retrieval was needed, whether the passages it got back were relevant, and whether its own answer is actually supported by them — folding the whole "should I retrieve, was this useful, am I grounded" loop into the model's own token stream instead of a separate pipeline stage wrapped around it.

A second group of fixes changes not when retrieval happens but what gets retrieved, and from where. RAPTOR builds a tree of recursive summaries over the corpus — clustering related chunks, summarizing each cluster, then clustering and summarizing those summaries again going up several levels — so retrieval can pull back a high-level summary for a broad question or a leaf-level chunk for a specific one, rather than being stuck at one fixed granularity. The contextual retrieval technique mentioned in the chunking section above is a lighter-weight relative of the same idea: rather than building a separate summary tree, it just prepends generated context directly onto each chunk before indexing. Multimodal RAG retrieves image embeddings (via CLIP-style models — see Computer Vision) alongside or instead of text, for questions that can only really be answered by looking at a chart, a diagram, or a photo in the source material. SQL RAG skips unstructured chunks altogether for questions whose answer genuinely lives in a database — it translates the natural-language question into an actual SQL query (see Databases), runs it, and hands the LLM the query result as grounding, which is the right move whenever the question is really asking for an aggregate or an exact count that no retrieved paragraph would reliably contain. Code RAG specializes the whole pipeline for source code — chunking around functions and classes instead of arbitrary token windows, and embedding with code-aware models — so an AI coding assistant can ground its suggestions in what a specific repository's code actually does, rather than in the model's general, repository-agnostic sense of how code like this is usually written.

One more architecture is worth knowing precisely because it rejects the whole premise: Cache-Augmented Generation (CAG) skips retrieval entirely by preloading an entire (small enough) corpus directly into the model's KV cache up front, letting the model attend over all of it on every query with no retrieval step at all. It only works when the corpus is small enough to fit, but when it does, it sidesteps every retrieval-quality failure mode on this page by construction — there's no "wrong chunk retrieved" if nothing was ever selectively retrieved in the first place.

GraphRAG

Standard RAG retrieves isolated chunks, which struggles with questions that require connecting facts across multiple documents (multi-hop reasoning). GraphRAG instead builds a knowledge graph (see Databases — Graph) from the source documents, and retrieval can traverse relationships between entities directly, rather than relying purely on chunk-level similarity.

RAG Optimization

Everything above this section establishes correctness — this section is about making the same pipeline fast and cheap enough to actually run at real production query volume, without rebuilding any of it.

Index optimization. Production vector search almost universally runs on HNSW (Hierarchical Navigable Small World) graphs rather than exact nearest-neighbor search, which stops scaling past a small corpus. Two parameters do most of the work, and they trade off at two different times. ef_construction controls how thoroughly the index explores candidate connections while it's being built — push it higher and you get a better-quality graph with better recall later, at the cost of slower, more expensive indexing paid once, up front. ef_search is the equivalent knob at query time, controlling how many candidates a search explores before returning results — and because it's tunable live, it's the lever teams actually reach for in production: push it up for better recall when latency has headroom, pull it down for faster, cheaper queries when it doesn't. If there's one knob to understand in a deployed vector index, it's this one.

None of this requires building an index from scratch, either — there's a real, well-established menu of vector databases to choose from, and the right one depends mostly on scale and what's already in your stack. If you're already running Postgres, pgvector is the pragmatic starting point and handles up to roughly a million vectors comfortably before it's worth migrating off; past that, purpose-built stores like Pinecone, Qdrant, Weaviate, Milvus, Turbopuffer, and ChromaDB are the common names you'll see in production RAG stacks, each with its own take on the same HNSW-plus-quantization foundation described below.

Quantization. Storing every embedding at full float32 precision gets expensive at real corpus scale. Scalar quantization (e.g. float32 → int8) shrinks storage with modest accuracy loss; Product Quantization (PQ) goes further — splitting each vector into sub-vectors and replacing each sub-vector with the nearest entry in a small learned codebook, trading some retrieval accuracy for a much larger compression ratio than scalar quantization alone. The right choice is a direct function of how much accuracy loss the application can tolerate versus how much the index costs to store and search at the corpus's real size.

Semantic caching. Real production query traffic repeats — not usually word-for-word, but semantically: many users ask close variants of the same handful of questions. Semantic caching embeds each incoming query and checks it against previously served queries by similarity (not exact string match); a close-enough hit reuses the cached retrieval (or even the cached final answer) instead of re-running the full pipeline. This is a genuine, increasingly standard production technique (tools like GPTCache implement it directly) — the tradeoff is the similarity threshold itself: too loose and semantically different queries get served a wrong cached answer; too tight and the cache rarely hits at all.

Embedding fine-tuning on domain data. A general-purpose embedding model is trained on broad web text, not your specific domain's vocabulary and relevance judgments — fine-tuning it on domain query-document pairs (contrastive/triplet-loss training, pulling truly relevant pairs closer and irrelevant ones apart) can meaningfully close the gap between "generically similar" and "actually relevant for this specific corpus," at the cost of needing labeled or weakly-labeled domain relevance data to train on.

Evaluating RAG

Pipeline step 9, and the last one. None of the eight steps above matter if there's no way to check whether the finished pipeline is actually doing its job — and "it gave a good-sounding answer" isn't a real check. It's worth separating out exactly what "working" means.

The first thing to check is faithfulness (also called groundedness): does the generated answer actually match what the retrieved context said, with every claim traceable back to a specific retrieved chunk, or has the model wandered off and asserted something the context never supported? A faithful answer still isn't automatically a good one, though — it also needs answer relevance, because a response that's perfectly grounded in the retrieved chunks but doesn't actually address what the user asked has still failed them. Then there's the retrieval step itself, evaluated independently of generation entirely: context precision asks, of the chunks that got retrieved, how many were actually relevant; context recall asks the reverse — of every relevant chunk that exists in the corpus, how many did retrieval actually find (see Model Evaluation & Metrics for where precision and recall come from generally). And because retrieval returns a ranked list, not just a set, it's worth also checking whether the single best chunk actually ranked first, rather than merely appearing somewhere in the top-k — which is exactly what MRR and NDCG, the same ranking metrics from Learning-to-Rank, are built to measure.

In practice, almost nobody computes these by hand — RAGAS is the open-source library most teams reach for, using an LLM as a judge to score faithfulness, relevance, and the precision/recall metrics above against a test set of real questions. Building that test set well matters as much as the metrics themselves: a good one has well over a hundred questions and deliberately spans exact lookups, multi-hop questions, questions with no real answer in the corpus, and genuinely ambiguous ones — not just the easy cases a system was obviously going to get right. And because the judge is itself an LLM, it comes with its own biases worth correcting for before trusting its scores blindly: a tendency to rate longer answers as better regardless of actual quality, a tendency to favor whichever answer it sees first when comparing two, and blind spots in domains the judge model itself doesn't know well — which is why it's worth validating a sample of the LLM judge's scores against real human labels before leaning on it at scale.

See LLM Evaluation & RAGOps for the full evaluation tooling (Ragas, DeepEval) and production RAG monitoring.

Common Failure Modes

A few failure patterns show up often enough in real systems to be worth recognizing by name, rather than re-diagnosing from scratch each time.

Sometimes the model hallucinates despite having correct context in front of it — it retrieved the right chunk and still ignored or misread it, which stronger grounding instructions and explicit citation requirements (forcing the model to point at which chunk supports each claim) help with directly. Long retrieved context has its own well-documented trap, "lost in the middle": models pay noticeably less attention to information buried in the center of a long prompt than to what's near the start or end, so simply reordering retrieved chunks to put the most relevant ones at the edges of the context — rather than wherever they happened to rank — measurably helps. Multi-hop questions, ones that need facts stitched together across several documents, are a structural mismatch for retrieving isolated chunks one at a time, and are better addressed with GraphRAG or explicit query decomposition than by throwing more chunks at the problem.

A stale knowledge base is less a RAG-algorithm problem than an infrastructure one. The real fix isn't periodically rebuilding the whole index from scratch — it's tracking a content hash or modification timestamp per source document, so the pipeline can detect exactly which documents actually changed and re-embed only those, incrementally, rather than paying the cost (and the staleness window) of a full rebuild every time. Parsing failures on PDFs and tables are a common, almost embarrassingly mundane cause of bad retrieval — naive text extraction mangles table structure badly enough that a dedicated document-layout parser is usually worth the extra step, as the ingestion step above already covers. And in any multi-tenant system, per-user access control has to be enforced at retrieval time itself, via metadata filtering scoped to what each specific user is actually authorized to see — not bolted on afterward by filtering the LLM's output, by which point the model has already read documents it should never have seen in the first place.

A smaller set of failures is worth watching for specifically because they're easy to misdiagnose as something else. Embedding mismatch happens when a team swaps embedding models but doesn't fully re-embed the existing index — old and new vectors end up sitting in the same store, geometrically incompatible with each other, and retrieval quality silently degrades in a way that looks like a model problem but is really a migration problem. Reranker failure is its own distinct category, not just "retrieval was wrong" — a reranker can confidently push the wrong candidate to the top, which is a different bug with a different fix (recalibrating or swapping the reranker, not touching the retrieval stage it operates downstream of). And in systems that lean on semantic caching for speed, cache leakage is a real risk worth naming explicitly: a cache hit threshold set too loosely can serve one user's cached answer to a different user's semantically-similar-but-not-identical query — including, in a multi-tenant system, serving content across a boundary it should never have crossed.

Next: Evaluation & Serving — measuring whether any of this actually works, and running it in production.