Retrieval & Reranking Architectures: Bi-Encoder, Cross-Encoder, Late Interaction

RAG's Re-ranking section introduces bi-encoders and cross-encoders as the two-stage retrieve-then-rank shape. There's a real third point on that spectrum — late interaction, best known via ColBERT — that's neither of the other two: it keeps a bi-encoder's precomputability and recovers much of a cross-encoder's token-level precision, by never pooling embeddings down to one vector per document in the first place.

All three architectures, same visual grammar, stepped through the same story — what goes in, what happens, what's lost or kept, and why it matters:

Bi-Encoder: Independent Encoding, Pooled to One Vector

What is it? A way to compare a query against a document by summarizing each one into a single number-list (a vector) separately, then measuring how close those two summaries are.

How does it work, concretely? Take a real, widely-deployed bi-encoder: BAAI/bge-large-en-v1.5 — BERT-large under the hood (24 transformer layers, hidden size 1024, 16 attention heads, verified directly against its real config.json, not a model-card summary).

• The query gets fed through the model on its own, through all 24 layers, producing one contextual embedding per token.
• Each document, completely independently, gets fed through the exact same 24 layers — identical, shared weights — the model never sees the query and the document together, and there is no cross-attention between them at any point.
• Both sides get pooled down into a single vector. This specific model uses CLS-token pooling (the final hidden state of the [CLS] token stands in for the whole sequence) — worth calling out explicitly, since mean pooling (averaging every token's final embedding) is the more common choice among SBERT-style models generally; the two are different design choices, not interchangeable defaults.
• The two single 1024-dim vectors are compared with cosine similarity, a fast, cheap number-crunching step.

Why is it useful? Because a document's embedding never depends on the query, it can be computed once, offline, for every document in the corpus, and stored in a fast approximate-nearest-neighbor index. That precomputation is what lets a bi-encoder search millions of documents in milliseconds — no other architecture on this page can do that.

What "the embedding model" actually is: every bi-encoder on this page is an SBERT (Sentence-BERT)-style model — a pretrained encoder fine-tuned specifically (contrastive learning against hard negatives, temperature 0.01 for BGE) so cosine similarity between pooled vectors becomes meaningful. That's not an implementation detail; it's the entire reason bi-encoder embeddings work for retrieval at all. One more real, practical detail: earlier BGE versions needed a special instruction prefix prepended to queries (e.g. "Represent this sentence for searching relevant passages:") to get good results — v1.5 specifically relaxed this, so it works well with no instruction at all, a small usability win over earlier versions.

In code, using the sentence-transformers library — the standard implementation of this pattern:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-large-en-v1.5")

# Document embeddings: computed once, offline -- this is what makes a
# bi-encoder index-able. In production this runs once per document at
# ingestion time, not on every query.
doc_embeddings = model.encode([
    "Gradient descent updates parameters by stepping opposite the loss gradient.",
    "The KV cache stores previously computed Key/Value vectors during generation.",
])

# Query embedding: the only thing computed at query time.
query_embedding = model.encode("how does attention caching work")
similarities = model.similarity(query_embedding, doc_embeddings)

Limitation: Pooling to one vector necessarily throws away token-level detail. Two documents that are similar "on average" but differ in one crucial specific term can end up looking identical after pooling — the model simply never gets a chance to compare that one term directly against the query.

A bi-encoder's blind spot — losing detail to pooling — is precisely the gap a cross-encoder exists to close, by giving up precomputability in exchange for letting the query and document interact directly.

Cross-Encoder: Full Joint Attention, One Score at a Time

What is it? A way to score a query against one specific document by feeding both through the model together, letting every part of one directly influence how the other gets read.

How does it work, concretely? The reranker already used in the code above, cross-encoder/ms-marco-MiniLM-L6-v2 — 6 transformer layers, hidden size 384, 12 attention heads (its own config.json, again — not the model card's own lineage description, which is genuinely misleading here: it references being distilled from a 12-layer MiniLM checkpoint, but the deployed model actually has 6 layers, matching the "L6" in its real name).

• The query and one candidate document are concatenated into a single sequence — [CLS] query [SEP] passage [SEP] — and fed through the model together, not processed separately.
• That combined sequence goes through all 6 layers as one pass, with real self-attention: every query token can attend to every passage token (and vice versa), including directly across the query/passage boundary, before anything gets summarized.
• The final hidden state at the [CLS] position feeds a small classification head (384 → 1) that outputs one scalar relevance score for that specific (query, passage) pair — not a reusable embedding for either side alone.

Why is it useful? Because the model reads query and document jointly, it can pick up on exactly the kind of fine-grained interaction a bi-encoder's separate pooling step throws away — genuine token-level comparison, not an after-the-fact distance between two summaries. This is the most accurate of the three architectures on this page: this specific model reports 39.01 MRR@10 on the MS MARCO Passage Ranking dev set and 74.30 NDCG@10 on TREC DL 2019, both real numbers from its own model card, verified against its listed benchmark table.

In code, same library, a different class:

from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

# Query and document fed through TOGETHER -- one score per pair, and
# nothing here is reusable for the next query the way an embedding is.
scores = model.predict([
    ("how does attention caching work", "The KV cache stores previously computed Key/Value vectors during generation."),
    ("how does attention caching work", "Gradient descent updates parameters by stepping opposite the loss gradient."),
])

Limitation: Nothing about a cross-encoder's score is precomputable — it only exists once you have both the query and a specific document in hand, together, at the same time. Running it against a full corpus of millions would mean millions of full forward passes per query, which is why it's only practical against a small candidate set: tens to low hundreds of documents, not millions.

Retrieve, Then Rerank: Both Stages Together in Code

The two code snippets above aren't run in isolation in production — they're chained, in exactly the order their limitations dictate: the bi-encoder narrows a corpus no cross-encoder could touch directly, and the cross-encoder refines a shortlist too small for the bi-encoder to do accurately on its own.

from sentence_transformers import SentenceTransformer, CrossEncoder

corpus = [
    "Linear regression fits a straight line by minimizing squared error.",
    "Gradient descent updates parameters by stepping opposite the loss gradient.",
    "Self-attention lets every token attend to every other token via Query, Key, Value projections.",
    "The KV cache stores previously computed Key/Value vectors during autoregressive generation.",
    "Retrieval-augmented generation grounds LLM answers in retrieved text chunks.",
]
query = "how does attention work in transformers"

# Stage 1 -- bi-encoder: cheap enough to run over the whole corpus. In
# production, corpus_embeddings would already be precomputed and indexed;
# encoding them inline here just keeps this example self-contained.
bi_encoder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
corpus_embeddings = bi_encoder.encode(corpus)
query_embedding = bi_encoder.encode(query)

sims = bi_encoder.similarity(query_embedding, corpus_embeddings)[0]  # similarity to every doc
top_k = sorted(range(len(corpus)), key=lambda i: -sims[i])[:3]
shortlist = [corpus[i] for i in top_k]

# Stage 2 -- cross-encoder: only reranks the shortlist, never the full corpus.
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
ranked = cross_encoder.rank(query, shortlist)

for r in ranked:
    print(f"{r['score']:.2f}  {shortlist[r['corpus_id']]}")

This is real, current sentence-transformers API (verified directly against sbert.net's own docs) — copy-pasteable in your own Python environment, not a browser demo: both models pull in PyTorch and download real weights on first use, too heavy to run inside this page the way the smaller RunnableCode examples elsewhere on this site do.

Cross-encoders buy accuracy by giving up precomputability entirely. Late interaction asks a different question: what if a document's tokens could stay precomputable without being pooled away into one vector at all?

Late Interaction: Per-Token Embeddings, Scored by MaxSim

What is it? A middle path — the architecture behind ColBERT (Khattab & Zaharia, 2020) — that keeps every individual token's own embedding, for both query and document, instead of collapsing either down to a single vector.

How does it work?

• Query and document tokens are each embedded independently, exactly like a bi-encoder — the model never needs both at once to produce them.
• Document token embeddings are precomputed and indexed offline, one vector per token, not one per document.
• At query time, for every query token, the model finds its single best-matching document token — the highest similarity, not an average — a step called MaxSim.
• Those per-token best-matches are summed into one score for the document.

Written as a formula, once the plain-language version above has landed:

$\text{MaxSim}(Q, D) = \sum_{i=1}^{|Q|} \max_{j=1}^{|D|} \left(q_i \cdot d_j\right)$

Why is it useful? No document ever gets collapsed into one vector, so no fine-grained detail is lost to pooling — but because every document token embedding was already computed and indexed ahead of time, nothing at query time requires running the model on the document again. It keeps a bi-encoder's precomputability and recovers much of a cross-encoder's precision.

Pure arithmetic — dot products, max, sum — so unlike the transformer models above, this runs for real, live, in your browser. The exact same query/doc token embeddings as the diagram above; edit them and re-run to see the score actually recompute:

Limitation: Keeping one vector per token instead of one per document is a real, unavoidable storage cost — see the numbers below.

Comparing the Three

Every architecture above assumes the right document was at least handed to it to score. That assumption can fail earlier than any of these three architectures — and no amount of reranking accuracy fixes it after the fact.

Reranking Can't Recover What Retrieval Missed

What is it? A production failure mode that has nothing to do with how good the reranker is: the reranker can only reorder the candidates it's given, so a relevant document that never made it into that candidate set is simply never seen, no matter how the ranking step performs.

How does it happen? First-stage retrieval (bi-encoder, sparse/BM25, or late interaction) narrows a full corpus down to a top-$k$ candidate set — typically top-100 or so. If the genuinely best-matching document doesn't make that cut, it's gone before reranking even starts; reranking operates strictly on the candidates it's handed, not on the corpus.

Why it matters: first-stage retrieval recall — the fraction of truly relevant documents that make it into the retrieved candidate set at all — is as important a production metric as reranking accuracy, not a secondary concern. It's also a direct argument for late interaction specifically: because MaxSim keeps token-level detail at the retrieval stage itself (not just at reranking), it can catch relevant documents a pooled bi-encoder vector would have missed in the first place — moving the "did we even find it" failure mode earlier, where it's cheaper to fix.

Use Cases: Which One, for What

Not a repeat of the architecture explanations above — the concrete jobs each one actually gets reached for in production, tied back to the real numbers already on this page.

Bi-encoder — anywhere a candidate set has to be found from millions of options, fast, using only a precomputed index:

• First-stage retrieval in any RAG pipeline — the only one of the three architectures that can search millions of chunks directly, so it's almost always the first stage regardless of what runs after it.
• Semantic search generally — "find documents like this query" over a large corpus, the same core operation RAG retrieval is a special case of.
• Recommendation candidate generation — embedding users and items into the same space so "find items similar to what this user likes" becomes the identical nearest-neighbor lookup (see Recommender Systems — Two-Tower Architecture for the same bi-encoder shape applied to users/items instead of queries/documents).

Cross-encoder — anywhere the candidate set is already small and the question is a precise pairwise judgment, not a search over millions:

• Final-stage reranking over a bi-encoder's (or late interaction's) shortlist — maximum accuracy exactly where it's affordable, tens to low hundreds of candidates.
• Duplicate and paraphrase detection — "are these two specific sentences saying the same thing," a direct pairwise-judgment task cross-encoders are built for, not a search problem.
• NLI-style pair classification — entailment/contradiction/neutral judgments between a premise and hypothesis (the same NLI data SBERT is fine-tuned on), where the model needs to read both sentences together, not compare precomputed vectors.

Late interaction / ColBERT — the middle ground, reached for when a bi-encoder's accuracy ceiling matters enough to justify the storage cost:

• As a middle-ground reranker: more accurate than a bi-encoder's cosine similarity, meaningfully cheaper than running a full cross-encoder over the same candidates — the ~175× latency / ~13,900× FLOPs gap from the ColBERT paper's own numbers above.
• As the first-stage retriever itself, when the storage budget allows: RAGatouille (already mentioned above) wraps ColBERT/PLAID specifically to make this deployable, and Vespa — a production search engine — ships a native colbert-embedder with built-in MaxSim ranking and int8 document-vector compression (a real ~32× storage reduction) for exactly this use case, not a research-only pattern.

Next: back to RAG for how retrieval and reranking fit into the full pipeline, or LLM Hosting & Serving Patterns for embedding/reranker inference in production.