LLM, RAG & Agent Evaluation

Three progressively harder evaluation problems — a single text output, a text output grounded in retrieved context, and a multi-step sequence of decisions and tool calls — each needing a different evaluation approach because "correct" means something structurally different for each.

Intuition: Every Evaluator Is Itself Something to Evaluate

The theme connecting every section below: the thing doing the measuring (an LLM-judge, a coarse success metric) has its own failure modes, and trusting it without checking those failure modes just relocates the "is this measurement trustworthy" problem from Evaluation Fundamentals one level down instead of solving it. An LLM-judge can be biased by irrelevant surface features. A single success/fail metric can hide exactly how an agent got there. Both are demonstrated below with real, computed numbers.

LLM Evaluation and LLM-as-a-Judge, In Depth

LLM Evaluation & RAGOps covers the tooling (Ragas, DeepEval, LangSmith, Arize Phoenix) and the traditional-metrics-vs-LLM-as-judge distinction. Going one level deeper into how an LLM-as-judge actually works, and where it breaks.

The basic mechanism: prompt a (typically strong, often larger) LLM with the input, the output being evaluated, and explicit grading criteria, asking it to produce a score or judgment — effectively using a model's own language understanding as the measurement instrument, which scales far better than human review while capturing semantic quality that n-gram metrics (BLEU/ROUGE) structurally cannot.

Position Bias, Made to Flip a Verdict

In pairwise comparisons, judges tend to favor whichever response is shown first. Toggle the display order below on the exact same two responses and watch the verdict actually flip:

Standard mitigation: evaluate both orderings and check for consistency — precisely what the toggle above simulates.

Verbosity Bias, Measured With a Real Correlation

Judges tend to rate longer responses as better, independent of actual quality. Real quality is generated independent of length below; watch a real Pearson correlation appear anyway as judge bias increases:

Mitigated by explicit rubric criteria that don't reward length — not by hoping the judge doesn't notice length at all.

Self-preference bias rounds out the three: a model tends to rate its own outputs more favorably than a different model's — a real reason to use a different, ideally stronger, model as judge rather than a model evaluating itself.

Rubric Design and Validating the Judge

A vague instruction ("rate this response 1-10") produces noisy, low-agreement scores; an explicit rubric (specific criteria, each with concrete examples of what a low/medium/high score looks like) produces far more consistent, defensible judgments. Before trusting an LLM-judge's scores at scale, check its agreement against human judgments on a sample — real agreement, computed, for two rubric styles on the same items:

An unvalidated judge is exactly the "untrustworthy measurement" problem Evaluation Fundamentals warns about, just one layer removed.

Judge Reliability Is Not Judge Validity

A judge can pass the validation check above and still be wrong in a way that check can't catch. Reliability means a judge gives consistent scores — run it twice on the same input, get the same answer. Validity means those consistent scores actually track the thing you care about. A judge can have both at once, or — the failure mode that matters here — have one without the other.

This distinction stopped being theoretical in 2026: a large-scale study found production LLM-judges with test-retest reliability above 0.95 (near-perfect internal consistency) whose rankings of the same candidate responses shifted by up to 14 positions when the same underlying quality was re-evaluated through a different but equally reasonable rubric framing. The judge wasn't noisy — it was consistently applying a biased standard. A reliability check alone (run it twice, confirm the scores match) would have passed this judge every time, because reliability only asks "does the judge agree with itself," never "does the judge agree with the truth."

What this changes in practice: validating a judge against raw percent-agreement with human labels isn't enough either, because percent agreement doesn't correct for chance — a judge issuing the same score on every item can score high percent-agreement against a lopsided human-label distribution without discriminating between good and bad outputs at all. Use Cohen's κ (kappa) instead, which subtracts out the agreement rate expected from chance:

$\kappa = \frac{p_o - p_e}{1 - p_e}$

where $p_o$ is the observed agreement rate between judge and human, and $p_e$ is the agreement rate two random raters would produce by chance given the label distribution. A judge scoring 90% raw agreement against a dataset where 90% of items are labeled "pass" could have $\kappa$ near zero — it may just be predicting the majority class, not actually discriminating quality. As a rule of thumb, treat $\kappa$ below 0.4 as weak agreement worth real scrutiny before trusting the judge at scale, regardless of how good the raw percentage looks.

Error Analysis: Finding Failure Modes Before You Can Fix Them

Every technique above helps you measure quality once you already know what to measure. Error analysis is the earlier, more foundational step: systematically reading through real failures to discover what actually goes wrong, before you can write a rubric, a metric, or an eval set that targets it. Skipping straight to metrics without this step is a common, expensive mistake — it's easy to build a precise measurement of the wrong thing.

The concrete method, borrowed from qualitative research and now standard practice in eval-driven development:

1. Open coding: pull a real sample of failures (20-50 is often enough to start) — actual production traces, not synthetic test cases — and, without any predetermined category list, write a short free-text note on why each one failed, in your own words, as specifically as possible ("retrieved the right document but cited the wrong section," not "bad answer").
2. Axial coding: once you have notes across the whole sample, group them into a small number of named failure categories that emerge from the notes themselves — not categories you guessed in advance. A category should be specific enough to point at a fix ("tool called with a malformed date argument" is actionable; "agent made a mistake" is not).
3. Count and prioritize: tally how often each category occurs across the sample. This turns "the agent seems flaky sometimes" into "38% of failures are one specific, fixable category" — a completely different, and far more actionable, problem statement.
4. Build the eval set from what you found: each real category becomes a targeted eval case — write held-out examples that specifically exercise the failure mode you just named, so a fix can be verified, and a regression can be caught, on the exact thing that was actually breaking, not a generic proxy for it.

RAG Evaluation

Covered in full in RAG — Evaluating RAG: faithfulness/groundedness, answer relevance, context precision/recall, and MRR/NDCG for retrieval ranking quality. The key framing worth repeating here: a RAG system's failure could be in retrieval (wrong or missing chunks), ranking (relevant chunks retrieved but not surfaced near the top), or generation (right context, but the model still didn't use it faithfully) — evaluating each stage separately, not just end-to-end answer quality, is what actually tells you which stage to fix.

Agent Evaluation

Evaluating an agent is structurally different from evaluating a single LLM call — the object being evaluated is a sequence of decisions and actions, not one output, and "did it get the right final answer" can be true even when the path there was unreliable. Click through a real toy trajectory below and watch task success and trajectory quality diverge:

• Task success rate: did the agent actually accomplish the end goal — the most important single metric, but a coarse one that says nothing about how it got there or how reliably it would succeed on a slightly different version of the same task.
• Trajectory analysis: evaluating the sequence of actions the agent took, not just the final outcome — did it use tools in a sensible order, did it recover gracefully from a failed tool call, did it waste steps on unproductive exploration, exactly like step 2 above? A trajectory can be evaluated by a human, by an LLM-judge given the full action log (extending the LLM-as-judge approach above to a sequence rather than a single output), or against a reference trajectory when tasks are constrained enough to have one.
• Tool-call accuracy: specifically, did the agent call the right tool, with correctly-formed and correctly-valued arguments, at each step — a narrower, more mechanical metric than overall trajectory quality, useful for isolating whether failures come from reasoning (choosing the wrong approach) or execution (choosing the right approach but calling the tool incorrectly).
• Efficiency metrics: number of steps/tool calls, total tokens used, and wall-clock latency to complete the task — two agents with identical success rates can have very different production cost/latency profiles, and efficiency is invisible if success rate is the only metric tracked (see AI Cost Engineering for the broader cost-optimization context this feeds into).
• Why this had no dedicated treatment before: single-turn LLM evaluation and RAG evaluation both inherited well-established metric traditions (classification metrics, information-retrieval metrics) to build on. Agent evaluation doesn't have an equally mature inherited tradition — it's a genuinely newer problem, closer to evaluating a multi-step plan or a reinforcement learning policy's rollout than to evaluating a single prediction, and the field's evaluation practice here is still actively maturing.

Regression Testing for Models and Prompts

Extends ML Workflow Fundamentals' classical regression-testing idea (does a change degrade performance against a fixed benchmark) to prompts and models specifically: every prompt change, model version bump, or fine-tune should run against the golden dataset (see Evaluation Fundamentals) before shipping, with a defined threshold for what counts as an acceptable regression — the same automated gate CI/CD & ML CI/CD describes for classical ML, applied to generative and agentic systems where "did this get worse" is measured by the evaluation methods on this page rather than a single accuracy number.

Code: A Real Judge-Bias Smoke Test

The same check the position/verbosity diagrams above simulate, runnable against a real judge model:

import numpy as np
from scipy.stats import pearsonr

# Position bias: run every pairwise comparison in both orders, flag flips.
def check_position_bias(judge_fn, pairs):
    flips = sum(1 for a, b in pairs if judge_fn(a, b) != judge_fn(b, a))
    return flips / len(pairs)  # real flip rate -- should be ~0 for an unbiased judge

# Verbosity bias: correlate judge score against response length directly.
lengths = [len(r.split()) for r in responses]
scores = [judge_fn(r) for r in responses]
r, p_value = pearsonr(lengths, scores)
if abs(r) > 0.3 and p_value < 0.01:
    print(f"Judge score correlates with length (r={r:.2f}) -- likely verbosity bias.")

Next: Human & Adversarial Evaluation — where automated evaluation isn't enough on its own.