Human & Adversarial Evaluation

Automated metrics and LLM-as-judge (see LLM, RAG & Agent Evaluation) scale well but aren't sufficient alone — humans remain the ground truth for genuinely subjective quality judgments, and a system that's never been actively attacked hasn't really been evaluated for robustness at all.

Intuition: Two Different Kinds of "Not Enough Signal"

Human evaluation and adversarial evaluation fail in two structurally different, both-real ways worth telling apart: human evaluation can look decisive while actually being statistical noise (a rater-agreement or sample-size problem — the first half of this page, both demonstrated with real numbers below), while adversarial robustness gaps can hide entirely, invisible until someone specifically goes looking for them (the second half). Neither is fixed by "collect more of the same" — one needs a rigor check, the other needs an entirely different kind of input.

Human Evaluation Methodology

Getting a trustworthy number out of human evaluation requires the same rigor as any other measurement instrument.

Rater Agreement

If two human raters given the same output disagree often, either the task is genuinely ambiguous or the rating instructions are — measured via inter-rater reliability statistics (e.g. Cohen's kappa for two raters, which corrects for the agreement expected by pure chance). Adjust the confusion matrix below and watch raw agreement and real kappa diverge:

both rate GOOD55

rater 1 GOOD, rater 2 BAD15

rater 1 BAD, rater 2 GOOD10

both rate BAD20

75.0%

raw agreement

0.432

Cohen's κ

Low κ invalidates the resulting scores regardless of how many raters were used -- it means the "measurement" isn't actually measuring something raters share a consistent understanding of.

Raw agreement = 75.0% -- looks decent on its own. But chance agreement alone (if both raters just guessed independently at their real marginal rates) would already be 56.0%. Real Cohen's κ = (0.750 − 0.560) / (1 − 0.560) = 0.432 -- "moderate" agreement, a very different verdict than the raw percentage alone suggested.

Low agreement invalidates the resulting scores regardless of how many raters were used, since it means the "measurement" isn't actually measuring something raters share a consistent understanding of.

Sample Size and Statistical Power

A human evaluation run on 20 examples can look decisive and still be statistical noise — the same statistical power consideration from hypothesis testing applies directly. A real power calculation, computable before running the study:

real effect size to detect5.0 pts

sample size (n per group)20

Smaller real effects need dramatically more samples to detect reliably -- exactly why "20 examples looked decisive" is a real statistical trap, not just intuition.

Real win-rate difference to detect: 5 points (50% vs 55%). At n=20 raters per side, real statistical power = 5.0% -- the probability this study actually detects the difference IF it's real. A study "not showing a clear winner" at low power is exactly as likely to mean "underpowered" as "no real difference," and this number is computable BEFORE running the study, not after.

Know the minimum sample size needed to detect a difference of the size you actually care about, before running the study, not after it "didn't show a clear winner."

Rubric Design, and Rater Expertise

The same lesson as LLM-judge rubrics (see LLM, RAG & Agent Evaluation) applies even more strongly to human raters — vague instructions ("is this good?") produce noisy, inconsistent ratings; specific, example-anchored criteria produce reliable ones. Domain-expert raters (a doctor rating medical text, a lawyer rating legal text) are essential for domain-specific quality judgments a general crowdworker can't reliably make — but expert time is expensive and scarce, which is exactly why LLM-as-judge, once validated against expert human judgments on a sample, is used to scale evaluation beyond what human review alone could afford.

Pairwise and Preference Evaluation

Rather than asking a rater (human or LLM) to assign an absolute score to one output, pairwise evaluation shows two outputs side by side and asks which is better — generally produces more reliable, consistent judgments than absolute scoring, because "which of these two is better" is a much easier and more stable judgment to make than "rate this a 7 or an 8," especially for open-ended generation where there's no obvious absolute scale.

Win rate: the fraction of pairwise comparisons a given model/system wins against a baseline — the standard way pairwise judgments get aggregated into a single reportable number.
Elo/Bradley-Terry ranking: when comparing more than two systems, pairwise win/loss results can be aggregated into a single relative-skill ranking using the same rating-system math chess (Elo) or other competitive-ranking systems use. Run a real Elo simulation below — hidden true skill, real match outcomes, real rating updates:

Model A1200

Model B1200

Model C1200

The exact rating-system math chess (Elo) uses, and directly what LMSYS's Chatbot Arena runs at scale on real human pairwise preferences.

0 real pairwise matches played (each with a real, hidden win probability derived from true skill, and a real Elo update after every result: rating += K·(actual − expected)). Current Elo order matches the real hidden skill order (Model A > Model B > Model C) -- Elo needs enough real match volume to converge, exactly like the statistical-power point above applied to ranking instead of A/B testing.

LMSYS's Chatbot Arena is the best-known example of this applied to LLMs at scale, using large-scale human pairwise preferences to produce a leaderboard. This is also directly the mechanism behind preference data collection for RLHF and DPO — pairwise preference judgments aren't just an evaluation technique, they're the actual training signal for an entire alignment methodology, which is why getting pairwise evaluation methodology right matters for training pipelines, not only for reporting benchmark numbers.

Adversarial Evaluation

Testing a system specifically against inputs designed to break it, rather than the representative-but-cooperative inputs a golden dataset (see Evaluation Fundamentals) typically contains. Real accuracy degradation under increasing perturbation strength, standard vs. adversarially-trained:

perturbation strength0.4

⬤ standard model⬤ adversarially evaluated/trained model

At perturbation strength 0.40 (paraphrasing / distracting text / edge-case formatting, applied to an otherwise-identical input): standard model accuracy real-drops from 92% clean to 52.6%; an adversarially-trained model drops only to 80.0%. If the output changes substantially under a meaning-preserving perturbation, that's a real robustness gap, not a benchmark artifact.

Why standard evaluation misses this: a golden dataset built from realistic production inputs tells you how the system performs on the inputs it's likely to see — it says very little about how it behaves under a deliberately crafted worst case, and those worst cases are exactly what a motivated user (or attacker) will find first.
Constructing adversarial examples: small, meaning-preserving perturbations to a normal input (paraphrasing, injecting distracting/irrelevant text, edge-case formatting) that a robust system should handle identically to the unperturbed version — if the output changes substantially, that's a real robustness gap, not a benchmark artifact, exactly as shown above.
Automated adversarial generation: using another LLM specifically prompted to generate inputs likely to cause failures (incorrect answers, policy violations, crashes) — the same "use a model to stress-test a model" idea as LLM-as-judge, applied to generating hard inputs rather than judging outputs.

Red Teaming

A structured, adversarial security/safety evaluation practice — a dedicated team (or process) actively tries to make a system fail in specific, defined ways (produce harmful content, leak training data, be manipulated via prompt injection, bypass safety guidelines) before real-world adversaries get the chance to find the same weaknesses:

Structured red teaming: predefined categories of attack (the OWASP LLM Top 10 categories are a common starting taxonomy) systematically tested against, rather than unstructured "try to break it" exploration — ensures coverage rather than depending on whatever the red team happens to think of first.
Automated red teaming: using an LLM to generate large volumes of adversarial attack attempts at a scale human red-teamers can't match, then having humans review and triage what the automated process surfaces — a force-multiplier for human red-teaming effort, not a full replacement, since novel, creative attack strategies still tend to come from human red-teamers first.
The output is action, not just a report: red teaming that doesn't feed directly back into the golden dataset (as new hard/adversarial examples), the regression test suite, and model/prompt fixes is mostly wasted effort — the entire point is closing the loop back into Continuous Evaluation, not producing a one-time audit document.

Code: A Real Power Calculation Before Running a Study

The exact formula the power diagram above computes live, runnable before committing budget to a human evaluation:

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

effect_size = proportion_effectsize(0.55, 0.50)  # the real difference worth detecting
analysis = NormalIndPower()
required_n = analysis.solve_power(effect_size=effect_size, alpha=0.05, power=0.8)
print(f"Need {required_n:.0f} raters per group for 80% power -- know this BEFORE running the study.")

AI Evaluation section complete. Next: ML System Design — where these evaluation choices become part of a production system's design from the start, not bolted on afterward.