AI Evaluation — Roadmap

1. Evaluation Fundamentals

• Traditional ML metrics (see Model Evaluation & Metrics)
• Benchmark design: construct validity, contamination, saturation
• Golden datasets: building, maintaining, and knowing when they're stale
• Continuous evaluation in production

2. LLM, RAG & Agent Evaluation

• LLM evaluation and LLM-as-a-judge, in depth
• RAG evaluation (see RAG — Evaluating RAG)
• Agent evaluation: trajectory analysis, task success rate, tool-call accuracy
• Regression testing for models and prompts

3. Human & Adversarial Evaluation

• Human evaluation methodology: rater agreement, sample size, rubric design
• Pairwise and preference evaluation
• Adversarial evaluation
• Red teaming

Next: ML System Design — where evaluation choices become part of a production system's design from the start.