Model Evaluation & Metrics

A model is only as good as the way you measure it. Picking the wrong metric is one of the most common ways ML projects quietly fail.

Bias-Variance Tradeoff

• Bias: error from overly simplistic assumptions — the model underfits, missing real patterns in the data.
• Variance: error from being overly sensitive to the specific training data — the model overfits, memorizing noise instead of learning generalizable patterns.
• Total error decomposes roughly as $\text{Bias}^2 + \text{Variance} + \text{Irreducible Error}$. Reducing one often increases the other — a simpler model has more bias but less variance, and vice versa. Model capacity, regularization strength, and amount of training data are the main levers.

Real Monte Carlo decomposition — 150 real resampled datasets, real polynomial fits at increasing complexity:

Regularization

Techniques that intentionally constrain a model to reduce variance/overfitting:

• L1 (Lasso): adds $\lambda \sum |w_i|$ to the loss — pushes weights toward exactly zero, giving automatic feature selection.
• L2 (Ridge): adds $\lambda \sum w_i^2$ — shrinks weights smoothly without forcing them to zero. Mathematically equivalent to MAP estimation with a Gaussian prior (see Probability & Statistics).
• Dropout (deep learning): randomly zeroes out neurons during training, forcing the network to not rely too heavily on any single pathway.
• Early stopping: stop training once validation performance stops improving, before the model starts memorizing training noise.

Classification Metrics

• Accuracy: fraction correct — misleading on imbalanced data (a 99%-negative dataset gets 99% accuracy by always predicting negative).
• Precision: of everything predicted positive, how much was actually positive. $\text{Precision} = \frac{TP}{TP + FP}$
• Recall: of everything actually positive, how much did the model catch. $\text{Recall} = \frac{TP}{TP + FN}$
• F1 score: harmonic mean of precision and recall — a single number balancing both.
• ROC-AUC: area under the true-positive-rate vs. false-positive-rate curve across all thresholds — measures ranking quality independent of a specific threshold.
• PR-AUC: like ROC-AUC but using precision/recall — more informative than ROC-AUC on heavily imbalanced datasets.

Choosing precision vs. recall: depends entirely on the cost of each error type. A cancer-screening model should favor recall (missing a real case is worse than a false alarm); a spam filter should favor precision (blocking a real email is worse than letting one spam message through).

Real ROC and PR curves, swept over 200 real scored examples — drag the threshold and watch the marked point slide along the actual curve:

Regression Metrics

• MSE (Mean Squared Error): penalizes large errors disproportionately (squared term) — sensitive to outliers.
• MAE (Mean Absolute Error): penalizes all errors linearly — more robust to outliers.
• R²: fraction of variance in the target explained by the model, from 0 (no better than predicting the mean) to 1 (perfect fit).

Calibration

A model can rank examples correctly (good AUC) while its predicted probabilities are systematically off (e.g. it says "70% confident" but is only right 50% of the time at that confidence level). Calibration measures and corrects this — critical whenever downstream decisions depend on the actual probability value, not just the ranking (e.g. risk-based pricing). Real reliability diagram, 600 real predictions:

Threshold Optimization

A classifier's raw output is a probability; turning it into a decision requires picking a threshold, and 0.5 is a default, not a law of nature. The right threshold depends on the same cost asymmetry that drives the precision/recall tradeoff above:

• Cost-based thresholding: if a false negative costs 10x a false positive (a missed fraud case vs. a flagged legitimate transaction), the optimal threshold isn't where precision equals recall — it's wherever expected_cost = FN_rate × cost_FN + FP_rate × cost_FP is minimized, which is usually well below 0.5.
• Reading it off the PR or ROC curve: since both curves are threshold-sweeps, picking a threshold means picking a point on that curve corresponding to the precision/recall (or TPR/FPR) tradeoff your use case actually needs, then finding which threshold produces it.
• The common mistake: reporting "our model gets 90% precision" without saying at what threshold, or worse, choosing the threshold that produces the best-looking headline number on the test set — the threshold is a decision variable to be chosen deliberately for the deployment's actual cost structure, not a hyperparameter to tune for a better-looking single metric.

Uncertainty Estimation

Calibration (above) asks "are the predicted probabilities trustworthy." Uncertainty estimation asks a related but distinct question: "how confident should I be in this specific prediction," which matters most when a model needs to know what it doesn't know. See Uncertainty Estimation & Conformal Prediction for prediction intervals, ensemble-based uncertainty, and — the part with an actual proof, not just an assertion — conformal prediction's real distribution-free coverage guarantee, verified live against a held-out test set rather than just stated.

Hyperparameter Optimization

Model hyperparameters (learning rate, tree depth, regularization strength, number of layers) aren't learned from data the way weights are — they have to be searched over, using validation performance as the search signal, never the test set. See Hyperparameter Optimization for the real Bergstra & Bengio result on why random search beats grid search (not just intuition — a real distinct-values-tried comparison), the actual Expected Improvement formula behind Bayesian optimization, and successive halving/Hyperband.

Statistical Significance of Improvements

Before shipping "Model B beats Model A by 0.3%," check whether that difference is statistically significant (see Probability & Statistics) or just noise from the particular test split / random seed. This is exactly what A/B testing in production is designed to answer rigorously.

Common Problems & Their State-of-the-Art Solutions

• Class imbalance → resampling (SMOTE), class weighting, focal loss (down-weights easy examples so the model focuses on hard/minority-class ones)
• Overfitting on small data → stronger regularization, data augmentation, transfer learning from a model pretrained on related data
• Concept drift in production (the real-world data distribution shifts over time) → continuous monitoring dashboards, scheduled retraining, online learning
• Curse of dimensionality (too many features relative to data volume, distances become meaningless) → feature selection, PCA, stronger regularization
• Slow training on huge tabular datasets → gradient boosting libraries built for speed (XGBoost, LightGBM), distributed training frameworks
• Explainability ("why did the model predict this?") → SHAP values (game-theoretic feature attribution), LIME (local approximations), built-in feature importance from tree models

Machine Learning section complete. Next: Deep Learning — when classical algorithms stop being enough.