Uncertainty Estimation & Conformal Prediction

Model Evaluation & Metrics covers calibration — "are the predicted probabilities trustworthy." This page covers a related but distinct question: "how confident should I be in this specific prediction," which matters most exactly when a model needs to know what it doesn't know.

A weather forecaster saying "70% chance of rain" is making a calibration claim: across every day they said 70%, it should actually rain about 70% of the time. Conformal prediction answers a different question: instead of a single number's trustworthiness, it gives you a range — "the temperature tomorrow will be between 60°F and 75°F" — built specifically so that range genuinely contains the true answer at least as often as promised, no matter how good or bad the underlying forecasting model actually is. That last part is the surprising bit: the guarantee doesn't require the model to be any good, only that you calibrate the range using real held-out examples first.

Prediction Intervals and Ensembles

Two lighter-weight approaches, worth knowing before the main event below:

Prediction intervals via quantile regression: instead of a single predicted value, train separate models (or output heads) for different quantiles of the target (10th, 50th, 90th percentile) — giving a real range directly, rather than a point estimate plus an assumed error distribution.
Ensemble-based uncertainty: train several models (or use dropout at inference time — "Monte Carlo dropout") and treat disagreement across them as the uncertainty signal — high variance across members on a given input suggests the model is extrapolating into territory it hasn't seen reliable examples of.

Both are useful, and both share the same weakness: neither comes with a guarantee. A quantile model's 90% interval might actually contain the truth 70% of the time if the model is wrong about the error distribution; an ensemble's disagreement is a heuristic signal, not a calibrated probability.

Conformal Prediction: the Part With an Actual Guarantee

This is the part of "uncertainty estimation" that most explanations assert rather than prove: conformal prediction produces intervals with a real, distribution-free coverage guarantee — true regardless of what model produced the predictions, and regardless of whether the underlying data is Gaussian, heteroskedastic, or anything else. Angelopoulos & Bates' "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification" states the real recipe as four steps:

Start from any fitted model with some heuristic notion of uncertainty — even a bad one.
Define a score function $s(x, y) \in \mathbb{R}$ where larger scores mean worse agreement between $x$ and $y$ (for regression, the simplest real choice is the absolute residual, $s(x,y) = |y - \hat f(x)|$ ).
On a fresh calibration set of $n$ examples never used for training, compute $\hat q$ as the $\lceil (n+1)(1-\alpha) \rceil / n$ empirical quantile of the calibration scores $s_1, \dots, s_n$ — a small but load-bearing finite-sample correction on top of the "ordinary" $1-\alpha$ quantile.
Build the prediction set for a new point: $C(x_{\text{test}}) = \{y : s(x_{\text{test}}, y) \le \hat q\}$ — for the absolute-residual score, this is exactly the interval $[\hat f(x) - \hat q,\ \hat f(x) + \hat q]$ .

Theorem 1 (Vovk, Gammerman & Saunders, as stated in Angelopoulos & Bates): if the calibration points and the test point are exchangeable (i.i.d. is the standard special case), then

$P\big(Y_{\text{test}} \in C(X_{\text{test}})\big) \ge 1 - \alpha$

The proof sketch is genuinely simple, which is part of why the guarantee is so robust: exchangeability means the true test score is equally likely to land in any rank position among the $n+1$ scores (the $n$ calibration scores plus the test score itself) — so the probability the test score exceeds the $\lceil(n+1)(1-\alpha)\rceil$ -th largest of those $n+1$ values is, by construction, at most $\alpha$ . Nothing in that argument uses the quality of the underlying model or score function — a bad score function still gets valid coverage, it just produces wide, uninformative intervals rather than invalid ones. Coverage and informativeness are separate axes: conformal prediction only promises the first.

alpha (target miscoverage)0.1

● covered● missed

Target coverage: 90%. Measured on 150 fresh held-out test points (not the calibration set): 92.7% actually fell inside the interval -- q_hat = 0.661, computed as the real ceil((n+1)(1-alpha))/n quantile of 200 calibration residuals.

That diagram is the guarantee checked, not just stated: the calibration and test sets are two different random draws, $\hat q$ is computed only from calibration data, and the reported coverage is measured on the held-out test points afterward. Moving the target miscoverage slider changes $\hat q$ and the interval width, but the measured rate should track the target at every setting — including at $\alpha = 0.3$ , where the underlying regression fit is visibly worse in the noisy region, because the guarantee doesn't depend on the fit being good.

Check the Guarantee Yourself, in Real Python

The diagram animates the guarantee; this runs the actual four-step recipe from above — real synthetic data, a real (deliberately ordinary) least-squares fit, real calibration, and a real coverage measurement on fresh test data the calibration step never saw:

Run it yourself

import numpy as np

rng = np.random.default_rng(42)

def true_fn(x):
  return 2 * x + 5

n_train, n_cal, n_test = 500, 500, 2000

# Heteroskedastic noise (variance grows with x) -- on purpose. The
# guarantee below doesn't require the noise to be well-behaved.
x_train = rng.uniform(0, 10, n_train)
y_train = true_fn(x_train) + rng.normal(0, 1 + 0.5 * x_train, n_train)

# Step 1: any fitted model, even a plain OLS line -- ordinary least squares.
A = np.vstack([x_train, np.ones_like(x_train)]).T
a, b = np.linalg.lstsq(A, y_train, rcond=None)[0]

def predict(x):
  return a * x + b

# Step 2 + 3: score function = absolute residual, q_hat = the
# finite-sample-corrected empirical quantile of calibration scores.
x_cal = rng.uniform(0, 10, n_cal)
y_cal = true_fn(x_cal) + rng.normal(0, 1 + 0.5 * x_cal, n_cal)
scores = np.abs(y_cal - predict(x_cal))

alpha = 0.1  # target: 90% coverage
k = int(np.ceil((n_cal + 1) * (1 - alpha)))
q_hat = np.sort(scores)[k - 1]

# Step 4: build the interval, then check coverage on FRESH test data
# the calibration step never touched.
x_test = rng.uniform(0, 10, n_test)
y_test = true_fn(x_test) + rng.normal(0, 1 + 0.5 * x_test, n_test)
covered = np.abs(y_test - predict(x_test)) <= q_hat
coverage = covered.mean()

print(f"Fitted line: y = {a:.3f}x + {b:.3f}  (true: y = 2x + 5)")
print(f"q_hat (half-width of every interval): {q_hat:.3f}")
print(f"Target coverage: {(1 - alpha) * 100:.0f}%")
print(f"Measured coverage on {n_test} fresh test points: {coverage * 100:.2f}%")
print(f"Guarantee holds: {coverage >= 1 - alpha}")

# Try raising alpha to 0.3 (looser target) or changing the noise scale --
# the measured coverage keeps tracking the target, same as the diagram
# above, because the guarantee never depended on this being a good model.

Implement the Core Formula Yourself

Everything above leans on one function: turning $n$ calibration scores into $\hat q$ . The tests are hand-derivable directly from the formula in step 3 above — no simulation, no floating-point tolerance, just $k = \lceil (n+1)(1-\alpha) \rceil$ and the $k$ -th smallest score.

Implement it yourself

assert conformal_quantile([1, 2, 3, 4, 5, 6, 7, 8, 9], 0.2) == 8
assert conformal_quantile([10, 20, 30, 40], 0.5) == 30
assert conformal_quantile([42], 0.5) == 42
assert conformal_quantile(list(range(1, 20)), 0.05) == 19

Why This Matters in Production

A model that can say "I don't know" (a wide interval, or high ensemble disagreement) on out-of-distribution inputs enables routing those cases to a human reviewer or a fallback system, instead of returning a confident-looking wrong answer. Conformal prediction is increasingly used as a wrapper around black-box models — including LLMs, where a "score function" might be a self-reported confidence or an ensemble of sampled completions — specifically because the coverage guarantee holds no matter what's inside the box, which plain calibration or ensemble-disagreement heuristics can't promise.