Monitoring & Drift Detection

A deployed model doesn't stay accurate by default — the world it's making predictions about keeps changing, and unlike a normal software bug, a degrading model usually fails silently: it keeps returning confident, well-formed predictions that are just quietly getting worse.

Three Layers of Monitoring

• Infrastructure monitoring: is the serving system itself healthy — CPU/GPU utilization, memory, disk, request latency, error rate, uptime? This is standard software monitoring, nothing ML-specific about it.
• Application monitoring: is the API behaving correctly — request/response schemas valid, correct status codes, no unhandled exceptions? Also not ML-specific.
• ML-specific monitoring: is the model still good — are its predictions still accurate, and does the data it's seeing still resemble what it was trained on? This is the layer that has no equivalent in regular software monitoring, and the one most systems under-invest in.

A system can be 100% "up" by every infrastructure and application metric while the model underneath has silently degraded to near-random — this is why ML-specific monitoring is its own discipline, not a subset of the first two.

Data Drift: The Math

Data drift is a change in the distribution of the model's input features between training time and now. Several statistical distances quantify "how different are these two distributions":

• PSI (Population Stability Index): bins a feature's values and compares the proportion of data in each bin between a reference (training) distribution and a current distribution — PSI = Σ (current_% − reference_%) × ln(current_% / reference_%) summed over bins. PSI < 0.1 is typically considered stable, 0.1–0.25 moderate drift, > 0.25 significant drift — one of the most widely used thresholds in industry monitoring.

• KL divergence (Kullback-Leibler): measures how one probability distribution diverges from a reference distribution — asymmetric (KL(P‖Q) ≠ KL(Q‖P)), which matters when deciding which distribution is "reference" and which is "current."
• JS divergence (Jensen-Shannon): a symmetric, smoothed version of KL divergence — the fix when direction shouldn't matter, computed as the average of KL divergences from each distribution to their mixture.

• KS test (Kolmogorov-Smirnov): a statistical hypothesis test comparing two continuous distributions' empirical CDFs — gives a p-value for "these two samples plausibly come from the same distribution," rather than just a distance score.
• Chi-square test: the categorical-feature analog of the KS test — compares observed vs. expected frequencies across categories.

• Wasserstein distance (Earth Mover's Distance): the minimum "work" needed to transform one distribution into another — more robust to small distributional shifts than KL/JS divergence and doesn't blow up when distributions have non-overlapping support (where KL divergence is undefined).

Data Drift vs. Concept Drift

• Data drift: the input distribution changes (e.g. customer demographics shift), but the true relationship between input and output stays the same.
• Concept drift: the relationship between input and output itself changes (e.g. what counted as "fraud" behavior changes as fraudsters adapt) — the same input now genuinely warrants a different prediction.

This distinction matters for the fix: data drift can sometimes be handled by retraining on more recent data with the same labeling logic; concept drift means the model's target definition has moved, and retraining on stale labels won't help — the labeling/evaluation process itself needs revisiting.

A Real, Proven Example: PSI + KS-Test, Actually Validated

Everything above is the theory — real, but unverified until it's actually run against known ground truth. ml-drift-monitor implements the exact PSI formula from above, plus the KS-test, around a real XGBoost fraud model (trained on the real 284,807-transaction ULB credit-card-fraud dataset, AP 0.869 / ROC-AUC 0.975 on held-out data) — and then does the part most drift-detection writeups skip: proves the detector itself is correct, rather than just running it once and trusting the output.

The proof. A static historical dataset never contains real future drift by definition — validating a detector needs a known, controlled amount of drift to check the detector's response against. Two things get measured against that known ground truth:

• False-positive rate, empirically measured, not assumed: 50 independent 5,000-row samples of real, genuinely non-drifted data, scored across 29 features — 1,450 independent KS tests. 80 flagged, an empirical rate of 5.52% against KS's nominal 5% alpha. That's what "well-calibrated" actually looks like measured, not just claimed.
• Detection power as a function of injected severity: sweeping a synthetic feature shift from 0 to a full standard deviation produces a PSI that increases monotonically on the shifted feature (0.0004 → 1.40) — and shifting exactly 8 of 29 features flags exactly those 8, with zero false spillover onto the other 21.

A finding worth knowing before you rely on either metric: with real-world sample sizes (~57,000 rows here), the KS-test's p-value becomes extremely significant (p < 1e-79) at even a 10% injected shift — statistical significance stops meaning practical significance once N is large enough. PSI, which measures the magnitude of a distribution shift rather than the confidence that any shift occurred at all, correctly stays in "none"/"moderate" territory at that same severity. This is exactly why PSI is the practical-magnitude gate in production monitoring and a p-value-based test is at most a secondary, corroborating signal — not the reverse.

A real bug this caught: the project's first drift-verdict logic flagged "any drift detected" whenever either PSI was significant or the KS-test fired. Given the measured 5.52% KS false-positive rate across ~29 monitored features, that meant the headline verdict was True on almost every real comparison, drifted or not — caught by actually reading the dashboard's rendered output rather than trusting a passing test suite, and fixed by making PSI (practical magnitude) the verdict signal, with KS demoted to a secondary per-feature diagnostic. The exact kind of gap between "the code runs" and "the code is right" that only shows up when you check what a detector actually outputs against a case you already know the answer to.

Tools

• Evidently: open-source, generates drift and data-quality reports/dashboards comparing a reference and current dataset — the most common starting point.

from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=training_df, current_data=production_df)
report.save_html("drift_report.html")   # per-feature PSI/KS-test results, and an overall drift verdict

• WhyLabs: a managed observability platform for ML, focused on data and model quality monitoring at scale without shipping raw data to a third party (it profiles data locally, sends only statistical summaries).
• Arize: a managed ML observability platform with strong support for root-causing performance drops back to specific data segments (which slice of traffic is actually driving the drift).

Next: Observability — the general logging/metrics/tracing infrastructure that drift detection and everything else above is built on top of.