How would you detect that the data coming into a production model has drifted away from what it was trained on, using a concrete statistical method?

technical-conceptual · Junior level · data-ml

What the interviewer is really asking

Assess whether the candidate can move beyond the abstract idea of drift to a concrete detection mechanism — comparing a recent window of feature values against a training/reference baseline with a metric like PSI or a KS test, and reasoning about thresholds.

What to say

• Frame it as comparing distributions: hold a reference (training or a stable recent window) and compare each feature's recent distribution against it on a schedule.
• Name a concrete method and its meaning: Population Stability Index per feature (roughly under 0.1 stable, 0.1-0.2 moderate, above 0.2 significant), or a KS test for continuous features and chi-square for categorical.
• Note that thresholds are starting points to tune per use case and risk, and that you monitor per feature so you can tell which input shifted.

What to avoid

• Defining drift correctly but offering no actual method to measure it, leaving it at 'the data changes over time.'
• Treating a single threshold as universal truth rather than a tunable starting point per model and feature.
• Comparing only aggregate means, missing that a distribution can shift in shape while the mean stays the same.

Example answers

Strong: I'd keep a reference distribution from training and, on a schedule, compare a recent window of production data against it per feature. A common metric is PSI: under about 0.1 is stable, 0.1 to 0.2 is moderate, above 0.2 is a significant shift worth investigating. For continuous features you can also use a KS test and chi-square for categoricals. Doing it per feature tells me which input drifted, not just that something did, and I'd treat the thresholds as starting points to tune.

Weak: Drift is when the data changes over time so the model gets worse. I'd watch the data in production and if it looks different from training data we'd retrain the model on the newer data.