Once a model is deployed, how do you know when it's degrading, and what's the difference between data drift and concept drift?

technical-conceptual · Junior level · data-ml

What the interviewer is really asking

Assess whether the candidate understands that production models decay and can distinguish the two drift types, name monitoring signals, and connect them to a retraining response.

What to say

• Explain that models decay because the world changes after training, so you monitor production: track the live performance metric when ground truth eventually arrives, and watch input feature distributions and prediction distributions when labels are delayed.
• Distinguish the two: data drift is a shift in the input distribution (the features the model sees change) while the input-output relationship stays the same; concept drift is a change in the relationship between inputs and the target itself, so the same inputs now map to a different correct answer.
• Connect to action: set alerts on drift and on the performance metric, validate that drift actually hurts accuracy before reacting, and trigger retraining (ideally an automated pipeline) on fresh, representative data — keeping data and model versions so you can roll back.

What to avoid

• Treating deployment as the finish line and assuming a model that was accurate at launch stays accurate forever.
• Confusing data drift and concept drift, or assuming any drift automatically requires retraining without first checking whether performance actually dropped.
• Relying only on the offline test metric from training and never measuring anything about live traffic.

Example answers

Strong: For a demand-forecasting model I tracked prediction error weekly as actuals landed, plus the input feature distributions since labels lagged. When a new product line shifted the feature mix (data drift) I caught it on the input monitor before error spiked, validated the impact, and retrained on the newer data.

Weak: Once it's deployed and passed the test set, monitoring isn't really necessary — the model is done.