A model performed strongly in offline evaluation but its accuracy dropped sharply once deployed. You suspect a feature-engineering issue. How do you investigate whether leakage is the cause, and how do you prevent it going forward?
technical-conceptual · Senior level · data-ml
What the interviewer is really asking
Assesses the candidate's ability to diagnose train/serve leakage — features built from information unavailable at prediction time, preprocessing fit on the whole dataset, or temporal violations — and to put guardrails in place, which is a senior responsibility.
What to say
- Anchor the diagnosis on the principle: the offline-vs-online gap is the classic leakage signature — a feature used at training that won't be available (or has different values) at prediction time. Audit each feature for whether it could only be computed using future or post-outcome information.
- Check the three common sources: target/future leakage (a feature derived from the label or events after the prediction point), preprocessing fit on the full dataset before splitting (scaling/encoding/imputation that saw the test rows), and temporal violation (random split on time-ordered data instead of a forward-chaining split).
- Put in guardrails: split first then fit all transforms inside the training fold only, reconstruct features strictly as-of the prediction timestamp using a point-in-time feature store, and add a training/serving skew monitor that compares feature distributions and the offline metric to live performance.
What to avoid
- Blaming the model or hyperparameters first when the offline/online gap points squarely at leakage or train/serve skew.
- Re-running with a different algorithm hoping the gap closes, instead of auditing how each feature is computed and when.
- Fixing it once by hand without adding a monitor or a point-in-time guarantee, so the next feature reintroduces the same bug.
Example answers
Strong: A big offline-to-online drop is the textbook leakage signature, so I'd audit features for information not available at prediction time. I'd check three things: a feature derived from the label or from events after the decision point; preprocessing — scaler, encoder, imputer — fit on the full dataset before splitting so it saw test rows; and a random split on time-ordered data instead of forward-chaining. The fix is to split first and fit every transform inside the training fold, reconstruct features as-of the prediction timestamp via a point-in-time feature store, and add a train/serve skew monitor so the next leaky feature trips an alarm instead of shipping.
Weak: Offline and online differ all the time, so I'd retrain with more data and maybe a stronger model, and tune the threshold until online accuracy comes back up. If that doesn't work I'd try a different architecture.