A stakeholder points to a chart showing users who use feature X have much higher retention and concludes that feature X causes retention. What's wrong with that reasoning, and how would you push back constructively?
technical-conceptual · Junior level · data-ml
What the interviewer is really asking
Probes whether the candidate understands the correlation-versus-causation distinction, can name confounding, and can suggest how to actually establish causation.
What to say
- Explain that correlation doesn't imply causation: the two move together, but a third factor could drive both.
- Name a plausible confounder — for example, already-engaged users both adopt feature X and retain better, so the feature isn't necessarily the cause.
- Point to ways to test for causation: a randomized A/B test, or if that's not possible, a quasi-experimental or matched-comparison approach.
What to avoid
- Don't accept the causal claim just because the correlation is strong or the chart looks convincing.
- Don't dismiss the observation entirely — it's a useful signal worth investigating, just not proof.
- Don't propose acting on it as causal without any controlled experiment or attempt to control for confounders.
Example answers
Strong: The chart shows feature X and retention are correlated, but that alone doesn't mean X causes retention — a confounder could drive both. The likeliest one here is that already-engaged users are the ones who discover and use feature X, and they'd have retained anyway. I wouldn't dismiss the signal, but to actually claim causation I'd want a randomized A/B test that exposes some users to the feature and not others, or if we can't randomize, a matched comparison that controls for prior engagement.
Weak: The correlation is really strong and the chart is clear, so feature X is obviously driving retention. I'd recommend we push everyone toward feature X.