How do you decide when an A/B test result is trustworthy enough to ship, especially when the result is close to flat?

role-specific · Senior level · product-management

What the interviewer is really asking

Assesses experimentation rigor — whether the candidate understands power, significance, peeking, and the difference between 'not significant' and 'no effect' rather than treating any positive number as a win.

What to say

Check the experiment was designed right before reading it: was it powered for the effect size you cared about, did it run a full business cycle, and was the sample assigned cleanly with no peeking-driven early stop?
Distinguish 'statistically flat' from 'inconclusive' — a tight confidence interval around zero is a real null, while a wide one means you were underpowered and shouldn't conclude either way.
Weigh the decision cost, not just the p-value: for a low-risk, reversible change a flat-but-directional result plus qualitative signal can justify shipping, whereas a costly or risky change needs a clearer read.

What to avoid

Don't ship on a positive number that isn't significant just because it's directionally up; that's reading noise as signal.
Don't stop the test the moment it crosses significance (peeking), which dramatically inflates false positives.
Don't treat 'no significant difference' as proof the change does nothing without checking whether the test had the power to detect the effect you cared about.

Example answers

Strong: When a redesign test came back +0.8% but not significant, I checked the math first: we were underpowered for an effect that small, so it was inconclusive, not flat. The change was low-risk and reversible and qualitative feedback was positive, so I shipped behind a flag and kept measuring with a longer holdback rather than declaring victory. Had it been a pricing change, I'd have demanded a clean, powered, significant read before touching it.

Weak: It was up 0.8%, so even though it wasn't quite significant the direction was positive, so I'd go ahead and ship it.

Want questions matched to your role? Paste a job title, job description, or CV and get a personalized set, or go Pro to unlock the full bank.