You ran a standard A/B test on a two-sided marketplace feature and the treatment looked like a clear win, but a colleague worries the result is biased because treated and control users interact through the same supply. How do you reason about that risk and design an experiment you'd actually trust?
technical-conceptual · Senior level · data-ml
What the interviewer is really asking
Assesses whether the candidate recognizes interference / SUTVA violations in networked or marketplace settings and can choose an experiment design (cluster, geo, or switchback randomization) that restores an unbiased estimate, rather than trusting a naive user-level split.
What to say
- Name the assumption being violated: a standard user-level A/B test assumes one unit's outcome doesn't depend on others' treatment (SUTVA / no interference). In a marketplace or network, treated users consume shared, finite supply, so they affect control users' outcomes — which biases the naive estimate, usually overstating the effect.
- Pick a randomization unit that contains the interference: randomize at the level of the spillover. Cluster randomization (or geo/market-level splits) groups interacting users so spillover stays inside a unit; switchback designs randomize over time windows when the whole market must see one condition at a time.
- Acknowledge the trade-offs honestly: cluster/geo and switchback designs have far fewer effective units, so they need more total exposure for the same power and careful handling of carryover between time blocks — but they buy an estimate that isn't contaminated by cross-arm interference.
What to avoid
- Trusting the user-level result because the p-value was small — significance on a biased estimator just means you're confident in the wrong number.
- Claiming interference is negligible without checking whether treated and control units actually share supply or otherwise affect each other.
- Picking switchback or cluster randomization without acknowledging the cost: fewer effective units, lower power per dollar, and carryover/temporal-spillover effects to manage.
Example answers
Strong: The colleague is pointing at an interference problem: a user-level test assumes SUTVA — one user's outcome doesn't depend on others' assignment — but in a marketplace treated and control users compete for the same finite supply, so treatment leaks across arms and the naive estimate is biased, typically inflated. The fix is to randomize at the level that contains the spillover: cluster or geo/market-level randomization so interacting users land in the same arm, or a switchback design that flips the whole market between conditions over time. I'd be upfront that both cost power — far fewer effective units, and switchback adds carryover between time blocks — so I'd size for that. But I'd rather have a slightly noisier estimate that's unbiased than a tight one that's systematically wrong.
Weak: The test was randomized and significant, so the result should be trustworthy — randomization handles confounders. Marketplace effects probably wash out across enough users, so I'd ship it based on the A/B result we have.