Your pipeline keeps breaking because an upstream team changes their data — they rename a column or change a type without warning, and you find out when a downstream dashboard goes blank. The current fix is to add more validation tests on your side. How would you reason about a more durable solution?
technical-conceptual · Senior level · data-ml
What the interviewer is really asking
Assesses whether the candidate can move beyond reactive downstream testing toward data contracts and shift-left enforcement — making the producer accountable for schema and SLA, enforcing backward compatibility in the producer's CI before bad data ships, and assigning clear ownership — rather than only piling defensive checks onto the consumer side after breakage.
What to say
- Diagnose the structural issue: validation tests on the consumer side are reactive — they catch the break after the bad data has already landed, and you're paying to detect a problem the producer could have prevented. The durable fix moves enforcement upstream (shift-left) to the producer.
- Propose a data contract: an explicit agreement on schema (fields, types, semantics), SLAs (freshness, completeness), allowed evolution rules, and ownership — enforced in the producer's CI so a backward-incompatible change (dropping or retyping a field consumers rely on) fails their build before it ever reaches you.
- Cover evolution and ownership: default to backward-compatible changes (additive, nullable), require a versioning/deprecation path for breaking ones, and make the producer the accountable owner of the contract — keep your downstream tests as a safety net, not the primary line of defense.
What to avoid
- Just adding more downstream tests and alerts — it scales the reactive cost without fixing the root cause, and you still find out after the dashboard breaks.
- Framing it as the upstream team being careless, rather than as a missing contract and missing CI enforcement that no process currently requires of them.
- Proposing a contract with no enforcement mechanism — a wiki page or Slack agreement nobody's build checks is not a contract, it's a hope.
Example answers
Strong: The real problem is that enforcement lives on my side and runs after the data already shipped, so I'm always reacting. I'd push for a data contract on that feed: the producer and I agree on the schema, types, freshness SLA, and the rules for how it can evolve, and we encode it as a check in the producer's CI. A backward-incompatible change — dropping or retyping a field I depend on — then fails their pipeline before it ever lands, instead of failing my dashboard. Additive, nullable changes stay allowed; breaking ones go through a versioned deprecation. I'd keep my downstream tests as a backstop, but the producer owns the contract now. The hard part is organizational, so I'd frame it around the incident cost to get their buy-in.
Weak: I'd add stronger schema validation and more tests at the start of my pipeline so it fails fast and alerts me when the upstream data changes, and maybe a Slack channel where the other team posts when they're going to change something. That way at least I know about it quickly and can react before too many people see the broken dashboard.