You're shipping a feature built on a large language model, and the team asks how you'll know if it's good enough to launch. How do you define and measure success for a non-deterministic AI feature?

role-specific · Senior level · product-management

What the interviewer is really asking

Assesses whether a senior PM can define quality for a probabilistic model output — building an evaluation harness, pairing offline eval with online behavioral metrics and cost/latency guardrails — rather than treating an LLM feature like a deterministic one with a single pass/fail.

What to say

• Define a task-specific quality bar with a graded eval set, not a single accuracy number: build a representative golden set of inputs with rubric-scored expected behavior, since 'correct' for generative output is graded quality, not a binary.
• Pair offline eval (the golden set scored by humans or an LLM-judge you've validated against human ratings) with online signals — acceptance/edit rate, thumbs, task completion, escalation-to-human — because offline eval predicts launch readiness and online metrics prove real-world value.
• Treat cost, latency, and failure behavior as first-class guardrails alongside quality: a feature that's accurate but too slow, too expensive per call, or that fails ungracefully isn't shippable, so set explicit thresholds on each before launch.

What to avoid

• Don't propose a single accuracy percentage as the bar; generative quality is graded against a rubric, and one number hides the distribution of failures that actually hurt users.
• Don't rely only on vibes-based demos or a handful of cherry-picked prompts; without a representative held-out eval set you're measuring your own optimism, not the feature.
• Don't ignore cost-per-call, latency, and the behavior on the inputs the model gets wrong — unbounded inference cost or a confident wrong answer can sink an otherwise-accurate feature.

Example answers

Strong: For an AI summarization feature, I built a 300-example golden set covering the real input distribution, scored on a rubric for faithfulness and usefulness, and validated an LLM-judge against human ratings so we could iterate cheaply. Launch criteria were a faithfulness score above our bar AND p95 latency under 3s AND cost-per-summary under a set ceiling. Offline I gated on the golden set; in the canary I watched edit-rate and thumbs-down, and set an alert on hallucination reports so a quality regression would page us rather than show up in churn.

Weak: I'd measure its accuracy and launch once we're hitting something like 90% correct on our test prompts — if it's right most of the time, it's good enough to ship.