Your multi-agent system passes its demos but fails intermittently in production — sometimes a worker returns garbage the orchestrator trusts, sometimes the whole thing stalls. How do you make a system like this observable enough to actually diagnose and fix?
technical-conceptual · Senior level · data-ml
What the interviewer is really asking
Probes whether the candidate can instrument and reason about distributed-agent failures — attributing a failure to a specific agent/tool hop rather than treating the system as an opaque blob.
What to say
- Insist on step-level tracing stitched into one trace tree per request: every agent's reasoning step, the tools it considered and invoked, the arguments, the returned result, tokens, and latency per hop — so a failure has a location, not just a symptom.
- Evaluate at two levels — trace-level (was each individual step and tool call correct?) and session-level (did the system achieve the user's actual goal?) — because a system can take all-wrong steps and still stumble into the right answer, or vice versa.
- Call out the specific multi-agent failure modes to watch for: an orchestrator trusting a malformed worker output, recursive loops where an agent keeps returning to the same tool, and context bleed between agents — and use trajectory mapping to surface the loops.
What to avoid
- Relying on HTTP-level metrics (latency, 5xx) — those never show wrong tool selection, drifted retrieval, or a worker hallucinating an output the orchestrator accepts.
- Reading a handful of full transcripts by hand as the diagnostic method; that doesn't scale and won't isolate which hop failed.
- Treating the multi-agent system as one black box and 'just changing the prompt' without attributing the failure to a specific agent or tool.
Example answers
Strong: I'd instrument every request as a single trace tree where each agent's reasoning, tool calls, arguments, results, tokens, and latency are spans. Then I evaluate two layers: trace-level, did each step and tool call do the right thing, and session-level, did we hit the user's goal — because all-correct steps can still miss the goal. For the stalls, trajectory mapping shows the recursive loop and which tool the agent keeps returning to; for the garbage-trust case, I add a validation/judge step on worker outputs before the orchestrator consumes them, and I gate releases on at least ~50 eval cases per failure mode so a fix is proven, not anecdotal.
Weak: I'd add a lot more logging and turn up the log level so we can see what's happening. Then when it fails I'd read through the logs and the full conversation transcript to spot where it went wrong, and tweak the orchestrator's system prompt to tell it to be more careful about trusting bad outputs and to stop looping.