Your RAG service is over budget on inference spend and its p95 latency is too high, but nobody can say which step is responsible. How would you use tracing to attribute cost and latency, and then decide where to cut?
technical-conceptual · Senior level · data-ml
What the interviewer is really asking
Probes whether the candidate can turn traces into a cost/latency optimization tool — attributing spend and time to specific pipeline steps and reasoning about the right lever — rather than guessing.
What to say
- Attribute tokens and dollars to each span — embedding, retrieval, rerank, generation — so the trace tells you whether spend is dominated by oversized context, an expensive generation model, or a chatty rerank step.
- Separate latency the same way: look at p95 per step, not just end-to-end, so you can see whether retrieval, a cold model, or generation length is driving the tail, and whether independent steps could run in parallel.
- Pick the lever the data points to — trimming retrieved context or top-k, caching repeated retrievals, routing easy queries to a cheaper model — and confirm it on the same traces afterward instead of assuming the change helped.
What to avoid
- Optimizing on a hunch — for example quantizing or swapping the model before the traces show generation is actually the cost driver.
- Looking only at the end-to-end average, which hides which step owns the p95 tail and which requests are the expensive ones.
- Cutting context or top-k purely for cost without checking the effect on answer quality, trading a budget win for a regression you don't measure.
Example answers
Strong: I'd put token and cost counters on each span and break p95 down per step. On a service I ran, the traces showed generation tokens were 70% of spend because we stuffed twelve chunks into context; cutting to the top four most relevant and adding a retrieval cache dropped cost about 40% and trimmed the latency tail, and I verified both on the same traces with no measurable quality drop on our eval set.
Weak: It's too expensive, so I'd switch to a smaller, cheaper model and hope latency improves too.