A teammate proposes fixing RAG quality by switching to a long-context model and just stuffing the top 50 retrieved chunks into the prompt. What's your view on that approach, and how do you decide how much retrieved context to actually pass?

technical-conceptual · Senior level · data-ml

What the interviewer is really asking

Assesses whether the candidate understands that more context is not free — lost-in-the-middle degradation, cost, latency, and distraction from irrelevant chunks — and decides the context budget with reranking and evaluation rather than maximizing chunk count.

What to say

• Push back on 'more context is better': models attend unevenly across a long window — relevant facts buried in the middle are often missed (the lost-in-the-middle effect), and irrelevant chunks actively distract the model and can lower answer quality, so dumping 50 chunks can hurt, not help.
• Account for the real costs: a long stuffed prompt drives up token cost and latency on every request, and most of those 50 chunks add noise rather than signal, so you're paying more to get a worse, slower answer.
• Decide the budget empirically: over-retrieve a wide candidate set, rerank with a cross-encoder, and keep only the top few high-precision chunks, ordering the strongest near the start and end; then sweep k on an eval set and pick the point where answer quality plateaus rather than maximizing chunk count.

What to avoid

• Assuming a bigger context window means you can skip relevance work — irrelevant context distracts the model and degrades answers even when it technically fits.
• Ignoring that every extra chunk costs tokens and latency on every single request, which matters at production scale.
• Picking the number of chunks by intuition instead of sweeping k against a measured answer-quality metric.

Example answers

Strong: I'd push back. More context isn't free: models attend unevenly across a long window, so a key fact buried in the middle gets missed, and the irrelevant chunks among 50 actively distract the model and can lower quality. On top of that you pay the token cost and latency of a huge prompt on every request, mostly for noise. I'd instead over-retrieve a wide candidate set, rerank with a cross-encoder, and pass only the top few high-precision chunks, placing the strongest near the start and end. Then I'd sweep k on an eval set and pick where answer quality plateaus — usually a handful of chunks beats fifty, and it's cheaper and faster too.

Weak: Long-context models can handle huge prompts now, so passing the top 50 chunks is fine — more context just gives the model more to work with, and if it all fits in the window there's no real downside. I'd go with that since it's simpler than reranking.