For a retrieval-augmented (RAG) feature, how would you measure whether the model's answers actually stay grounded in the retrieved documents rather than making things up?
technical-conceptual · Junior level · data-ml
What the interviewer is really asking
Assess whether the candidate knows the specific concept of faithfulness/groundedness in RAG evaluation — checking that each claim in the answer is supported by the retrieved context — and can name a practical way to measure it rather than only judging fluency.
What to say
- Name the concept: faithfulness/groundedness — every claim in the answer should be supported by the retrieved context, separate from whether the answer reads well.
- Describe a practical measure: break the answer into claims and check each against the passed-in context, often with an LLM judge that is shown the context so it can verify support, or by citation-checking.
- Distinguish it from retrieval quality: groundedness assumes the right context was retrieved, so you'd also track retrieval metrics, and assemble a small evaluation set from real queries.
What to avoid
- Judging only fluency or relevance, missing that a fluent answer can still contradict or invent beyond the source.
- Confusing faithfulness with retrieval recall and conflating 'did we fetch the right doc' with 'did the answer use it honestly.'
- Saying you'd just read a few answers manually, with no repeatable metric or eval set.
Example answers
Strong: I'd measure faithfulness: does every claim in the answer trace back to the retrieved context, separate from whether it sounds good. A practical way is to decompose the answer into individual claims and check each against the context, often using an LLM judge that's given the context so it can verify support, or by requiring citations and checking them. I'd keep that distinct from retrieval quality — if we fetched the wrong doc that's a retrieval problem — and run it over an eval set built from real queries.
Weak: I'd look at whether the answers are relevant and well written and maybe have a couple of people rate them. If they read well and are on topic, the RAG system is working.