Generative AI & LLMs

LLM Evaluation & Testing

10 practice questions. Free questions open a full answer guide; the rest unlock with Pro.

  • For a retrieval-augmented (RAG) feature, how would you measure whether the model's answers actually stay grounded in the retrieved documents rather than making things up? Junior level
  • How would you build an offline evaluation for an LLM feature so that you can catch regressions before they ship when you change the prompt, model, or retriever?Go Pro Mid level
  • You've been evaluating your agent by checking whether its final answer is correct. Why is that not enough for an agent that takes multiple steps and calls tools, and how would you actually evaluate it?Go Pro Senior level
  • A team is shipping an LLM-powered feature and the only evaluation so far is engineers reading a handful of outputs before each release. How would you build an evaluation harness for an LLM feature that you'd actually trust to gate a release?Go Pro Senior level
  • You're using an LLM as the judge to score another model's outputs at scale. How do you make sure the judge's scores are trustworthy?Go Pro Mid level
  • When evaluating an LLM feature's output quality, how do you decide between reference-based metrics, an LLM-as-judge, and human review? When does each make sense?Go Pro Mid level
  • What is the "LLM-as-a-judge" approach to evaluating an LLM feature, and when is it a reasonable thing to do?Go Pro Junior level
  • Your team uses a strong model as an LLM-as-a-judge to score outputs in your evaluation pipeline. What are the failure modes of that approach, and how do you make the judge's scores trustworthy enough to act on?Go Pro Senior level
  • You built an eval suite to gate LLM releases, but running it on the same model gives a different pass rate each time — 91%, then 88%, then 90%. How do you turn that into an evaluation you can actually base a ship or no-ship decision on?Go Pro Senior level
  • You changed the prompt for a production LLM feature and want to be confident you didn't make its answers worse. How would you test that before shipping?Go Pro Junior level
Want questions matched to your role? Paste a job title, job description, or CV for a personalized set, or go Pro to unlock the full bank.