For a new feature that answers user questions from your company's internal documentation, how would you decide between prompting a base LLM, retrieval-augmented generation, and fine-tuning?

technical-conceptual · Mid level · data-ml

What the interviewer is really asking

Assesses whether the candidate can choose an LLM adaptation strategy from the actual requirement — freshness of knowledge, grounding/citation needs, and cost — rather than defaulting to fine-tuning, and understands that RAG and fine-tuning solve different problems.

What to say

• Separate what each technique actually changes: prompting changes instructions only, RAG injects fresh external knowledge at query time without retraining, and fine-tuning bakes behaviour or style into the weights. Internal docs that change often are a knowledge problem, which points at RAG, not fine-tuning.
• Match RAG to the grounding requirement: because it retrieves passages you can cite, it reduces hallucination and lets you show sources, and you update the answer by re-indexing a document instead of retraining — critical for docs that change weekly.
• Reserve fine-tuning for form, not facts: use it when you need a consistent format, tone, or a narrow task the base model handles poorly, and call out that fine-tuning on a fast-changing corpus is expensive, risks catastrophic forgetting, and goes stale the moment the docs change. The common production answer is RAG first, fine-tune later only for the highest-value patterns.

What to avoid

• Reaching for fine-tuning as the default to 'teach the model our docs', ignoring that it embeds a snapshot that goes stale and can't cite a source.
• Treating RAG and fine-tuning as interchangeable rather than solutions to different problems (fresh/grounded knowledge vs learned behaviour or style).
• Skipping the cheapest option — better prompting and a good retrieval setup — and jumping straight to a training pipeline you'll have to maintain.

Example answers

Strong: For an internal support assistant over docs that changed weekly, I built RAG first: chunked the docs, embedded them, retrieved the top passages per query, and grounded the answer with citations — when a doc changed I just re-indexed it, no retraining, and hallucinated answers dropped sharply because the model quoted retrieved text. I held fine-tuning in reserve only for output format.

Weak: I'd fine-tune the model on all our internal documents so it knows our content — that's the most thorough way to make it an expert on our domain.