Walk me through how you'd design a question-answering feature over our internal documents using an LLM. What are the main pieces?
technical-conceptual · Junior level · data-ml
What the interviewer is really asking
Assess whether the candidate can outline a retrieval-augmented QA system end to end — ingestion, retrieval, generation, grounding, evaluation — at a coherent high level appropriate for junior, without getting lost or missing whole stages.
What to say
- Outline the two phases: an offline ingestion phase (load docs, chunk them, embed the chunks, store vectors in an index) and an online query phase (embed the question, retrieve the most relevant chunks, put them in the prompt, generate a grounded answer).
- Insist the answer is grounded in retrieved content and cites its sources, so users can verify and so the model is less likely to make things up.
- Name what surrounds the core: access control so users only see docs they're allowed to, a way to keep the index fresh as documents change, and an evaluation loop to measure answer quality and retrieval relevance.
What to avoid
- Proposing to paste all the documents into the prompt instead of retrieving the relevant ones — it won't fit and it's expensive.
- Skipping the grounding/citation step, so the model answers from its parametric memory and hallucinates about internal content it never saw.
- Forgetting operational concerns entirely — permissions, keeping the index up to date, and measuring quality.
Example answers
Strong: Offline: I'd chunk the documents, embed each chunk, and store them in a vector index. Online: embed the question, retrieve the top relevant chunks, and prompt the LLM to answer using only those chunks with citations. Around that I'd add access control so users only retrieve docs they're allowed to see, a refresh job to re-index changed docs, and an eval set to track answer quality and retrieval relevance.
Weak: I'd fine-tune the model on all our documents so it just knows the answers.