Why do we split data into training, validation, and test sets, and what is each one used for?

technical-conceptual · Junior level · data-ml

What the interviewer is really asking

Assess whether the candidate understands the distinct role of each split and why touching the test set during development invalidates the final estimate.

What to say

• Give each its job: the training set is what the model learns from; the validation set is for tuning hyperparameters and comparing models during development; the test set is touched ONCE at the end to give an unbiased estimate of how the chosen model generalises.
• Explain the test-set discipline: if you tune against the test set or peek repeatedly, it stops being held-out and your reported performance becomes optimistic — the test set must stay untouched until the very end.
• Mention cross-validation as the alternative when data is limited: k-fold CV rotates the validation role across folds so every point is used for both training and validation, giving a more stable estimate without a fixed validation split.

What to avoid

• Conflating the validation and test sets, or tuning hyperparameters on the test set — that leaks the test set into model selection and inflates your final number.
• Splitting randomly when the data is time-ordered — for time series you must split chronologically so the model never trains on the future.
• Splitting after preprocessing the full dataset, which leaks statistics across sets; fit transforms on training only.

Example answers

Strong: On a classification project I trained on 70% of the data, used a 15% validation set to grid-search the regularisation strength and pick between logistic regression and a gradient-boosted model, then evaluated the single winning model once on the held-out 15% test set — that test number was the one I reported to stakeholders.

Weak: I split into train and test, then keep checking the test score after each change until it looks good.