What is the curse of dimensionality, and what practical problems does it cause in ML pipelines?

technical-conceptual · Mid level · data-ml

What the interviewer is really asking

Assess understanding of how high-dimensional feature spaces affect distance-based methods, data density, and model training.

What to say

• As the number of dimensions grows, the volume of the feature space grows exponentially — data becomes increasingly sparse, distances between points converge, and models need exponentially more samples to generalise.
• Practical problems: k-NN and clustering degrade because distances become uninformative; models overfit because the ratio of parameters to samples worsens; visualisation becomes impossible.
• Mitigations: PCA or other dimensionality reduction, feature selection, regularisation, or switching to algorithms less sensitive to high dimensions (tree-based models).

What to avoid

• Saying more features always help because they give the model more information.
• Confusing high dimensionality with multicollinearity (related but distinct issues).
• Not connecting the theory to practical impact on specific algorithms.

Example answers

Strong: I had a k-NN classifier with 300 features and near-random performance. After applying PCA to reduce to 30 components capturing 95% of variance, k-NN accuracy jumped from 54% to 79%.

Weak: The curse of dimensionality means you have too many features for the amount of compute you have available.