You want to quantize an LLM to cut serving cost. How do you decide how far to quantize, and how do you make sure you haven't degraded quality?

technical-conceptual · Mid level · data-ml

What the interviewer is really asking

Tests whether the candidate understands quantization as a memory/cost-vs-accuracy trade — the difference between INT8 and INT4 and KV-cache quantization, that lower precision saves more but risks more degradation, and that the only honest way to choose is task-specific evaluation, not a perplexity headline.

What to say

Frame quantization as trading numeric precision for memory and cost: going FP16 to INT8 typically halves memory and roughly halves cost while holding 95-99% of accuracy, while INT4 saves more but the degradation risk climbs — so the precision level is a dial set by how much quality headroom your task has, not a fixed choice.
Separate weight quantization from KV-cache quantization: weights shrink the model footprint, while quantizing the KV cache (e.g. INT8) lets you serve longer contexts and more concurrent requests, and the two stack — but each adds its own small accuracy cost you have to measure independently.
Validate on YOUR task, not a generic benchmark: a small drop in perplexity can hide a real regression on your specific use case, so run the quantized model against your own eval set and compare task metrics and a side-by-side on hard cases — and prefer post-training quantization first, reaching for quantization-aware training only if PTQ loses too much.

What to avoid

Quantizing straight to INT4 to maximise savings without checking what it does to quality on the actual task.
Trusting a headline like 'negligible perplexity change' as proof quality is fine, instead of measuring your own task metrics.
Forgetting that the KV cache can be quantized separately from the weights, leaving an easy memory and concurrency win on the table.

Example answers

Strong: I wanted to cut serving cost on a classification-style extraction feature, so I quantized FP16 to INT8 first and ran it against our 400-example eval set: task accuracy held within half a point and cost dropped about 45%, so I shipped it. I tried INT4 too, but it dropped accuracy by 4 points on the hard slice, which wasn't worth the extra savings for this feature.

Weak: I'd quantize to INT4 since it gives the biggest savings, and the quality loss is usually small enough not to matter.

Want questions matched to your role? Paste a job title, job description, or CV and get a personalized set, or go Pro to unlock the full bank.