To run a small neural network on a microcontroller, models are usually converted from 32-bit floats to 8-bit integers. What does that quantization buy you, and what's the catch?

technical-conceptual · Junior level · software-engineering, embedded-iot

What the interviewer is really asking

Assesses understanding of why INT8 quantization is the enabling step for TinyML on a microcontroller and awareness of the accuracy trade-off and validation it requires, not just the definition of quantization.

What to say

• Explain the benefit concretely: converting weights from 32-bit float to 8-bit int shrinks the model roughly four times and lets it fit in the tens of kilobytes a microcontroller has, and integer math also runs faster and cheaper than float on parts with no floating-point unit.
• Name the catch: quantization is lossy, so you can lose some accuracy, and you have to re-evaluate the quantized model on real data rather than trusting the float model's numbers.
• Mention input scaling — the int8 model expects inputs in a specific range, so if you feed it raw sensor values without matching the quantizer's scale the accuracy can collapse even though the model 'works.'

What to avoid

• Saying quantization is free with no downside, which ignores the accuracy hit and the need to validate.
• Confusing model size with runtime memory, or not knowing that a microcontroller typically can't fit a full float model at all.
• Forgetting that the runtime on these parts allocates a fixed buffer up front and does no dynamic allocation, so a model that doesn't fit simply won't run.

Example answers

Strong: Going from float32 to int8 cuts the model size about four times so it fits in the tens of kilobytes the chip has, and integer math is faster on a part with no FPU. The catch is it's lossy: I'd re-test the quantized model on real data, not assume the float accuracy carries over. I'd also make sure my input scaling matches what the quantizer expects, because a range mismatch can wreck accuracy even when the model loads fine.

Weak: Quantization makes the model 8-bit so it's smaller and faster, and you use it because microcontrollers are slow, so you always want the smallest model.