Model Quantization

Model quantization stores weights in lower precision, such as 8-bit integers instead of 32-bit floats.

Layers, connections, and learned patterns stay. Each parameter uses a smaller type. The model uses a fraction of the memory and runs faster, because integer math is simpler and uses less power than floating-point math. Quantized models fit on phones, wearables, and microcontrollers, so inference can run on-device without a cloud round-trip.

That cuts latency to milliseconds, keeps data on the device, and lowers compute cost.

Predictions can differ slightly from the full-precision model. For most applications the accuracy drop is small. Voice assistants and drones that need real-time decisions use quantization as the practical path from a research checkpoint to a product that fits on the device.

FP32 weights become INT8 or lower. Matmuls become integer ops. Memory and energy fall. A phone, wearable, or microcontroller can then run the net without a cloud hop, which means millisecond latency and on-device privacy. Accuracy usually drops a little; calibration and quantization-aware training keep the drop small for many tasks.

Voice assistants and drones need that trade: a research checkpoint in 32-bit will not fit the device, a quantized one will. The usual path is 32-bit floats down to 8-bit integers so a phone GPU or NPU can run the net.