---
title: Model Quantization
description: Page on Vedang Vatsa's site: https://veda.ng/glossary/model-quantization
canonical: https://veda.ng/glossary/model-quantization
last_updated: 2026-10-03
type: text/markdown
---
# Model Quantization

Source: https://veda.ng/glossary/model-quantization
Author: Vedang Vatsa (https://veda.ng/about)

Model quantization stores weights in lower precision, such as 8-bit integers instead of 32-bit floats.

Layers, connections, and learned patterns stay. Each parameter uses a smaller type. The model uses a fraction of the memory and runs faster, because integer math is simpler and uses less power than floating-point math. Quantized models fit on phones, wearables, and microcontrollers, so inference can run on-device without a cloud round-trip. That cuts latency to milliseconds, keeps data on the device, and lowers compute cost.

Predictions can differ slightly from the full-precision model. For most applications the accuracy drop is small. Voice assistants and drones that need real-time decisions use quantization as the practical path from a research checkpoint to a product that fits on the device.

FP32 weights become INT8 or lower. Matmuls become integer ops. Memory and energy fall. A phone, wearable, or microcontroller can then run the net without a cloud hop, which means millisecond latency and on-device privacy. Accuracy usually drops a little; calibration and quantization-aware training keep the drop small for many tasks. Voice assistants and drones need that trade: a research checkpoint in 32-bit will not fit the device, a quantized one will. The usual path is 32-bit floats down to 8-bit integers so a phone GPU or NPU can run the net.

Glossary index: https://veda.ng/glossary