---
title: Quantization
description: Page on Vedang Vatsa's site: https://veda.ng/glossary/quantization
canonical: https://veda.ng/glossary/quantization
last_updated: 2026-10-03
type: text/markdown
---
# Quantization

Source: https://veda.ng/glossary/quantization
Author: Vedang Vatsa (https://veda.ng/about)

Quantization is a model compression technique that reduces the precision of a neural network's numerical weights, making models smaller, faster, and cheaper to run. Neural networks store their parameters as floating-point numbers, typically 32-bit or 16-bit values. Quantization reduces these to lower precision formats like 8-bit integers or even 4-bit values. The size reduction is dramatic: a 32-bit float model shrinks to one-eighth the size when quantized to 4-bit. This matters enormously for deployment. Running a 70-billion-parameter LLM at full precision requires hundreds of gigabytes of GPU memory. Quantized, the same model might fit on a single consumer GPU. The trade-off is accuracy: lower precision means less detail in the weights, which can degrade performance on complex tasks. But techniques like GPTQ, AWQ, and GGUF have made quantization surprisingly lossless, especially at 8-bit and even 4-bit. The emergence of llama.cpp and Ollama brought quantized models to consumer hardware, democratizing access to powerful LLMs. Quantization is now required to local AI deployment. Google's 2018 quantization paper showed 8-bit integer inference with little accuracy loss, which is how many mobile models ship.

Glossary index: https://veda.ng/glossary