How it works
AI models store their weights as floating point numbers, usually 16 or 32 bits. Quantization converts those numbers to lower precision formats, 8, 4 or even 2 bits, reducing model size and speeding up computation.
The idea is that the difference between 16 bits and 4 bits is small for most tasks: the model loses a little precision, but uses less memory and runs faster. It is like reducing the resolution of an image: you lose detail, but the image is still recognizable.
There are two main types: post training quantization, PTQ, applied after training, and quantization aware training, QAT, where the model is trained already considering quantization.
Practical example
A 70 billion parameter model in 16 bits takes about 140 GB of VRAM. Quantized to 4 bits, it takes about 35 GB. This makes it possible to run the model on a single 48 GB GPU, instead of needing multiple GPUs.
Tools like llama.cpp, vLLM and Ollama apply quantization automatically. Formats like GGUF for llama.cpp and AWQ/GPTQ for vLLM are the most used for quantized models.
What it is for
It is used to run large models on more modest hardware, reduce inference costs and increase response speed. It is essential for local inference and for edge deployment.
Difference between quantization and distillation
Quantization reduces the precision of the same model, same parameters, fewer bits. Distillation creates a smaller model with fewer parameters. Quantization is faster to apply and does not require retraining. Distillation can give better quality, but is more expensive. The two can be combined.