Quantization

Quantization is a model compression technique that reduces the numerical precision of neural network weights and activations from higher-bit formats (typically FP32 or FP16) to lower-bit representations (INT8, INT4, or even lower). This reduction in precision decreases model memory footprint, increases inference throughput, and reduces power consumption, often with minimal impact on model accuracy.

Quantization approaches fall into two main categories. Post-training quantization (PTQ) converts a trained model to lower precision without retraining, using calibration data to determine optimal quantization parameters. This is fast and convenient but may cause accuracy degradation, especially at very low bit widths. Quantization-aware training (QAT) simulates the effects of quantization during training, allowing the model to adapt its weights to maintain accuracy at the target precision. QAT generally produces better results than PTQ but requires access to training infrastructure and data.

For large language models, specialized quantization methods have been developed to handle their unique challenges. GPTQ applies layer-wise quantization with compensation for accuracy preservation. AWQ (Activation-aware Weight Quantization) preserves the most important weights at higher precision. GGML/GGUF formats provide flexible quantization for CPU inference through llama.cpp. FP8 quantization, supported natively on NVIDIA Hopper GPUs, provides a favorable balance between compression and accuracy for both training and inference.

Quantization is a key enabler for practical AI deployment. It allows large models to fit on smaller GPUs, reducing infrastructure costs. It enables edge deployment on resource-constrained hardware. Combined with other optimization techniques like pruning and distillation, quantization can reduce model serving costs by an order of magnitude while maintaining production-quality outputs. The optimal quantization strategy depends on the target hardware, acceptable accuracy trade-off, and whether the deployment prioritizes latency, throughput, or memory efficiency.