Pruning

Pruning is a neural network compression technique that identifies and removes parameters (weights, neurons, or entire layers) that contribute minimally to model output, producing a smaller, faster model that retains most of the original accuracy. The insight behind pruning is that trained neural networks are typically over-parameterized, containing many weights that are near zero or redundant, and can be removed without significantly affecting performance.

Pruning methods are categorized by granularity. Unstructured pruning removes individual weights based on magnitude or other importance criteria, creating sparse weight matrices. While this can achieve high compression ratios, the resulting irregular sparsity patterns require specialized hardware or software support for actual speed improvements. Structured pruning removes entire neurons, channels, or attention heads, producing models that run faster on standard hardware without specialized sparse computation support. Block-sparse pruning removes contiguous blocks of weights, balancing compression ratio with hardware efficiency.

The pruning workflow typically involves training a model to convergence, applying a pruning criterion to identify removable parameters, removing those parameters, and fine-tuning the pruned model to recover accuracy lost during pruning. Iterative pruning repeats this cycle multiple times, gradually increasing the pruning ratio. Lottery ticket hypothesis research suggests that within randomly initialized networks there exist sparse subnetworks that can be trained to match full model performance, providing theoretical grounding for the effectiveness of pruning.

Pruning complements other compression techniques in practical deployment scenarios. Combined with quantization, pruning enables aggressive model size reduction for edge deployment. Combined with distillation, pruning helps create efficient student models. NVIDIA hardware supports structured sparsity natively on Ampere and later architectures, providing two-times speedup for models with fifty percent structured sparsity. Understanding pruning trade-offs is important for teams optimizing model serving costs and deploying AI on resource-constrained platforms.