LoRA (Low-Rank Adaptation)

Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning method that enables adaptation of large language models by injecting trainable low-rank decomposition matrices into each layer of the transformer architecture while keeping the original pre-trained weights frozen. Instead of updating billions of parameters during fine-tuning, LoRA typically trains less than one percent of the total parameters while achieving performance comparable to full fine-tuning.

The core insight behind LoRA is that the weight updates during fine-tuning have a low intrinsic rank, meaning the change in weights can be effectively approximated by the product of two much smaller matrices. For a weight matrix W of dimensions d by k, LoRA decomposes the update into matrices A (d by r) and B (r by k) where r is the rank, typically between 4 and 64. This reduces the number of trainable parameters from d times k to r times the sum of d and k, which can be orders of magnitude smaller.

QLoRA extends this approach by combining LoRA with 4-bit quantization of the base model, enabling fine-tuning of models with tens of billions of parameters on a single consumer GPU. This democratization of fine-tuning capability has made custom model adaptation accessible to organizations without massive compute budgets. Additional variants include AdaLoRA, which adaptively allocates rank across layers, and DoRA, which decomposes weight updates into magnitude and direction components.

In production workflows, LoRA adapters offer unique deployment advantages. Multiple task-specific adapters can share a single base model, with adapters hot-swapped at inference time based on the request type. This enables multi-tenant model serving where different customers or use cases have their own fine-tuned behavior without the cost of maintaining separate full model copies. LoRA adapters are typically only a few megabytes in size, making them easy to version, distribute, and manage in model registries.