How it works

LoRA (Low-Rank Adaptation) is a fine-tuning technique that does not touch the model's original weights. Instead, it adds two small, low-rank matrices alongside the original layers. During training, only these new matrices are updated; the base model stays frozen.

The core idea is that the change needed to adapt a model to a new task can be represented by a low-rank matrix. In other words, instead of updating a huge weight matrix, you only need to learn a compact correction. This drastically reduces the number of trainable parameters and video memory (VRAM) consumption.

In practice, you choose which layers of the model will receive the LoRA matrices (usually the attention layers) and set a small rank, such as 8 or 16. The larger the rank, the more learning capacity, but also more memory and training time.

Practical example

Imagine you want to teach a code model to follow your company's commit style. With LoRA, you prepare a set of commit examples and run training on a single GPU. The result is a small file (tens of megabytes) with the LoRA matrices, which you load on top of the base model at inference time. The original model is not changed.

Tools like Hugging Face PEFT and training libraries (Axolotl, Unsloth) already ship with LoRA ready to use. The typical command passes the base model, the dataset and the rank:

Terminal
python train.py --model base-model --dataset commits.jsonl --lora-r 16 --output lora-adapter

What it's used for

LoRA is used to adapt language models to specific tasks with little hardware and little data. It is the basis for most fine-tuning done by small teams. It also allows you to keep multiple adaptations (one per client, one per style) without duplicating the base model, saving storage and memory.

Terms like QLoRA take the idea further by combining LoRA with quantization, enabling training on consumer GPUs.