A small change to a large matrix

A pretrained language model already knows most of what it needs. Adapting it to a task, such as a style, a domain or a format, usually means nudging its weights, not relearning them. Yet full fine-tuning updates every one of those weights, keeps optimiser state for each, and leaves you with a complete new copy of the model for every task.

Low-rank adaptation, or LoRA, starts from a bet about the shape of that nudge.Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, 2021. The bet builds on earlier evidence that fine-tuning has a low “intrinsic dimension” (Aghajanyan et al., 2020).

The bet

Take one weight matrix in the network, $W_0 \in \mathbb{R}^{d\times k}$, say a projection inside an attention block. Fine-tuning replaces it with $W_0 + \Delta W$. The bet is that $\Delta W$ has low rank: most of what it does lies along a handful of directions.

So, rather than learn $\Delta W$ freely, LoRA writes it as a product of two thin matrices and learns those:

$$h = W_0 x + \frac{\alpha}{r}\, B A x, \qquad B \in \mathbb{R}^{d\times r},\; A \in \mathbb{R}^{r\times k},\; r \ll \min(d,k).$$

$W_0$ is frozen. Only $A$ and $B$ receive gradients. A rank-$r$ product has $r(d+k)$ entries instead of $dk$. For a $4096 \times 4096$ projection at $r = 8$, that is 65,536 trainable weights in place of 16.8 million, about 0.4%.

How much is lost

How much of the update survives depends on how quickly its singular values fall off. The best any rank-$r$ matrix can do is to keep the $r$ largest singular directions and drop the rest. This is the Eckart–Young theorem,Precisely: among all matrices of rank at most $r$, the truncated SVD minimises $\lVert \Delta W - M\rVert$ in both the Frobenius and spectral norms. and it is the ceiling a trained $BA$ can approach but never exceed.

The figure builds an update with a spectrum you control, then shows the best rank-$r$ approximation of it.

At the starting settings, rank 4 already captures about nine-tenths of the update, because its spectrum falls quickly. Now slow the fall-off. The update spreads across many directions, the kept fraction drops, and the approximation on the top right loses its structure. LoRA works when the task is close to what the model already does. A task that requires broad change is a poor fit for a low-rank update.

A rising noise floor has a subtler effect. The tail it adds is real energy that rank $r$ cannot reach. But in a trained model, much of that tail is noise in the gradient estimates, and leaving it out can act as regularisation.

Details that matter in practice

Initialisation. $A$ starts random and $B$ starts at zero, so $BA = 0$ and training begins exactly at the pretrained model.Why not start both at zero? See the second exercise.

The α/r factor. When you change $r$, the size of $BA$ at a given learning rate changes with it. Dividing by $r$ means a learning rate tuned at one rank roughly carries over to another. Later work argues for $\alpha/\sqrt{r}$ when $r$ is large.Kalajdzievski, 2023, “rank-stabilised” LoRA.

Merging. After training, $W_0 + \tfrac{\alpha}{r}BA$ is an ordinary $d\times k$ matrix. Fold it in and inference costs exactly what it did before. Keep it separate instead, and one base model can serve many tasks by swapping small adapters.

Where to put it. The original paper adapted only the query and value projections of attention. Current practice usually adapts every linear layer, at a modest rank.

A minimal layer in PyTorch:

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=8, alpha=16):
        super().__init__()
        self.base = base.requires_grad_(False)          # W0, frozen
        self.A = nn.Parameter(torch.randn(r, base.in_features) / r**0.5)
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
        self.scale = alpha / r

    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A.T @ self.B.T)

The same idea combines with a quantised base model. Storing $W_0$ in four bits and training $A, B$ in higher precision is QLoRA, which makes fine-tuning very large models possible on a single GPU.Dettmers et al., 2023.