LoRA: Low-Rank Decomposition, Rank r, and Alpha

Plain LoRA's core mechanism, underlying all 8 variants this topic compares, is decomposing a large weight update into two much smaller matrices. For a pretrained weight matrix W with dimensions d × k, instead of fine-tuning the full W directly, LoRA introduces a low-rank update via two matrices: A (with dimensions d × r) and B (with dimensions r × k). During training, only A and B are updated while W itself stays frozen — dramatically reducing the number of trainable parameters compared to fine-tuning the full weight matrix.

The hyperparameter r — the rank — is the key capacity dial. Both A and B have one dimension sized by r, and since r is chosen to be much smaller than d or k, this is what makes the decomposition parameter-efficient in the first place: instead of learning d × k parameters directly, you learn only (d × r) + (r × k) parameters, which is far smaller when r is small relative to d and k.

Initialization matters specifically here too: matrix A is initialized from a Gaussian distribution (optionally scaled down so initial values aren't too large), while matrix B is initialized as a zero matrix. This specific choice ensures the product A × B is exactly zero at the very start of fine-tuning — meaning the original model's behavior is exactly preserved until any actual training happens, since W_final = W + (alpha/r)(B × A) reduces to just W when B × A is zero.

Alpha, the second key hyperparameter, is a scaling factor controlling how much impact the LoRA layer's adjustment actually has on the final output. A higher alpha means more pronounced, significant changes from the LoRA layer; a lower alpha means more subtle changes, since the transformation's impact is reduced. Together, r controls how much CAPACITY the adaptation has (how expressive it can be), while alpha controls how STRONGLY that capacity is actually applied to the model's output — two genuinely independent dials, both worth tuning deliberately rather than leaving at arbitrary defaults.

import torch
import torch.nn as nn

class LoRAWeights(nn.Module):
    def __init__(self, d: int, k: int, r: int, alpha: float):
        super().__init__()
        self.r = r
        self.alpha = alpha
        # Gaussian init for A (optionally scaled down); zero init for B —
        # ensures A @ B = 0 at the very start of fine-tuning
        self.A = nn.Parameter(torch.randn(d, r) * 0.01)
        self.B = nn.Parameter(torch.zeros(r, k))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # alpha/r scales how strongly this adaptation affects the output
        return (self.alpha / self.r) * (x @ self.A @ self.B)

d, k = 4096, 4096

# r controls CAPACITY — trainable params grow with r
for r in [4, 8, 16]:
    lora = LoRAWeights(d, k, r=r, alpha=16)
    trainable = sum(p.numel() for p in lora.parameters())
    full_finetune = d * k
    print(f"r={r}: {trainable:,} trainable params ({trainable / full_finetune:.4%} of full fine-tuning)")