LoRA: Low-Rank Decomposition, Rank r, and Alpha
~15 min read
LoRA decomposes a weight update into two small matrices A and B, governed by two key hyperparameters — rank r (how much capacity the adaptation has) and alpha (how strongly it's applied).
LoRA: Low-Rank Decomposition, Rank r, and Alpha is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •LoRA decomposes a weight update into two small matrices A (d×r) and B (r×k), training only these while the base weight W stays frozen
- •Rank r is the capacity dial — smaller r means far fewer trainable parameters than full fine-tuning
- •A is Gaussian-initialized, B is zero-initialized — guaranteeing A×B=0 at the start, so the model's original behavior is exactly preserved before any training
- •Alpha is a separate scaling factor controlling how strongly the adaptation affects the output, independent of r's capacity
- •r controls how expressive the adaptation CAN be; alpha controls how strongly that expressiveness is actually applied