advanced~4h
8 LoRA Fine-Tuning Variants Compared
LoRA, LoRA-FA, VeRA, Delta-LoRA, LoRA+, plus bonus techniques LoRA-drop, QLoRA, and DoRA — 8 distinct ways to parameter-efficiently fine-tune LLMs, compared.
8 LoRA Fine-Tuning Variants
Each variant targets one specific remaining inefficiency in plain LoRA — pick based on which resource (activation memory, base-model memory, trainable params, convergence speed, or accuracy) is your actual bottleneck.
| What's Trainable | Key Modification | Primary Benefit | |
|---|---|---|---|
| LoRA | A, B | Baseline low-rank update | Reduces trainable params from full W to A+B |
| LoRA-FA | B only | Freeze A | Further reduces activation memory |
| VeRA | Scaling vectors b, d | Shared frozen random A/B across layers | Massively reduces trainable params vs. per-layer A/B |
| Delta-LoRA | A, B, + delta on W | Also updates W | Closes the gap to full fine-tuning expressiveness |
| LoRA+ | A at lr₁, B at lr₂ | Differential learning rates | Faster / better convergence |
| LoRA-drop | A, B in high-impact layers only | Activation-based pruning | Cuts training cost with minimal accuracy loss |
| QLoRA | A, B, with W quantized | Quantized frozen base | Dramatically reduces base-model memory footprint (~75% at 8-bit) |
| DoRA | Magnitude m, direction V (separately) | Weight decomposition | Improved parameter efficiency & accuracy vs. plain LoRA |
4
Subtopics
1
Exercises
1
Projects
5
Quiz Qs
4
Flashcards
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Explain how each of the 8 LoRA variants modifies the base LoRA technique and why
- •Compute the memory savings QLoRA achieves via quantization on a concrete example
- •Explain DoRA's magnitude/direction decomposition and why it improves on plain LoRA
- •Choose the right LoRA variant given a memory budget, layer-importance profile, or accuracy target
8 LoRA Fine-Tuning Variants Compared is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.