DoRA: Decomposing Weights into Magnitude and Direction
~15 min read
DoRA separates a pretrained weight matrix into magnitude and direction components, fine-tuning each independently — a refinement that improves parameter efficiency and performance over plain LoRA's simple additive update.
DoRA: Decomposing Weights into Magnitude and Direction is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •DoRA decomposes a pretrained weight matrix into magnitude (m) and direction (V) components, rather than applying one combined additive update
- •Magnitude captures 'how big' a weight is; direction captures 'which way it points' — genuinely different aspects of a weight update
- •Fine-tuning these two components independently improves parameter efficiency and performance over plain LoRA's simple additive update
- •DoRA is a refinement built explicitly on top of LoRA's low-rank adaptation principles, preserving its efficiency
- •Worth reaching for when chasing the best accuracy within a LoRA-style budget, at the cost of modest added implementation complexity