8 LoRA Fine-Tuning Variants Compared
LoRA, LoRA-FA, VeRA, Delta-LoRA, LoRA+, plus bonus techniques LoRA-drop, QLoRA, and DoRA — 8 distinct ways to parameter-efficiently fine-tune LLMs, compared.
Frozen Pretrained Base Weights W_0
The original parameters W_0 are frozen. They do not calculate or store gradients, reducing GPU memory by up to 90%.
8 LoRA Fine-Tuning Variants
Each variant targets one specific remaining inefficiency in plain LoRA — pick based on which resource (activation memory, base-model memory, trainable params, convergence speed, or accuracy) is your actual bottleneck.
| What's Trainable | Key Modification | Primary Benefit | |
|---|---|---|---|
| LoRA | A, B | Baseline low-rank update | Reduces trainable params from full W to A+B |
| LoRA-FA | B only | Freeze A | Further reduces activation memory |
| VeRA | Scaling vectors b, d | Shared frozen random A/B across layers | Massively reduces trainable params vs. per-layer A/B |
| Delta-LoRA | A, B, + delta on W | Also updates W | Closes the gap to full fine-tuning expressiveness |
| LoRA+ | A at lr₁, B at lr₂ | Differential learning rates | Faster / better convergence |
| LoRA-drop | A, B in high-impact layers only | Activation-based pruning | Cuts training cost with minimal accuracy loss |
| QLoRA | A, B, with W quantized | Quantized frozen base | Dramatically reduces base-model memory footprint (~75% at 8-bit) |
| DoRA | Magnitude m, direction V (separately) | Weight decomposition | Improved parameter efficiency & accuracy vs. plain LoRA |
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Explain how each of the 8 LoRA variants modifies the base LoRA technique and why
- •Compute the memory savings QLoRA achieves via quantization on a concrete example
- •Explain DoRA's magnitude/direction decomposition and why it improves on plain LoRA
- •Choose the right LoRA variant given a memory budget, layer-importance profile, or accuracy target
What is it?
This is a comparison of 8 distinct techniques in the LoRA family for parameter-efficient fine-tuning: the original LoRA (train two small low-rank matrices A and B alongside a frozen weight matrix W), LoRA-FA (freeze matrix A, only train B, saving activation memory), VeRA (freeze random A/B shared across all layers, train only tiny per-layer scaling vectors), Delta-LoRA (also update W itself using the delta between consecutive low-rank product steps), LoRA+ (use different learning rates for A and B), plus three bonus techniques: LoRA-drop (prune LoRA from low-impact layers), QLoRA (combine LoRA with weight quantization), and DoRA (decompose weights into magnitude and direction components, fine-tuning each separately).
Why it exists
Traditional full fine-tuning is infeasible for LLMs because these models have billions of parameters and are hundreds of gigabytes in size — not everyone has access to the computing infrastructure needed. The core LoRA technique already solves most of this by training only two small low-rank matrices instead of the full weight matrix, but researchers kept finding specific remaining inefficiencies — activation memory, the assumption every layer needs its own trainable matrices, no update to W at all, no learning-rate tuning between A and B, wasted training on low-impact layers, storage of full-precision W, and undifferentiated magnitude/direction updates — and each of these 8 variants exists to address one specific one of those remaining inefficiencies.
Problem it solves
Each variant solves a specific, narrow limitation still present in plain LoRA. LoRA-FA solves LoRA's remaining activation-memory cost by freezing matrix A entirely. VeRA solves the 'every layer needs its own A/B pair' redundancy by sharing frozen random matrices across all layers and training only tiny scaling vectors. Delta-LoRA solves the fact that plain LoRA never actually updates the original weight matrix W at all, by injecting a delta signal into W too. LoRA+ solves a convergence inefficiency by recognizing A and B don't need or benefit from the same learning rate. LoRA-drop solves the wasted-compute problem of training LoRA uniformly across every layer when many layers barely benefit. QLoRA solves the remaining large memory footprint of storing the frozen base model weights W themselves, via quantization. DoRA solves a representational limitation of LoRA's simple additive update by separately fine-tuning a weight's magnitude and direction.
Intuition
Think of the base model's weight matrix W as an enormous, expensive-to-move piece of furniture you're not allowed to actually reposition (frozen). LoRA is like attaching a small, cheap add-on shelf (matrices A and B) that captures the useful adjustments, without moving the furniture itself. Each variant is a further refinement of that shelf idea: LoRA-FA bolts one side of the shelf permanently in place, only letting the other side adjust. VeRA uses the exact same shelf design everywhere in the house, only letting you tweak a couple of small knobs per room. Delta-LoRA occasionally nudges the actual furniture slightly based on how the shelf has been adjusted. LoRA+ lets one side of the shelf turn faster than the other. LoRA-drop removes shelves from rooms where they're barely being used. QLoRA compresses the furniture itself to take up less space. DoRA lets you separately adjust how big something is (magnitude) versus which way it's facing (direction).
Analogy
If plain LoRA is 'add a small correction on top of a frozen model,' think of the 8 variants as 8 different ways to make that correction cheaper or smarter: LoRA-FA is like only paying for half the correction mechanism (freeze A). VeRA is like reusing the exact same correction template everywhere and only paying to customize a tiny dial per location. Delta-LoRA is like occasionally letting the correction bleed back into the original, not staying purely separate. LoRA+ is like letting one part of the correction mechanism move faster than the other because it turns out that converges better. LoRA-drop is like removing the correction entirely from places where it's not doing anything useful. QLoRA is like shrinking the original (frozen) object itself so the whole setup takes less space, correction included. DoRA is like separating 'how big' from 'which direction' when making the correction, rather than moving everything at once.
Technical explanation
(1) LoRA: for a pre-trained weight matrix W of dimensions d×k, instead of fine-tuning W directly, LoRA introduces a low-rank update ΔW = B×A, where A has dimensions d×r and B has dimensions r×k (r being the chosen rank, much smaller than d or k) — during training, only A and B are updated while W stays frozen, and at inference W_final = W + (alpha/r)(B×A).
(2) LoRA-FA freezes matrix A entirely and only updates matrix B, reducing the activation memory LoRA still requires despite its already-small parameter count.
(3) VeRA takes this further: instead of every layer having its own distinct, trainable A/B pair, VeRA shares frozen, randomly-initialized A and B matrices across all model layers, and the only trainable parameters are small, layer-specific scaling vectors (denoted b and d) — dramatically shrinking the trainable parameter count versus standard LoRA.
(4) Delta-LoRA additionally updates the frozen matrix W itself, not just A and B — specifically, the difference (delta) between the A×B product across two consecutive training steps is added to W, letting some signal flow back into the base weights despite them nominally being frozen.
(5) LoRA+ addresses a convergence inefficiency: standard LoRA updates both A and B with the same learning rate, but LoRA+'s authors found that using a higher learning rate for matrix B specifically leads to more optimal convergence.
(6, bonus) LoRA-drop observes that not all layers benefit equally from LoRA updates — it first adds LoRA to every layer and trains briefly, measures each layer's LoRA activation strength, and removes LoRA from layers whose activations stay near zero (minimal influence on output), reducing training cost and speeding up fine-tuning with little to no accuracy loss.
(7, bonus) QLoRA further addresses memory limitations by quantizing the frozen weight matrix W itself — representing its parameters with fewer bits (e.g., 8-bit or 4-bit instead of 32-bit float), which can reduce memory usage by roughly 75% at 8-bit, at the cost of a precision/size tradeoff that QLoRA's special techniques work to minimize.
(8, bonus) DoRA (Weight-Decomposed Low-Rank Adaptation) decomposes the pretrained weight matrix W into two separate components — magnitude (m) and direction (V) — and fine-tunes each independently, improving parameter efficiency and performance over plain LoRA's simple additive update by targeting these two aspects of a weight update separately.
Architecture
All 8 variants share the same underlying skeleton — a frozen base model W plus some additional trainable structure — but differ in exactly what that additional structure is and how much of it interacts with W. LoRA/LoRA-FA/LoRA+ keep W completely frozen and vary only what's trained in A/B. VeRA keeps A/B themselves frozen too, training only small scaling vectors on top. Delta-LoRA is the only variant that lets training signal flow back into W. QLoRA operates at a different layer of the stack entirely — it compresses W's storage representation, orthogonal to which LoRA variant runs on top. DoRA restructures the update itself into magnitude and direction rather than the standard low-rank additive form.
Workflow
- Start with plain LoRA as your baseline — it already provides most of the memory savings over full fine-tuning.
- If activation memory (not just parameter count) is your bottleneck, switch to LoRA-FA.
- If you're fine-tuning many similar tasks/layers and want to minimize trainable parameters even further, consider VeRA's shared frozen matrices.
- If plain LoRA's expressiveness ceiling is limiting your task performance, consider Delta-LoRA to let some signal reach W.
- If training seems slow to converge, try LoRA+'s differential learning rates for A and B — often a free win with no architecture change.
- Once you have a working LoRA setup, profile per-layer activation strength and consider LoRA-drop to prune low-impact layers and speed up subsequent training runs.
- If your GPU memory is the binding constraint (not just trainable parameter count), add QLoRA's quantization on top of whichever LoRA variant you're using.
- If you need the best possible accuracy within a LoRA-style budget and can afford the added complexity, evaluate DoRA's magnitude/direction decomposition against plain LoRA on your specific task.
Example
QLoRA memory-savings calculation for a 25M-parameter weight matrix
def memory_bytes(num_params: int, bits_per_param: int) -> float: return num_params * (bits_per_param / 8) # bytes
params = 25_000_000 fp32_bytes = memory_bytes(params, 32) # 100,000,000 bytes = ~0.1 GB int8_bytes = memory_bytes(params, 8) # 25,000,000 bytes = ~0.025 GB savings_pct = (1 - int8_bytes / fp32_bytes) * 100 # ~75% reduction
Simplified DoRA-style decomposition: separate magnitude and direction
import numpy as np
def decompose_weight(W: np.ndarray): magnitude = np.linalg.norm(W, axis=0, keepdims=True) # m direction = W / magnitude # V (unit vectors) return magnitude, direction
During DoRA fine-tuning: magnitude (m) and direction (V, itself LoRA-adapted)
are updated with separate, independent learning dynamics, rather than a single
combined additive update as in plain LoRA.
Real-world usage
QLoRA is the most widely adopted variant in practice, popularized by its original paper enabling fine-tuning of 65B-parameter models on a single consumer GPU, and is now the default fine-tuning approach in libraries like Unsloth and Hugging Face PEFT for anyone with limited GPU memory. LoRA+'s differential learning rate insight has been adopted as a near-free improvement in several fine-tuning libraries since it requires no architectural change, only a training-config tweak. VeRA and LoRA-drop are used in settings where extreme parameter efficiency matters — e.g., serving many different fine-tuned task variants from the same base model, where VeRA's shared-matrix design means each new task variant costs almost nothing in extra storage. DoRA has been adopted by teams chasing the last few percentage points of accuracy in LoRA-style fine-tuning where full fine-tuning-level performance is the target but the memory budget of full fine-tuning is not available.
Trade-offs
Simpler variants (plain LoRA, LoRA+) are easy to adopt with minimal added complexity and are a safe default. More specialized variants trade implementation complexity for a specific benefit: VeRA and LoRA-drop trade some accuracy/flexibility for extreme parameter efficiency; Delta-LoRA and DoRA trade added computational complexity for closing some of the expressiveness gap to full fine-tuning; QLoRA trades a small amount of precision for dramatic memory savings, which matters enormously when GPU memory (not just parameter count) is the actual bottleneck. There's no single best variant — the right choice depends on which specific resource (activation memory, base-model memory, trainable parameter count, wall-clock training time, or final accuracy) is your binding constraint.
Visual explanation
A comparison table with 8 rows (one per technique) and columns: Technique | What's Trainable | Key Modification | Primary Benefit. LoRA: [A, B] | baseline | reduces trainable params from full W to A+B. LoRA-FA: [B only] | freeze A | further reduces activation memory. VeRA: [scaling vectors b, d] | shared frozen random A/B | massively reduces trainable params vs. per-layer A/B. Delta-LoRA: [A, B, and a delta applied to W] | also updates W | closes the gap to full fine-tuning expressiveness. LoRA+: [A at lr₁, B at lr₂] | differential learning rates | faster/better convergence. LoRA-drop: [A, B in high-impact layers only] | activation-based pruning | reduces training cost with minimal accuracy loss. QLoRA: [A, B, with W quantized] | quantized frozen base | dramatically reduces base-model memory footprint. DoRA: [magnitude m, direction V, separately] | weight decomposition | improved parameter efficiency and performance vs. plain LoRA.
Advantages
- —
Provides 8 distinct, well-documented ways to further reduce memory or improve accuracy beyond plain LoRA, each targeting a different specific bottleneck
- —
QLoRA specifically has made fine-tuning very large models feasible on consumer-grade hardware
- —
LoRA+'s differential learning rate improvement requires zero architectural change, making it a nearly free win
- —
DoRA and Delta-LoRA narrow the accuracy gap to full fine-tuning while retaining most of LoRA's efficiency
Disadvantages
- —
More variants means more decisions to make and more configuration surface area versus plain LoRA
- —
Some variants (VeRA, LoRA-drop) trade some accuracy or flexibility for their efficiency gains
- —
QLoRA's quantization introduces a genuine precision/size tradeoff that can degrade output quality on sensitive tasks
- —
Newer, less battle-tested variants (Delta-LoRA, DoRA) have less production track record than plain LoRA or QLoRA
Common mistakes
- —
Defaulting to plain LoRA when activation memory or base-model memory (not trainable parameter count) is the actual bottleneck, missing LoRA-FA's or QLoRA's specific fix
- —
Assuming QLoRA's memory savings come free — the quantization/precision tradeoff can measurably affect output quality and should be validated on the specific task
- —
Not profiling per-layer activation strength before assuming LoRA-drop's pruning will help — the layers that matter vary by task and model
- —
Using the same learning rate for A and B (missing LoRA+'s free convergence improvement) without ever testing a differential rate
- —
Reaching for the newest, most complex variant (DoRA, Delta-LoRA) by default rather than starting from plain LoRA and adding complexity only when a specific bottleneck justifies it
📂 Subtopics
LoRA: Low-Rank Decomposition, Rank r, and Alpha
LoRA decomposes a weight update into two small matrices A and B, governed by two key hyperparameters — rank r (how much capacity the adaptation has) and alpha (how strongly it's applied).
~15 min
QLoRA: Quantizing the Frozen Base Model for Massive Memory Savings
QLoRA quantizes the frozen weight matrix W itself, not just A and B — representing parameters with fewer bits achieves roughly 75% memory reduction at 8-bit, with a real precision trade-off.
~15 min
DoRA: Decomposing Weights into Magnitude and Direction
DoRA separates a pretrained weight matrix into magnitude and direction components, fine-tuning each independently — a refinement that improves parameter efficiency and performance over plain LoRA's simple additive update.
~15 min
Choosing Rank r: Quality vs. Efficiency Trade-offs
The book establishes r as the rank hyperparameter controlling adaptation capacity — this subtopic extends that with practical guidance on picking r=4 vs r=8 vs r=16 for a real fine-tuning task.
~12 min