Fine-Tuning: LoRA & PEFT Math
Learn parameter-efficient fine-tuning, LoRA matrix arithmetic, rank selection, and SFT dataset generation.
Fine-Tuning Controls:
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Grasp the LoRA matrix multiplication math: $W = W_0 + rac{alpha}{r} (B imes A)$
- •Explain how rank parameter $r$ determines parameter sizes
- •Write a clean dataset pipeline for supervised fine-tuning
What is it?
Fine-tuning is continued training of a pre-trained LLM on a task-specific or domain-specific dataset to adapt its behavior. PEFT (Parameter-Efficient Fine-Tuning) methods like LoRA achieve this by training only a tiny fraction of parameters — typically 0.1–1% of the total — while freezing the original model weights, making fine-tuning feasible on consumer hardware.
Why it exists
Full fine-tuning of a 70B model requires ~560GB of GPU memory for optimizer states alone — completely impractical without a large cluster. PEFT methods emerged to democratize fine-tuning: LoRA achieves comparable task-specific performance by training only ~50M parameters instead of 70B, reducing memory requirements from hundreds of GB to 10–20GB.
Problem it solves
There are two scenarios where prompting alone is insufficient: (1) the model needs to produce output in a very specific format consistently (JSON schemas, proprietary templates), and (2) the domain vocabulary or reasoning style is so specialized that lengthy few-shot examples consume most of the context window. Fine-tuning bakes the behavior directly into the weights, solving both without token overhead.
Intuition
Think of a pre-trained LLM as a brilliant generalist employee on their first day. They know everything about the world but nothing about your company's specific processes. Fine-tuning is the onboarding program — you show them 500–10,000 examples of how your company does things, and they internalize that style without forgetting their general knowledge.
If you come from Java/Spring Boot: LoRA adapters are like writing custom @Interceptors on top of a framework — you do not modify Spring's source code (frozen base weights), you layer your behavior on top via small adapter matrices. The framework's JAR stays unchanged; your few-KB adapter customizes the behavior for your application.
If you come from React/Frontend: think of LoRA as a CSS override stylesheet. The base model is the browser's default stylesheet (huge, covers everything). Your LoRA adapter is a small custom.css file that overrides specific behaviors. The cascade applies — your rules win, the base handles everything else.
Analogy
LoRA is like adding sticky notes to a textbook. The textbook (base model) contains all human knowledge — you do not reprint it to customize it. Instead you add targeted sticky notes (low-rank matrices) at specific pages (attention layers) that redirect or supplement the original text. The book stays the same; your annotations reshape how a reader interprets it.
Technical explanation
LoRA (Low-Rank Adaptation) math: for a pretrained weight matrix W ∈ ℝ^(d×k), instead of learning the full update ΔW (which would require d×k parameters), LoRA decomposes it as ΔW = B×A where B ∈ ℝ^(d×r) and A ∈ ℝ^(r×k), with r << min(d,k).
The modified forward pass: h = (W + α/r × BA)x
Parameter savings example for Llama 3 8B (d=4096, k=4096, LoRA r=16): Full ΔW: 4096 × 4096 = 16.7M parameters per layer LoRA: (4096×16) + (16×4096) = 131K parameters per layer → 127× fewer parameters, per targeted layer
Typical LoRA targets Q and V attention projections in all layers. With r=16 on a 7B model: ~50M trainable params (0.7% of total).
QLoRA (Quantized LoRA): quantizes the frozen base model to 4-bit (NF4 format) before training. This reduces VRAM for the base model from 14GB (FP16) to ~4GB. Combined with LoRA adapters, a 7B model fine-tuning run fits on a single RTX 3090 (24GB VRAM).
Training data needs: format consistency requires 100–500 examples; domain adaptation needs 1K–10K; significant behavior change needs 10K–100K. Quality beats quantity — 500 carefully crafted examples outperform 5000 noisy ones.
Architecture
LoRA training architecture:
Base model (frozen, FP16 or INT4): [W_q] [W_k] [W_v] [W_o] per attention layer [W_1] [W_2] per FFN layer
LoRA adapters (trainable, tiny): For each targeted W: A ∈ ℝ^(r×k), initialized with random Gaussian B ∈ ℝ^(d×r), initialized to zeros → ΔW=0 at start
Forward pass: h = W·x + (α/r)·B·A·x
Training: only A and B receive gradients Optimizer: AdamW, lr=2e-4 Batch: 4–8 sequences per step with gradient accumulation
Post-training: W_merged = W + (α/r)·B·A Merged model has zero inference overhead vs base
Workflow
- Prepare dataset: collect 500–10K examples in instruction format: {"instruction": "...", "input": "...", "output": "..."}
- Quality filter: remove examples with outputs under 50 chars, with hallucinations, or mismatched input/output
- Choose base model: Llama 3 8B for <24GB VRAM, Llama 3 70B for >80GB or multi-GPU
- Configure LoRA: LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj","v_proj"], lora_dropout=0.05)
- Configure QLoRA (if VRAM-constrained): load_in_4bit=True with BitsAndBytesConfig
- Train: SFTTrainer from TRL library, 1–3 epochs, monitor loss for convergence without overfitting
- Evaluate: compare against base model and a GPT-4 baseline on 100 held-out examples
- Merge: model.merge_and_unload() fuses LoRA weights into base model (zero inference overhead)
- Export: save as safetensors for HuggingFace or GGUF for llama.cpp deployment
Example
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig from peft import LoraConfig, get_peft_model from trl import SFTTrainer, SFTConfig
QLoRA: 4-bit base model (~4GB) + FP16 LoRA adapters (~200MB)
bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype="float16") model = AutoModelForCausalLM.from_pretrained( "meta-llama/Meta-Llama-3-8B", quantization_config=bnb_config, device_map="auto" )
lora_config = LoraConfig( r=16, lora_alpha=32, lora_dropout=0.05, target_modules=["q_proj", "v_proj"], # ~50M trainable params bias="none", task_type="CAUSAL_LM" ) model = get_peft_model(model, lora_config) model.print_trainable_parameters() # trainable params: 0.62% of 8B
trainer = SFTTrainer(model=model, train_dataset=dataset, args=SFTConfig(output_dir="./lora", num_train_epochs=2, per_device_train_batch_size=4)) trainer.train()
Real-world usage
Mistral AI fine-tuning API: used by European enterprises (banking, legal) to adapt Mistral 7B for domain-specific tasks while keeping data on EU infrastructure. LoRA fine-tuning costs ~$2–5 per 1M training tokens vs $8 for GPT-3.5 fine-tuning.
Meta LLaMA commercial fine-tunes: companies like Databricks (DBRX) and Teknium (Hermes series) have released LoRA fine-tuned Llama models for coding, reasoning, and instruction following — with performance competitive with GPT-3.5 at zero inference cost on own hardware.
Medical AI: MedLLaMA and BioMedGPT are LoRA fine-tunes of Llama/BioGPT on medical literature and clinical notes. Fine-tuned models score 85%+ on USMLE Step 1 vs ~60% for base Llama. Training cost: ~$50–200 on a single A100 for 7B models.
Trade-offs
LoRA rank r: r=4 is highly efficient (few params) but limited capacity — good for format/style changes. r=64 approaches full fine-tuning quality at much lower cost. Start with r=16; increase to r=32 or r=64 if validation loss plateaus early.
Fine-tuning vs few-shot prompting: fine-tuning is worth it when (a) you need consistent structured outputs the prompt cannot guarantee, (b) task-specific prompt would consume 2K+ tokens per request at scale, or (c) you need <1B token model performance on specific task. For everything else, prompting is faster and cheaper to iterate.
Catastrophic forgetting: fine-tuning on narrow tasks can degrade general capabilities. Mix 10–20% general instruction-following examples into your fine-tuning data to preserve base model abilities.
Visual explanation
LoRA Adapter weight updates: Frozen Weight W0 (d x d) ──(Parallel path)──> Low-rank matrices A (d x r) and B (r x d) ──(Multiply)──> Add updates to output
Advantages
- —
99% VRAM savings during training
- —
Zero execution latency when adapters are merged
- —
Enables rapid hot-swapping of tasks
- —
1,000 carefully curated examples can achieve significant quality improvement over base model
Disadvantages
- —
Adapters must match base architecture
- —
Tuning hyperparameters (rank, alpha) requires expertise
- —
Risk of catastrophic forgetting on general capabilities if training data is too narrow
Common mistakes
- —
Training on GPT-4 or Claude outputs to distill a smaller model — this violates OpenAI and Anthropic Terms of Service. Many published "fine-tuned" models use this approach illegally. Use only data you own or that is permissively licensed (ShareGPT, FLAN, OpenHermes with attribution).
- —
Setting learning rate too high (>1e-3). LoRA training diverges quickly at high LRs — you will see loss spike after a few hundred steps, then the model outputs garbage. Use lr=1e-4 to 3e-4 for LoRA, 5e-6 to 1e-5 for full fine-tuning.
- —
Only using positive examples. Fine-tuning on what the model should say without including refusal examples means the model never learns to say "I cannot help with that." Include 15–20% examples of correct refusals or out-of-scope handling.
- —
Skipping evaluation against the base model. Always compare fine-tuned model to base model on a held-out set before deploying. Fine-tuning improves task metrics but can harm general reasoning, instruction following, or safety behaviors.
- —
Merging LoRA adapters before hosting on HuggingFace. If you merge and push the full model, you distribute the base model weights (may violate Llama license). Instead push only the adapter weights (LoRA safetensors) and let users load them on top of the base model they download separately.
🎤 Interview questions
Derive the parameter calculation for a LoRA layer. How does QLoRA save additional VRAM?