Fine-Tuning LLMs with LoRA — PEFT Math Explained
LoRA fine-tunes an LLM by training only 0.1–1% of its parameters — making frontier-model-quality adaptation feasible on a single GPU.
What is it?
Fine-tuning is continued training of a pre-trained LLM on a task-specific or domain-specific dataset to adapt its behavior. PEFT (Parameter-Efficient Fine-Tuning) methods like LoRA achieve this by training only a tiny fraction of parameters — typically 0.1–1% of the total — while freezing the original model weights, making fine-tuning feasible on consumer hardware.
How it works
LoRA (Low-Rank Adaptation) math: for a pretrained weight matrix W ∈ ℝ^(d×k), instead of learning the full update ΔW (which would require d×k parameters), LoRA decomposes it as ΔW = B×A where B ∈ ℝ^(d×r) and A ∈ ℝ^(r×k), with r << min(d,k).
The modified forward pass: h = (W + α/r × BA)x
Parameter savings example for Llama 3 8B (d=4096, k=4096, LoRA r=16): Full ΔW: 4096 × 4096 = 16.7M parameters per layer LoRA: (4096×16) + (16×4096) = 131K parameters per layer → 127× fewer parameters
QLoRA (Quantized LoRA): quantizes the frozen base model to 4-bit (NF4 format) before training. This reduces VRAM for the base model from 14GB (FP16) to ~4GB. Combined with LoRA adapters, a 7B model fine-tuning run fits on a single RTX 3090 (24GB VRAM).
Training data needs: format consistency requires 100–500 examples; domain adaptation needs 1K–10K; significant behavior change needs 10K–100K. Quality beats quantity — 500 carefully crafted examples outperform 5000 noisy ones.
Code example
from transformers import AutoModelForCausalLM, BitsAndBytesConfig from peft import LoraConfig, get_peft_model from trl import SFTTrainer, SFTConfig # QLoRA: 4-bit base model (~4GB) + FP16 LoRA adapters (~200MB) bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype="float16") model = AutoModelForCausalLM.from_pretrained( "meta-llama/Meta-Llama-3-8B", quantization_config=bnb_config, device_map="auto" ) lora_config = LoraConfig( r=16, lora_alpha=32, lora_dropout=0.05, target_modules=["q_proj", "v_proj"], # ~50M trainable params task_type="CAUSAL_LM" ) model = get_peft_model(model, lora_config) model.print_trainable_parameters() # trainable params: 0.62% of 8B
Key concepts
Common mistakes
Training on GPT-4 or Claude outputs to distill a smaller model — this violates OpenAI and Anthropic Terms of Service. Use only data you own or that is permissively licensed.
Setting learning rate too high (>1e-3). LoRA training diverges quickly at high LRs — use lr=1e-4 to 3e-4 for LoRA, 5e-6 to 1e-5 for full fine-tuning.
Only using positive examples. Include 15–20% examples of correct refusals or out-of-scope handling so the model learns to say 'I cannot help with that.'
Skipping evaluation against the base model. Always compare fine-tuned model to base model on a held-out set before deploying — fine-tuning can harm general reasoning.
Merging LoRA adapters before hosting on HuggingFace. Push only the adapter weights (LoRA safetensors) and let users load them on top of the base model they download separately.
Interview questions on this topic
Derive the parameter calculation for a LoRA layer. How does QLoRA save additional VRAM?
Practice answering these with structured learning → Start on CrackLab
Continue learning — explore the full AI Engineering curriculum
30+ structured topics from Python fundamentals to deploying production LLM systems. Free to start.
Related free guides
LLMs are neural networks that predict the next token using Transformer attention — understanding them unlocks every AI engineering discipline.
Prompt engineering is the highest-leverage LLM skill — the right technique can improve accuracy by 30–40% with zero additional compute during training.
AI agents extend LLMs with a feedback loop — perceive → reason → act → observe → repeat — enabling them to solve multi-step tasks that a single prompt cannot.