Choosing Rank r: Quality vs. Efficiency Trade-offs

Note on sourcing: this course precisely defines what rank r controls (the previous subtopic covers this directly), but doesn't spell out a specific r=4-vs-8-vs-16 comparison table — this subtopic extends this course's own rank/alpha framing with practical selection guidance that's standard LoRA practice, consistent with (not contradicting) what this course establishes about what r actually controls.

The core trade-off, following directly from what r determines: since both LoRA matrices A and B have one dimension sized by r, and the number of trainable parameters scales with r (roughly linearly — (d × r) + (r × k) grows proportionally as r grows), a smaller r means fewer trainable parameters, faster training, lower memory usage, but also less CAPACITY for the adaptation to represent a complex behavioral change. A larger r means more trainable parameters (though still far fewer than full fine-tuning), more capacity to represent nuanced adaptations, but more compute, more memory, and — past a certain point — diminishing returns, since LoRA's whole premise is that most fine-tuning updates don't actually need full-rank expressiveness to be effective.

In practice, small ranks (r=4 to r=8) tend to work well for narrower adaptations — adjusting tone, format compliance, or a fairly contained domain vocabulary shift, the kind of task where the model mostly already 'knows' what to do and just needs a light nudge. Medium ranks (r=16 to r=32) suit more substantial behavioral changes — genuinely new task capabilities, or adapting to a domain meaningfully different from the base model's training distribution. Very high ranks start to erode LoRA's core efficiency advantage without reliably buying proportional quality gains, since at that point you're approaching the parameter budget of full fine-tuning anyway while still being constrained by the low-rank assumption's structural limits.

The practical recommendation: start with a small-to-moderate r (8 or 16 are common defaults), monitor validation performance, and increase r specifically if you observe the model underfitting the target behavior — plateauing at a quality level that's clearly below what the task should be achievable at. Rarely is it worth guessing a large r upfront 'to be safe,' since that sacrifices most of LoRA's efficiency benefit before you've confirmed a smaller rank was actually insufficient.

def lora_capacity_tradeoff(d: int, k: int, ranks: list[int]) -> None:
    """Illustrating how trainable parameter count scales with r —
    the concrete basis for the quality/efficiency trade-off."""
    full_finetune_params = d * k
    for r in ranks:
        lora_params = (d * r) + (r * k)
        pct_of_full = lora_params / full_finetune_params * 100
        print(f"r={r:>3}: {lora_params:>10,} trainable params "
              f"({pct_of_full:.3f}% of full fine-tuning)")

lora_capacity_tradeoff(d=4096, k=4096, ranks=[4, 8, 16, 32, 64])

# Practical selection heuristic — start small, increase only on
# observed underfitting, matching the book's alpha/r framing
def suggest_starting_rank(task_type: str) -> int:
    guidance = {
        "tone_or_format_adjustment": 8,      # narrow, light-touch adaptation
        "domain_vocabulary_shift": 8,
        "new_task_capability": 16,            # more substantial behavioral change
        "significant_domain_shift": 32,
    }
    return guidance.get(task_type, 8)  # default to a small, efficient starting point