When to Use Which: Pre-training vs. Post-training Distillation Choices

With all three techniques covered, the practical question is which one to actually reach for. Three factors drive the decision: weight access to the teacher, memory budget, and which training STAGE you're distilling at.

Distillation in LLMs can happen at two stages, and this choice narrows the options immediately. At the pre-training stage, you train the bigger teacher LLM and the smaller student LLM together — this is exactly co-distillation's use case, and it's what Llama 4 did to train Scout and Maverick. At the post-training stage, you train the bigger teacher LLM first, completely, and THEN distill its already-finished knowledge into a smaller student — this is where soft-label and hard-label distillation apply, and it's what DeepSeek did distilling DeepSeek-R1 into Qwen and Llama 3.1. You can also combine both — apply distillation during both stages — which is what Gemma 3 did.

Within the post-training branch, the choice between soft-label and hard-label distillation comes down to weight access and memory budget. If you have full weight access to the teacher AND the memory budget to store full probability distributions across your training corpus, soft-label distillation gives you the richest knowledge transfer. If either of those conditions doesn't hold — no weight access to a closed teacher, or the storage requirement is simply infeasible at your corpus scale (which, per the earlier worked example, it very often is) — hard-label distillation is the practical choice, trading some transfer fidelity for a dramatically smaller footprint.

A simple decision path: are you training the student ALONGSIDE the teacher from scratch, or distilling from an already-finished teacher? Alongside → co-distillation. Already-finished → do you have weight access and a feasible memory budget for full soft labels? Yes → soft-label distillation. No → hard-label distillation.

def choose_distillation_technique(
    teacher_already_trained: bool,
    have_teacher_weight_access: bool,
    corpus_tokens: int,
    vocab_size: int,
    memory_budget_gb: float,
) -> str:
    """A concrete decision path across the 3 techniques, based on
    training stage, weight access, and memory budget."""
    if not teacher_already_trained:
        return "co-distillation"  # pre-training stage — train both together

    # Post-training stage — teacher is already finished
    if not have_teacher_weight_access:
        return "hard-label distillation"  # closed teacher — no distribution access anyway

    soft_label_gb = (corpus_tokens * vocab_size * 1) / (1024 ** 3)  # float8 estimate
    if soft_label_gb <= memory_budget_gb:
        return "soft-label distillation"  # richest transfer, budget allows it
    return "hard-label distillation"  # budget doesn't allow full soft labels

print(choose_distillation_technique(
    teacher_already_trained=True, have_teacher_weight_access=True,
    corpus_tokens=5_000_000_000_000, vocab_size=100_000, memory_budget_gb=10_000,
))  # -> "hard-label distillation" — soft labels would need ~500M GB, far over budget