Hands-On: Fine-Tuning a Reasoning Model with Unsloth and HuggingFace TRL

This course's hands-on GRPO walkthrough uses two specific libraries together: UnslothAI for efficient fine-tuning, and HuggingFace TRL to actually apply GRPO — turning any base model into a reasoning powerhouse without any labeled data or human intervention, using a 5-step process.

Step 1, load the model: this course loads Qwen3-4B-Base and its tokenizer using Unsloth (any other open-weight LLM could substitute here — Qwen3-4B-Base is this course's specific choice, not a GRPO requirement). Step 2, define the LoRA config: rather than fine-tuning the entire model's weights (which GRPO-style RL training makes even more expensive than ordinary fine-tuning, since it requires generating and scoring multiple candidates per prompt), this course uses Unsloth's PEFT/LoRA integration, specifying the base model, the LoRA rank r, and which modules to actually adapt.

Step 3, create the dataset: this course loads the Open R1 Math dataset (a collection of math problems) and formats it for reasoning — each sample includes a system prompt enforcing structured reasoning (the 'think step by step' framing from the 'why GRPO' subtopic), the actual question, and the expected answer in the format the reward functions expect to check against. Step 4, define reward functions: this is exactly the 4 deterministic functions covered in the previous subtopic (match format exactly, match format approximately, check the answer, check numbers) — no manual labeling needed anywhere in this pipeline.

Step 5, use GRPO and start training: HuggingFace TRL provides everything the GRPO algorithm needs out of the box, in the form of GRPOConfig (training hyperparameters) and GRPOTrainer (the actual training loop, handling group generation, reward aggregation, and the GRPO loss described in the earlier subtopic). This course's own comparison after training shows GRPO genuinely turning a base model into a meaningfully stronger reasoner — concrete evidence that this specific Unsloth+LoRA+TRL+4-reward-function pipeline works as described, not just a theoretical recipe.

# The book's actual 5-step GRPO recipe, using Unsloth + HuggingFace TRL.
from unsloth import FastLanguageModel
from trl import GRPOConfig, GRPOTrainer
from datasets import load_dataset

# Step 1: load the model (Qwen3-4B-Base, or any other open-weight LLM)
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen3-4B-Base", max_seq_length=2048, load_in_4bit=True,
)

# Step 2: LoRA config — avoids fine-tuning the full model, especially
# important since GRPO generates+scores MULTIPLE candidates per prompt
model = FastLanguageModel.get_peft_model(
    model, r=16, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)

# Step 3: dataset — Open R1 Math, formatted with a reasoning-focused
# system prompt ("think step by step") plus question + expected answer
dataset = load_dataset("open-r1/OpenR1-Math-220k", split="train")

# Step 4: reward functions — the 4 deterministic functions from the
# previous subtopic (match_format_exactly, match_format_approximately,
# check_answer, check_numbers)
reward_funcs = [match_format_exactly, match_format_approximately, check_answer, check_numbers]

# Step 5: train with GRPOConfig + GRPOTrainer — TRL handles group
# generation, reward aggregation, and the GRPO loss internally
config = GRPOConfig(output_dir="./grpo-reasoning-model", num_generations=8, learning_rate=5e-6)
trainer = GRPOTrainer(model=model, args=config, train_dataset=dataset, reward_funcs=reward_funcs)
trainer.train()