RL for Language Models: How RLHF Uses RL, and the Connection to GRPO

The previous two subtopics covered RL in the abstract (dog training, generic algorithms); this subtopic makes the mapping onto LLMs completely concrete, since that mapping is genuinely not obvious the first time you see it.

In the LLM setting, the AGENT is the language model itself. The POLICY is exactly the model's learned behavior — its weights define a mapping from 'what's the current text so far' (the state) to 'what token comes next' (the action), expressed as a probability distribution over the vocabulary (recalling the softmax output from the probability-basics prerequisite topic). Generating an entire response is really a whole SEQUENCE of actions — one token at a time — making an LLM's generation process structurally a genuine RL problem, not just a metaphor for one.

RLHF (Reinforcement Learning from Human Feedback) applies this framework directly. The reward signal comes from human preference judgments — historically, humans compare pairs of model outputs and say which is better, and this preference data trains a separate 'reward model' that learns to predict, for any given output, roughly how much a human would like it. That reward model's score THEN becomes the reward signal used to update the LLM's policy via an RL algorithm — historically PPO (previous subtopic) — nudging the model's weights so it becomes more likely to generate outputs the reward model scores highly, and less likely to generate ones it scores poorly. This is exactly the RLHF branch of the sft-vs-rft topic's decision tree — the path taken when a task isn't automatically verifiable, so humans must supply the preference signal instead of a deterministic reward function.

GRPO (Group Relative Policy Optimization), covered extensively in the sft-vs-rft and rl-environments-for-agents topics, is a more recent variant built for the OPPOSITE case — tasks that ARE automatically verifiable (math, logic, code correctness), where you don't need a learned reward MODEL trained on human preferences at all; a deterministic reward FUNCTION (does the answer match, does the code pass its tests) does the job directly and far more cheaply. The 'Group Relative' part of its name reflects a specific algorithmic choice: rather than needing a separately-trained value function (unlike classic PPO), GRPO generates a GROUP of candidate outputs for the same prompt and uses their RELATIVE rewards within that group as the training signal — simpler to implement and cheaper to run than the full PPO-plus-reward-model pipeline RLHF traditionally uses.

The unifying picture: whether the reward comes from a human-trained reward model (RLHF) or a deterministic checker (GRPO/RFT), the underlying loop is the same RL cycle from the first subtopic — the model (agent/policy) generates text (takes actions), gets scored (reward), and updates its weights (policy improvement) to favor higher-reward generations going forward.

# Mapping RL vocabulary onto LLM text generation explicitly, and
# contrasting RLHF's reward-MODEL path with GRPO's reward-FUNCTION path.

def llm_policy_generate_token(context: str, vocab_probs: dict) -> str:
    """The LLM AS a policy: state (context so far) -> action (next token),
    expressed as a probability distribution (softmax, from probability-basics)."""
    import random
    tokens, probs = list(vocab_probs.keys()), list(vocab_probs.values())
    return random.choices(tokens, weights=probs)[0]

def rlhf_reward_model(output: str, learned_human_preferences: dict) -> float:
    """RLHF's reward: a MODEL trained on human preference comparisons,
    used because 'is this response good' isn't automatically checkable."""
    return learned_human_preferences.get(output, 0.5)   # predicted human preference score

def grpo_reward_function(output: str, correct_answer: str) -> float:
    """GRPO's reward: a DETERMINISTIC checker, used because the task
    (does the math answer match) IS automatically verifiable."""
    return 1.0 if output.strip() == correct_answer.strip() else 0.0

def grpo_group_relative_reward(group_outputs: list[str], correct_answer: str) -> list[float]:
    """The 'Group Relative' part of GRPO: reward a GROUP of candidate
    outputs for the same prompt, relative to each other -- no separately
    trained value function needed, unlike classic PPO."""
    raw_rewards = [grpo_reward_function(o, correct_answer) for o in group_outputs]
    mean_reward = sum(raw_rewards) / len(raw_rewards)
    return [r - mean_reward for r in raw_rewards]   # relative to the group's average

next_token = llm_policy_generate_token("The capital of France is", {" Paris": 0.85, " London": 0.15})
print(f"Policy (LLM) chose action (token): {next_token!r}")

human_prefs = {"That's a great question, let me explain clearly.": 0.9}
print("RLHF reward (from a learned reward MODEL):",
      rlhf_reward_model("That's a great question, let me explain clearly.", human_prefs))

group = ["4", "5", "4", "3"]
print("GRPO group-relative rewards:", grpo_group_relative_reward(group, correct_answer="4"))