Key Algorithms: Q-Learning, Policy Gradient, and PPO (Intuition Only)

The previous subtopic's vocabulary (policy, value function) describes WHAT gets learned; this subtopic covers three major algorithm families for HOW that learning actually happens, at an intuitive level with no deep math required.

Q-learning learns action VALUES, and picks actions indirectly by always choosing whichever action currently looks best. It maintains a 'Q-value' for every (state, action) pair — an estimate of how much total future reward you'd expect if you took THIS action in THIS state, and played optimally afterward. Training repeatedly nudges these estimates toward reality: try an action, see what reward and next-state you actually got, and adjust the Q-value estimate slightly toward that observed outcome (this is exactly the policy.update() step in the previous subtopic's code example, generalized to full state-action pairs rather than just actions). Once Q-values are reasonably accurate, the POLICY becomes trivial: in any state, just pick whichever action has the highest Q-value. Q-learning works well for problems with a manageable number of distinct states and actions, but struggles when actions are CONTINUOUS (like a precise steering angle) rather than a small discrete menu.

Policy Gradient methods take a fundamentally different approach: instead of learning action VALUES and picking the best one indirectly, they learn the POLICY directly — a function that outputs action probabilities given a state, adjusted through training to make higher-reward actions more probable and lower-reward actions less probable. This directness is what makes Policy Gradient methods naturally suited to CONTINUOUS action spaces (instead of comparing Q-values across infinite possible actions, you just sample directly from a probability distribution) and to the kind of high-dimensional, structured action spaces LLM text generation involves (the 'action' is which token to generate next, from a huge vocabulary) — this is exactly the algorithm family GRPO (covered in sft-vs-rft) belongs to.

PPO (Proximal Policy Optimization) addresses a real weakness of naive Policy Gradient methods: training can be unstable, because a large, well-intentioned policy update can sometimes accidentally make the policy dramatically WORSE, and there's no built-in safeguard against that in the basic approach. PPO's core intuition, true to its name, is to keep each update 'proximal' — close to the previous policy — by explicitly limiting (clipping) how much the policy is allowed to change in any single update step. This trades some raw learning speed for much greater training stability, which is exactly why PPO became the dominant algorithm behind RLHF (next subtopic) for years, and why GRPO (a more recent alternative, covered extensively in sft-vs-rft) was specifically designed as a simpler, cheaper alternative to PPO for LLM fine-tuning.

# Contrasting Q-learning (learn VALUES, pick the best indirectly) with
# a Policy Gradient-style approach (learn action PROBABILITIES directly),
# on the same tiny 2-action problem.

import random

ACTIONS = ["A", "B"]
TRUE_REWARDS = {"A": 0.3, "B": 0.8}   # unknown to the agent -- it must learn this

def get_reward(action: str) -> float:
    return TRUE_REWARDS[action] + random.gauss(0, 0.1)   # noisy reward signal

# --- Q-learning style: learn a VALUE per action, act greedily ---
q_values = {a: 0.0 for a in ACTIONS}
for _ in range(200):
    action = max(q_values, key=q_values.get) if random.random() > 0.2 else random.choice(ACTIONS)
    reward = get_reward(action)
    q_values[action] += 0.1 * (reward - q_values[action])   # nudge toward observed reward

print("Q-learning learned values:", {k: round(v, 2) for k, v in q_values.items()})
print("Q-learning's chosen action:", max(q_values, key=q_values.get))

# --- Policy Gradient style: learn action PROBABILITIES directly ---
policy_probs = {"A": 0.5, "B": 0.5}   # starts uniform, no notion of "value" at all
for _ in range(200):
    action = random.choices(ACTIONS, weights=[policy_probs[a] for a in ACTIONS])[0]
    reward = get_reward(action)
    # Increase this action's probability proportional to how good the reward was
    policy_probs[action] += 0.02 * reward * (1 - policy_probs[action])
    other = "B" if action == "A" else "A"
    policy_probs[other] = 1 - policy_probs[action]

print("Policy Gradient learned probabilities:", {k: round(v, 2) for k, v in policy_probs.items()})