Reinforcement Learning Foundations
Learn agents, states, actions, policy optimization, and reward signals used to align models in RLHF and GRPO.
Fine-Tuning Controls:
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Understand the relationship between Environment, Agent, Actions, and Rewards
- •Explain Policy and how it dictates agent decisions
- •Describe how reward signals reinforce optimal decisions
What is it?
[AI Engineering Prerequisite] Reinforcement Learning (RL) is an area of machine learning concerned with how intelligent agents take actions in an environment to maximize cumulative reward — taught here only in the context of AI Engineering (RLHF and GRPO).
Why it exists
Some tasks do not have simple "correct" answers for SFT training. RL lets agents learn optimal policies through trial, error, and feedback.
Problem it solves
Solves complex multi-step optimization problems: models learn to plan and align their behaviors with human goals.
Intuition
RL is training a dog: you give it a treat (reward) when it sits (action) in response to your command (state). The dog learns to sit more often.
Analogy
RL is like playing a video game: you try different moves, get points (rewards), and learn what strategies get high scores.
Technical explanation
For AI Engineering, three RL concepts matter:
-
Reward signal: a scalar score that evaluates how good an action was. In RLHF a reward model scores how human-preferred the LLM response is.
-
Policy: the function (the LLM itself) that maps a given state (prompt) to an action (token sequence). The policy is what gets updated during RLHF fine-tuning.
-
How RLHF uses RL: a pre-trained reward model scores LLM outputs for human preference; policy gradient updates shift the LLM weights to generate higher-scored responses — without needing labeled answers for every case.
-
GRPO (Group Relative Policy Optimization): samples a group of responses, computes relative advantages (which responses scored above average), and updates the policy to make high-advantage responses more probable — without a separate critic model, making it cheaper than PPO.
Architecture
Consists of an Agent Policy (actor), an Environment State space, an Action space, and a Reward Function that scores transitions.
Workflow
- Observe state -> 2. Sample action -> 3. Apply action -> 4. Receive reward -> 5. Optimize parameters -> 6. Repeat.
Example
state = "equation: 2x = 4" action = "x = 2" reward = 1.0 if evaluate(action) else -1.0
Policy updates to increase probability of action under similar state
Real-world usage
Using RL algorithms in DeepSeek-R1 to train models to format chain-of-thought inside <think> tags.
Trade-offs
Model-free RL requires massive trial volumes but is easier to implement than model-based RL.
Visual explanation
Reinforcement Learning Loop: Agent ──(Action)──> Environment ──(State & Reward)──> Agent adjusts Policy
Advantages
- —
Can learn complex multi-step strategies
- —
Does not require labelled instruction pairs for every logical step
Disadvantages
- —
Highly unstable training curves
- —
Vulnerable to reward hacking (agent finding shortcuts to exploit reward calculation)
Common mistakes
- —
Not scaling rewards correctly, leading to gradients exploding
- —
Allowing the reward function to evaluate intermediate steps incorrectly, causing reward hacking
🎤 Interview questions
What is the Exploration-Exploitation tradeoff in Reinforcement Learning? How does temperature control it?
📂 Subtopics
RL Fundamentals: Agent, Environment, State, Action, Reward, Policy, Value
The core vocabulary of reinforcement learning, explained through a simple dog-training analogy — agent, environment, state, action, reward, policy, and value function.
~13 min
Key Algorithms: Q-Learning, Policy Gradient, and PPO (Intuition Only)
Three foundational RL algorithm families, explained through intuition rather than equations: learning action VALUES (Q-learning), learning a policy DIRECTLY (Policy Gradient), and learning stably (PPO).
~13 min
RL for Language Models: How RLHF Uses RL, and the Connection to GRPO
Mapping RL's abstract vocabulary onto LLM training concretely: the model IS the policy, generating text IS taking actions, and human/reward-function feedback IS the reward signal RLHF and GRPO both optimize against.
~12 min
Practical RL Today: Where It's Actually Used in AI Engineering
Beyond theory, RL shows up in specific, identifiable places in modern AI engineering — model alignment, reasoning fine-tuning, agent training, and recommendation/ranking systems — each already covered elsewhere in this curriculum.
~11 min