RL Fundamentals: Agent, Environment, State, Action, Reward, Policy, Value

Imagine training a dog to sit. You don't hand the dog a rulebook explaining exactly how to bend its legs — you watch what it does, and when it happens to sit, you give it a treat. Over many repetitions, the dog learns 'sitting leads to treats' purely through trial and reward, without ever being told the rule directly. Reinforcement learning (RL) formalizes exactly this learning-through-reward process, with precise vocabulary for each piece.

The agent is the learner — the dog, or in AI terms, the algorithm being trained. The environment is everything the agent interacts with and has no direct control over — the room, you, the treats. The state is a snapshot of the current situation the agent can observe — is the dog sitting or standing right now, is a treat currently visible. An action is something the agent can DO from a given state — sit, stand, bark, walk away. A reward is a number telling the agent how good or bad an action turned out to be — a treat is a positive reward; being ignored is roughly a zero or negative one.

Two more concepts round out the vocabulary, and they're the actual OBJECTS being learned. A policy is the agent's strategy — a mapping from 'what state am I in' to 'what action should I take.' Training doesn't hand-code this mapping; it's exactly what gets learned through experience. Early in training, the dog's policy is basically random (it doesn't know sitting matters); after enough reward signal, its policy shifts toward 'when a human says sit, sit' because that policy reliably earns reward. A value function estimates how good a given state (or state-action pair) is EXPECTED to be in the long run — not just the immediate reward, but the total reward the agent can expect to accumulate from here onward, given how it's currently behaving. This distinction matters because sometimes the best action isn't the one with the biggest immediate reward — a chess move that sacrifices a piece now might have high VALUE if it sets up a winning position later, even though its immediate reward looks bad.

Put together: an agent, in some state, uses its policy to choose an action, receives a reward (and moves to a new state) from the environment, and over many repetitions of this loop, updates its policy — informed by value estimates — to favor actions and states that lead to more reward over time. This same seven-term vocabulary is exactly what the rl-environments-for-agents topic's OpenEnv framework formalizes into a standard software interface, and what the sft-vs-rft topic's RFT relies on when applying this loop to LLM training specifically.

# The dog-training analogy, implemented as a minimal RL vocabulary
# demonstration -- agent/environment/state/action/reward/policy/value.
import random

STATES = ["standing", "sitting"]
ACTIONS = ["sit", "stay_standing"]

def environment_step(state: str, action: str) -> tuple[str, float]:
    """The environment: given a state and action, returns the new state
    and a reward."""
    if action == "sit":
        return "sitting", 1.0     # a treat!
    return "standing", 0.0        # no treat

class SimpleDogPolicy:
    """The policy: a mapping from state -> action, which gets UPDATED
    based on which actions historically earned reward."""
    def __init__(self):
        self.action_values = {a: 0.0 for a in ACTIONS}   # a simple value estimate per action

    def choose_action(self) -> str:
        # Early on (all values equal) this is essentially random;
        # as training progresses, it favors the higher-value action
        return max(self.action_values, key=self.action_values.get)

    def update(self, action: str, reward: float, learning_rate: float = 0.3):
        """Nudge the value estimate toward observed reward -- this IS
        the learning: the policy improves purely from reward feedback."""
        self.action_values[action] += learning_rate * (reward - self.action_values[action])

policy = SimpleDogPolicy()
state = "standing"
for trial in range(10):
    action = random.choice(ACTIONS) if trial < 3 else policy.choose_action()
    new_state, reward = environment_step(state, action)
    policy.update(action, reward)
    state = new_state

print("Learned action values (the policy's 'preferences'):", policy.action_values)
print("Policy now prefers:", policy.choose_action())