ART (Agent Reinforcement Trainer): Training Agentic LLMs from Trajectories

The previous subtopic's OpenEnv gives you a standard interface for environments; this subtopic covers a framework that addresses a DIFFERENT, LLM-specific complication: reinforcement learning becomes more complex when the 'agent' is an LLM. Instead of choosing a simple action like moving left or right, an LLM agent produces multi-step reasoning traces, tool calls, conversations and plans — a single 'action' in this setting might be an entire multi-paragraph response, or a sequence of several tool calls with reasoning interspersed between them, not one discrete move from a small fixed set.

Training such agents requires a system that can collect these trajectories, assign rewards and update the model reliably. ART (Agent Reinforcement Trainer), built by OpenPipe, provides exactly that system. It is an open-source framework designed specifically for training agentic LLMs from experience. ART handles the pieces that are difficult to engineer manually: running the agent to generate full trajectories, capturing decisions, tool use and reasoning steps, scoring each trajectory with a custom reward function, and updating the model using reinforcement learning.

The architecture: ART uses a lightweight client that wraps your existing agent with minimal changes. The client communicates with an ART training server, which manages rollouts, reward computation, batching and optimization. This 'wrap, don't rewrite' design matters practically — you don't need to redesign your agent's code around ART's requirements; ART adapts to whatever agent you've already built and observes its behavior from the outside.

A key feature is ART's support for GRPO — the same algorithm from the grpo-reasoning and sft-vs-rft topics, here applied at the TRAJECTORY level rather than the single-answer level. GRPO allows the model to learn from trajectory-level rewards rather than token-level labels, which is essential for improving behaviors like planning, correction and tool use — you can't meaningfully assign a 'correct token' label to an agent's decision to call a particular tool at a particular moment in a multi-step task, but you CAN score the entire resulting trajectory (did the task ultimately succeed, was it efficient, were the tool calls appropriate) and let GRPO propagate that signal backward across the whole sequence of decisions that led there.

The full workflow, per this course: you start with your existing agent code — ART simply wraps it so you don't need to rewrite anything. The agent runs and produces a trajectory. The trajectory is scored using a reward function. ART applies GRPO (or another supported RL method) to update the policy. The loop repeats, gradually improving the agent's behavior. By handling rollout execution, reward processing and policy optimization, ART lets developers focus on designing effective reward signals and agent strategies rather than building RL infrastructure — the same underlying goal as OpenEnv (previous subtopic), applied to the harder, LLM-specific problem of trajectory-based agentic behavior rather than single discrete actions.

# Illustrating the ART workflow: wrap an existing agent (unchanged),
# run it to produce a TRAJECTORY (not a single action), score the
# whole trajectory, and conceptually feed that into GRPO.

class ExistingAgent:
    """Your agent, written with NO awareness of ART -- this is the
    'don't rewrite anything' point the book makes."""
    def run(self, task: str) -> list[dict]:
        """Produces a TRAJECTORY: a sequence of reasoning steps and
        tool calls, not one discrete action."""
        return [
            {"type": "reasoning", "content": f"Breaking down: {task}"},
            {"type": "tool_call", "tool": "search", "args": {"query": task}},
            {"type": "tool_result", "content": "found 3 relevant results"},
            {"type": "final_answer", "content": f"Answer for: {task}"},
        ]

class ARTClient:
    """A lightweight wrapper -- runs the agent, captures the full
    trajectory, and scores it with a custom reward function."""
    def __init__(self, agent, reward_fn):
        self.agent = agent
        self.reward_fn = reward_fn

    def collect_trajectory(self, task: str) -> dict:
        trajectory = self.agent.run(task)          # unmodified agent, run as-is
        reward = self.reward_fn(trajectory)        # trajectory-level reward, not per-token
        return {"task": task, "trajectory": trajectory, "reward": reward}

def task_success_reward(trajectory: list[dict]) -> float:
    """Reward the WHOLE trajectory based on outcome + efficiency --
    exactly the kind of signal that can't be assigned to any single token."""
    reached_answer = any(step["type"] == "final_answer" for step in trajectory)
    efficient = len(trajectory) <= 5
    return 1.0 if (reached_answer and efficient) else 0.3

client = ARTClient(ExistingAgent(), task_success_reward)
result = client.collect_trajectory("Find the capital of France")
print(f"trajectory length: {len(result['trajectory'])}, reward: {result['reward']}")
# ART's training server would then apply GRPO across many such
# trajectories to update the underlying policy (the LLM's weights)