ART (Agent Reinforcement Trainer): Training Agentic LLMs from Trajectories
~13 min read
ART, built by OpenPipe, is purpose-built for training LLM agents whose 'actions' are multi-step reasoning traces and tool calls, not simple game moves — it wraps your existing agent with minimal changes and applies GRPO to trajectory-level rewards.
ART (Agent Reinforcement Trainer): Training Agentic LLMs from Trajectories is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •LLM agents complicate RL because their 'actions' are multi-step reasoning traces and tool calls, not simple discrete moves from a small fixed set
- •ART (built by OpenPipe) is an open-source framework purpose-built for training agentic LLMs from experience — running trajectories, capturing steps, scoring, and updating the model
- •Its architecture wraps your EXISTING agent with minimal changes via a lightweight client, communicating with an ART training server that manages rollouts and optimization
- •ART supports GRPO applied at the TRAJECTORY level, learning from whole-trajectory rewards rather than token-level labels — essential since you can't label individual tool-call decisions as 'correct' in isolation
- •Workflow: agent runs unchanged -> produces a trajectory -> reward function scores it -> GRPO updates the policy -> repeat, gradually improving behaviors like planning, correction, and tool use