Reward Shaping for Language Agents: From Format Checks to Trajectory Rewards
~13 min read
Designing a good reward function is the hardest part of RFT/ART-style training. The book's own GRPO reward functions show a practical pattern — layer cheap, deterministic checks — that generalizes to trajectory-level agent rewards.
Reward Shaping for Language Agents: From Format Checks to Trajectory Rewards is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •The book's own GRPO reward functions layer 4 separate checks (format-exact, format-approximate, answer, numbers) rather than one binary signal, giving smoother partial credit
- •The same layering principle extends to trajectory-level agent rewards: outcome rewards (sparse, fires at the end), process rewards (denser, scores intermediate steps)
- •Sparse outcome-only rewards give the model little information about WHICH steps in a long trajectory actually helped — process rewards help address this
- •Efficiency penalties discourage unnecessarily long or redundant trajectories, mirroring the book's 'match format' checks rewarding the right SHAPE, not just correct content
- •Prefer automatically-checkable, hard-to-game reward signals over subjective ones — the same reward-hacking risk from the sft-vs-rft tradeoffs subtopic applies directly here