Reward Signal Design: 4 Deterministic Reward Functions

GRPO's reliance on 'deterministic reward functions' isn't an abstract idea — this course's actual hands-on implementation defines 4 concrete, specific reward functions used together to validate each generated response and assign a reward, with no manual labeling required at all.

The 4 reward functions: (1) match format exactly — checks whether the response precisely follows the required structural format (e.g., reasoning wrapped in specific tags, followed by a clearly-marked final answer); (2) match format approximately — a softer version of the same check, giving partial credit for responses that are close to the required format but not letter-perfect, rather than an all-or-nothing pass/fail on formatting alone; (3) check the answer — the core correctness check, comparing the response's final answer against the known ground truth for that problem; and (4) check numbers — a more granular check specifically for numerical answers, likely allowing for reasonable numeric equivalence (e.g., '0.5' matching '1/2') rather than requiring exact string matching that would incorrectly penalize a mathematically correct but differently-formatted answer.

Using multiple reward functions together, rather than a single pass/fail correctness check, is a deliberate design choice: it lets the training signal reward partial progress (getting the format right even if the final answer is still wrong, which matters early in training) and reward genuine correctness precisely (via the answer and number checks), rather than a single binary signal that would give the model no useful gradient when it's directionally correct but not yet exactly right.

This multi-function design is exactly what makes GRPO's reward signal 'deterministic' in a meaningful sense: none of these 4 checks require a human judgment call or a learned reward model — each is a straightforward, automatable comparison (does this string match this pattern? does this number equal that number?), which is precisely what makes RFT methods like GRPO able to scale training without labeled data or human-in-the-loop review at all.

import re

def match_format_exactly(response: str) -> float:
    """Full credit only if the exact required structure is present."""
    pattern = r"<reasoning>.*?</reasoning>\s*<answer>.*?</answer>"
    return 1.0 if re.search(pattern, response, re.DOTALL) else 0.0

def match_format_approximately(response: str) -> float:
    """Partial credit for close-but-imperfect formatting."""
    has_reasoning_tag = "<reasoning>" in response or "<answer>" in response
    return 0.5 if has_reasoning_tag else 0.0

def check_answer(response: str, ground_truth: str) -> float:
    """Core correctness check against the known ground truth."""
    match = re.search(r"<answer>(.*?)</answer>", response, re.DOTALL)
    if not match:
        return 0.0
    return 2.0 if match.group(1).strip() == ground_truth.strip() else 0.0

def check_numbers(response: str, expected_number: float, tolerance: float = 1e-3) -> float:
    """Numeric equivalence, not exact string matching — '0.5' should
    match '1/2' style differences in formatting."""
    numbers = re.findall(r"-?\d+\.?\d*", response)
    if not numbers:
        return 0.0
    return 1.0 if any(abs(float(n) - expected_number) < tolerance for n in numbers) else 0.0

def total_reward(response: str, ground_truth: str, expected_number: float) -> float:
    return (
        match_format_exactly(response) + match_format_approximately(response)
        + check_answer(response, ground_truth) + check_numbers(response, expected_number)
    )