Rohan Paul
@rohanpaul_ai
New Anthropic paper finds that when a monitor catches only some reward hacks, penalizing the flagged ones teaches models to hack in ways the monitor misses, while routing those flagged updates into a deletable adapter removes the hacking instead.
With a penalty, Qwen3-8B dropped the hack the monitor could see and started removing test constraint checks on about 26% of answers.
With GRAFT, updates from flagged samples go only to a separate "forget" adapter that is deleted after training. Hacking fell to nearly zero, and clean solve rate rose about 16 points over the base model. A Qwen3-32B trained this way never hacked, even when prompted to.
If your hack detector is imperfect, use it to decide where the model learns, not what it gets rewarded for.
