Reward-Hacking Model Organisms

Second Author on (Lin et al., 2025) as part of MARS (Cambridge AI Safety Hub) (Jul – Oct 2025), released as a Preprint.

Abstract

Studying reward-hacking monitors requires model organisms: constructed model systems that exhibit failures similar to those the monitors are meant to catch. Two common construction recipes are synthetic backdoors and capability-pressure RL, in which a model is optimized against a misspecified reward on tasks it often cannot solve. We evaluate the capability-pressure recipe in code generation and report a bounded negative result. Using GRPO with QLoRA on Qwen2.5-Coder-7B and CodeContests, we train seven RL variants, include a no-RL base reference, and evaluate four non-executing monitors. Difficulty and adversarial reward both increase explicit hardcoding: holding reward and public-test setup fixed, input-conditional hardcoding rises from 19.8% on mixed-difficulty problems to 82.6% on hard-only problems, and a malicious hard-only reward yields 76.0% hardcoding. However, neither construction knob produces the target failure: reward exploitation that is hard to identify by static inspection. Across the eight variants and a 64-category behavior taxonomy, generations remain dominated by explicit hardcode-with-fallback, template-wrapped lookup, and default-value catch-all patterns. Monitor transfer shows the consequence: a monitor trained on synthetic backdoors is weaker than real-RL monitors on core RL-hacking variants, while real-RL monitors fail to transfer back to held-out synthetic backdoors. Together, these results caution that current model-organism recipes provide useful baselines for obvious reward hacking, but do not yet produce the subtle, hard-to-inspect failures that reward-hacking monitors are meant to evaluate.