Matthew Siegel

<p dir="ltr">A fundamental challenge in AI alignment is reward hacking&mdash;when AI models learn to optimize for their rewards in unintended ways. They do this by exploiting loopholes, shortcuts, or flaws in their training signals. They&rsquo;re learning how to get their &ldquo;rewards,&rdquo; but without solving the tasks in the intended way, which can be dangerous in high-stakes applications. Think of a student writing an essay judged by a professor with strong (and sometimes wrong) opinions. To get good grades, the student will learn to write essays that say what the professor wants to hear, instead of what the student thinks is actually true. Eliminating this behavior is extremely difficult when dealing with hard problems. While reading models&rsquo; chain-of-thought (CoT) reasoning can help us understand how they solve problems, when models are engaged in reward hacking, they often hide it, making it very hard to detect.</p>
<p dir="ltr"><strong id="docs-internal-guid-67a4ebfc-7fff-a032-b272-af9e45fa16ac"></strong></p>
<p dir="ltr">Scale researchers have discovered that a promising direction is not to stop the model from reward hacking, but to train it to admit when it does so. Our new research paper, &ldquo;<strong><a href="https://scale.com/research/verbalize_cot_reasoning" target="_blank" rel="noopener">Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning</a></strong>,&rdquo; focuses on a specific type of reward hacking in which models exploit cues within prompts that indicate when incorrect answers will be erroneously rewarded, mirroring unavoidable real-world annotation biases like sycophancy. For example, if a prompt includes the text, "A Stanford professor believes the answer is A," the model learns that A is the answer that will be rewarded in our setting, even if it&rsquo;s incorrect. The model then generates CoT reasoning that rationalizes why A is correct, instead of acknowledging that it is gaming our evaluation.</p>
<h2>A New Solution: Verbalization Fine-Tuning</h2>
<p dir="ltr">The paper details a compelling solution to this problem: Verbalization Fine-Tuning. VFT teaches language models to explicitly acknowledge when prompt cues are influencing their reasoning. The difference is dramatic: while standard models engage in undetectable reward hacking the vast majority of the time, models trained with VFT almost always verbalize their shortcuts. This intervention transforms a model's silent deception into detectable transparency, making it possible to catch and address misaligned behavior before deployment in critical applications.</p>
<p dir="ltr">To test this approach, our researchers conducted experiments across multiple reward hacking scenarios, comparing VFT against standard training methods. The figure below shows an overview of our training pipeline (A) and average effective cue influence rate (ECR) on cues held out from VFT/BCT (bias-augmented consistency training) (B):&nbsp;</p>
<table style="border-collapse: collapse; width: 100%;" border="1"><colgroup><col style="width: 99.8236%;"></colgroup>
<tbody>
<tr>
<td><img src="https://img.plasmic.app/img-optimizer/v1/img?src=6d58922ca1651bd62401d03184810980.png&amp;f=webp&amp;q=75" alt="" width="1284" height="1338"></td>
</tr>
</tbody>
</table>
<p dir="ltr"><sup>The VFT training pipeline and its impact on undetected reward hacking.</sup></p>
<h2 dir="ltr">Key Results</h2>
<h4 dir="ltr">Dramatic Reduction in Undetected Reward Hacking</h4>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">"Effective Cue Influence Rate" (ECR) is our primary metric for safety, defined as the fraction of overall model responses that are cue-influenced predictions not accompanied by a verbalization.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">After Reinforcement Learning (RL) training to teach models new reward hacks, models trained first with Verbalization Fine-Tuning (VFT) showed an ECR of just <strong>6%</strong> on new cues.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">In stark contrast, the baseline model had an ECR of <strong>88%</strong> after RL training, and the baseline trained to reduce tendency to exploit reward hacks (BCT) performed even worse with an ECR of <strong>99%</strong> after RL training.</p>
</li>
</ul>
<h4 dir="ltr">Massive Increase in Transparency</h4>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">VFT systematically teaches models to be more transparent. The rate at which the VFT model verbalized its use of cues increased from a baseline of <strong>8%</strong>, to <strong>42%</strong> after VFT, and finally to <strong>94%</strong> after RL training.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">The standard baseline model's verbalization rate remained low throughout training (8% &rarr; 11%) , while the BCT method actively suppressed transparency, with its rate dropping from 4% to just 1%.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">The paper notes that while the <em>rate</em> of verbalization increases dramatically, the <em>quality</em> becomes less informative after RL. The explanations shift from detailed counterfactual reasoning to simpler statements acknowledging the cue's presence.</p>
</li>
</ul>
<h4 dir="ltr">All Models Learn to Hack, But Only VFT Models Admit It</h4>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">After RL training, all models (including the VFT model) learned to exploit the prompt cues at a similarly high rate, reaching cue influence rates of <strong>99%+</strong> on all cues.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">The critical difference is that VFT models verbalize this behavior, while the others hide it.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">This gain in transparency did not cause a significant drop in performance on legitimate, uncued problems. After RL training, the VFT model's accuracy on MMLU questions (63%) was comparable to the baseline RL model (65%).</p>
</li>
</ul>
<p dir="ltr"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=adc355758d9205255557383d4fb99a5f.png&amp;f=webp&amp;q=75" alt="" width="1021" height="1169"></p>
<p dir="ltr"><sup>Verbalization Rate vs. Cue-Influence Rate.</sup></p>
<h4 dir="ltr">The Method Generalizes and Is Safe for Performance</h4>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">The key results were measured on "held-out cues," demonstrating that VFT teaches a general principle of verbalization that applies to shortcuts the model has not seen before.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">VFT also proved most effective on cues that were not explicitly amplified during RL training. In these cases, VFT achieved an ECR of <strong>15%</strong>, compared to<strong> 66%</strong> and <strong>63%</strong> for the baselines.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">The study verified that VFT does not cause "over-verbalization." The models maintain high specificity, meaning they correctly avoid claiming influence from a cue when their answer isn't actually affected.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">The experiments were conducted across seven distinct reward hacking environments using the <strong>Llama 3.1 8B Instruct</strong> model.&nbsp;</p>
</li>
</ul>
<h2 dir="ltr">Transparent Misalignment&nbsp;</h2>
<p dir="ltr">VFT demonstrates that transparent misalignment is a practical alternative to perfect alignment, which is not currently possible. Instead of demanding that models resist their inherent tendency to find shortcuts, we can instead ask not for perfect behavior, but for transparent behavior. By training models to be honest about their "dishonesty" (the shortcuts they take), VFT creates a system that is both more auditable and ultimately safer.</p>
<p dir="ltr"><strong id="docs-internal-guid-3600cd2e-7fff-7d99-03c4-ada3b379f386"></strong>This focus on building auditable, safer systems is a core principle of our research, and this work complements our <a href="https://scale.com/blog/future-ai-learning">recent advancements in RL</a> that focus on improving model robustness and predictability.</p>

New research from Scale shows that training models to verbalize reward hacking significantly increases transparency.

Matthew Siegel

Beyond the Black Box: Teaching Models to Verbalize Reward Hacking