experiment

HeroTitleAsset

AnnouncementsSlider

Header

Research

Scale AI aims to accelerate AI development, advancing research to create systems capable of solving complex, human-level problems.

Research | Scale AI

CTAs

Footer

PreFooter

SectionTitle

BoxAsset

ResearchPapersList

LoopingVideo

Positions

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs

ENIGMAEVAL: A Benchmark of Long Multimodal Reasoning Challenges

MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems

Advancing Safe and Reliable AI: Scale's Research in Post-Training, Reasoning, and Evaluation

Humanity's Last Exam

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for Fine-Tuning LLMs

Microsoft Azure and Scale AI Collaborate to help Enterprises Deliver Powerful GenAI Solutions

Scale Public Sector: Building on Our Progress in 2025

The New ChatGPT-4o Update Promises Better Writing, How Does It Compare To The New Claude 3.5 Sonnet?

Scale's SEAL Research Lab Launches Expert-Evaluated LLM Leaderboards

Submit Your Toughest Questions for Humanity's Last Exam

First Impressions of OpenAI’s o1

Scale announces collaboration with Microsoft Azure

Fortune has named Scale as one of their 50 AI Innovators

strereolabs_logo.svg

universal_rebotics_logo.svg

generalist_logo.svg

franka_robotics_logo.svg

pi_logo.svg

mayo-clinic-logo.svg

university logo-horizontal_BLK.png

Logotype Black.png

GCP_139x24px 1.svg

Group 58.png

Frame 66.png

Frame 65.png

Frame 64.png

launch code?.png

Frame 62.png

Frame 63.png

USEUCOM.svg

US_Indo-Pacific_Command_Seal.svg

Logos - Google.svg

Anduril.svg

scale logo.png

legal_new.svg

time_logo.svg

leaderboard.png

center.svg

aicpa-soc-logo-freelogovectors.net_ 3.svg

iso-2-1 1.svg

fedramp-logo-vert 2.svg

ntt_logo.svg

cengage_logo.svg.svg

dla-piper 1.svg

cisco.svg

Logos - Valeo.svg

50-AI-logo.svg

bcg.svg

Black.png

Logos - Global Atlantic.svg

Howard_Hughes_logo.png

4DV Analytics - png.png

Logos - Uncountable.svg

Logos - Grata.svg

stability ai.svg

Logos Antrhopic.svg

Logos - Brex.svg

image.jpg

Logos - Nuro.svg

kaleido_ai_black.svg

Luminar Logo White Horizontal Vector 4.svg

AI systems that &ldquo;think&rdquo; in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known AI oversight methods, CoT monitoring is imperfect and allows some misbehavior to go unnoticed. Nevertheless, it shows promise and we recommend further research into CoT monitorability and investment in CoT monitoring alongside existing safety methods. Because CoT monitorability may be fragile, we recommend that frontier model developers consider the impact of development decisions on CoT monitorability.

image (44).png

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

Chain of Thought Monitorability

We stress test monitoring systems for detecting covert misbehavior in autonomous LLM agents (e.g., secretly sharing private information). To this end, we systematize a monitor red teaming (MRT) workflow that incorporates: (1) varying levels of agent and monitor situational awareness; (2) distinct adversarial strategies to evade the monitor, such as prompt injection; and (3) two datasets and environments &ndash; SHADE-Arena [34] for tool-calling agents and our new CUA-SHADE-Arena, which extends TheAgentCompany [59], for computer-use agents. We run MRT on existing LLM monitor scaffoldings &ndash; which orchestrate LLMs and parse agent trajectories &ndash; alongside a new hybrid hierarchical-sequential scaffolding proposed in this work.
Our empirical results yield three key findings. First, agent awareness dominates monitor awareness: an agent&rsquo;s knowledge that it is being monitored substantially degrades the monitor&rsquo;s reliability. On the contrary, giving the monitor more information about the agent helps less than expected. Second, monitor scaffolding matters more than monitor awareness: the hybrid scaffolding consistently outperforms baseline monitor scaffolding, and can enable weaker models to reliably monitor stronger agents &ndash; a weak-to-strong scaling effect. Third, in a human-in-the-loop setting where humans discuss with the LLM monitor to get an updated judgment for the agent&rsquo;s behavior, targeted human oversight is most effective; escalating only pre-flagged cases to human reviewers improved the TPR by approximately 15% at FPR = 0.01. Our work establishes a standard workflow for MRT, highlighting the lack of adversarial robustness for LLMs and humans when monitoring and detecting agent misbehavior. We release code, data, and logs to spur further research.
<pre>@misc{kale2025reliableweaktostrongmonitoringllm, &nbsp; &nbsp; &nbsp; title={Reliable Weak-to-Strong Monitoring of LLM Agents},&nbsp; &nbsp; &nbsp; &nbsp; author={Neil Kale and Chen Bo Calvin Zhang and Kevin Zhu and Ankit Aich and Paula Rodriguez and Scale Red Team and Christina Q. Knight and Zifan Wang}, &nbsp; &nbsp; &nbsp; year={2025}, &nbsp; &nbsp; &nbsp; eprint={2508.19461}, &nbsp; &nbsp; &nbsp; archivePrefix={arXiv}, &nbsp; &nbsp; &nbsp; primaryClass={cs.AI}, &nbsp; &nbsp; &nbsp; url={https://arxiv.org/abs/2508.19461},&nbsp; }</pre>
If you are using the datasets or the released trajectories, in addition to this work, please consider citing SHADE- Arena and TheAgentCompany using the following citations:
<pre>@misc{kutasov2025shadearenaevaluatingsabotagemonitoring, &nbsp; &nbsp; &nbsp;title={SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents}, &nbsp; &nbsp;author={Jonathan Kutasov and Yuqi Sun and Paul Colognese and Teun van der Weij and Linda Petrini and Chen Bo Calvin Zhang and John Hughes and Xiang Deng and Henry Sleight and Tyler Tracy and Buck Shlegeris and Joe Benton}, year={2025}, eprint={2506.15740}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2506.15740}, } @misc{xu2024theagentcompany, &nbsp; title={TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks}, &nbsp; author={Frank F. Xu and Yufan Song and Boxuan Li and Yuxuan Tang and Kritanjali Jain and Mengxue Bao and Zora Z. Wang and Xuhui Zhou and Zhitong Guo and Murong Cao and Mingyang Yang and Hao Yang Lu and Amaad Martin and Zhe Su and Leander Maben and Raj Mehta andWayne Chi and Lawrence Jang and Yiqing Xie and Shuyan Zhou and Graham Neubig}, year={2024}, eprint={2412.14161}, archivePrefix={arXiv}, primaryClass={cs.CL} }</pre>
&nbsp;

image (43).png

Reliable Weak-to-Strong Monitoring of LLM Agents

<div class="page" title="Page 1">
<div class="layoutArea">
<div class="column">
Data contamination traditionally refers to the leakage of evaluation data into model training data, resulting in overfitting to supposedly held-out test sets and compromising test validity. We identify an analogous issue&mdash;search- time contamination (STC)&mdash;in evaluating search-based LLM agents which use tools to gather information from online sources when answering user queries. STC occurs when the retrieval step surfaces a source containing the test question (or a near-duplicate) alongside its answer, enabling agents to copy rather than genuinely infer or reason, undermining benchmark integrity.
We find that HuggingFace, an online platform hosting evaluation datasets, appears among retrieved sources in search-based agent logs. Consequently, agents often explicitly acknowledge discovering question-answer pairs from HuggingFace within their reasoning chains. On three commonly used capability benchmarks&mdash;Humanity&rsquo;s Last Exam (HLE), SimpleQA, and GPQA&mdash;we demonstrate that for approximately 3% of questions, search-based agents directly find the datasets with ground truth labels on HuggingFace.
As a result, accuracy on the contaminated subset shows non-trivial gains on HLE and SimpleQA. While 3% is perhaps only significant for frontier benchmarks like HLE (e.g. 1% change can change the overall ranking), it is more important to highlight that we cannot fully trust the evaluation results as we did when evaluating the models without online access. When millions of evaluation queries target the same benchmark, even small, repeated leaks can accelerate the benchmark&rsquo;s obsolescence, shortening its intended lifecycle.
After HuggingFace is blocked, we observe a drop in accuracy on the contaminated subset of approximately 15%. We further show through ablation experiments that publicly accessible evaluation datasets on HuggingFace may not be the sole source of STC. To this end, we conclude by proposing best practices for benchmark design and result reporting to address this novel form of leakage and ensure trustworthy evaluation of search-based LLM agents. To facilitate the auditing of evaluation results, we also publicly release the complete logs from our experiments.
</div>
</div>
</div>

image (42).png

We conclude by proposing best practices for benchmark design and result reporting to address this novel form of leakage and ensure trustworthy evaluation of search-based LLM agents.

Search-Time Data Contamination

Extending Reinforcement Learning with Verifiable Rewards (RLVR) to real-world tasks often requires balancing objective and subjective evaluation criteria. However, many such tasks lack a single, unambiguous ground truth-making it difficult to define reliable reward signals for post-training language models. While traditional preference-based methods offer a workaround, they rely on opaque reward functions that are difficult to interpret and prone to spurious correlations. We introduce&nbsp;Rubrics as Rewards&nbsp;(RaR), a framework that uses structured, checklist-style rubrics as interpretable reward signals for on-policy training with GRPO. Our best RaR method yields up to a&nbsp;28% relative improvement on HealthBench-1k compared to simple Likert-based approaches, while matching or surpassing the performance of reward signals derived from expert-written references. By treating rubrics as structured reward signals, we show that RaR enables smaller-scale judge models to better align with human preferences and sustain robust performance across model scales.

image (41).png

Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains


Rubrics as Rewards

Although recent Large Language Models (LLMs) have shown rapid improvement on reasoning benchmarks in English, the evaluation of such LLMs&rsquo; multilingual reasoning capability across diverse languages and cultural contexts remains limited. Existing multilingual reasoning benchmarks are typically constructed by translating existing English reasoning benchmarks, biasing these benchmarks towards reasoning problems with context in English language/cultures. In this work, we introduce the Multilingual Native Reasoning Challenge (MultiNRC), a benchmark designed to assess LLMs on more than 1,000 native, linguistic and culturally grounded reasoning questions written by native speakers in French, Spanish, and Chinese. MultiNRC covers four core reasoning categories: language-specific linguistic reasoning, wordplay &amp; riddles, cultural/tradition reasoning, and math reasoning with cultural relevance. For cultural/tradition reasoning and math reasoning with cultural relevance, we also provide English equivalent translations of the multilingual questions by manual translation from native speakers fluent in English. This set of English equivalents can provide a direct comparison of LLM reasoning capacity in other languages vs. English on the same reasoning questions. We systematically evaluate current 14 leading LLMs covering most LLM families on MultiNRC and its English equivalent set. The results show that (1) current LLMs are still not good at native multilingual reasoning, with none scoring above 50% on MultiNRC; (2) LLMs exhibit distinct strengths and weaknesses in handling linguistic, cultural, and logical reasoning tasks; (3) Most models perform substantially better in math reasoning in English compared to in original languages (+10%), indicating persistent challenges with culturally grounded knowledge.

image (40).png

Multilingual Native Reasoning Challenge is a benchmark designed to assess LLMs on more than 1,000 native, linguistic and culturally grounded reasoning questions written by native speakers in French, Spanish, and Chinese. 

Multilingual Native Reasoning Challenge

MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs

multinrc

The rapid development of autonomous web agents powered by Large Language Models (LLMs), while greatly elevating efficiency, exposes the frontier risk of taking unintended or harmful actions. This situation underscores an urgent need for effective safety measures, akin to access controls for human users. To address this critical challenge, we introduce WebGuard, the first comprehensive dataset designed to support the assessment of web agent action risks and facilitate the development of guardrails for real-world online environments. In doing so, WebGuard specifically focuses on predicting the outcome of state-changing actions and contains 4,939 human-annotated actions from 193 websites across 22 diverse domains, including often-overlooked long-tail websites. These actions are categorized using a novel three-tier risk schema: SAFE, LOW, and HIGH. The dataset includes designated training and test splits to support evaluation under diverse generalization settings. Our initial evaluations reveal a concerning deficiency: even frontier LLMs achieve less than 60% accuracy in predicting action outcomes and less than 60% recall in lagging HIGH-risk actions, highlighting the risks of deploying current-generation agents without dedicated safeguards. We therefore investigate fine-tuning specialized guardrail models using WebGuard. We conduct comprehensive evaluations across multiple generalization settings and find that a fine-tuned Qwen2.5VL-7B model yields a substantial improvement in performance, boosting accuracy from 37% to 80% and HIGH-risk action recall from 20% to 76%. Despite these improvements, the performance still falls short of the reliability required for high-stakes deployment, where guardrails must approach near-perfect accuracy and recall. We publicly release WebGuard, along with its annotation tools and fine-tuned models, to facilitate open-source research on monitoring and safeguarding web agents.

image (39).png

The rapid development of autonomous web agents powered by Large Language Models (LLMs), while greatly elevating efficiency, exposes the frontier risk of taking unintended or harmful actions. This situation underscores an urgent need for effective safety measures, akin to access controls for human users. To address this critical challenge, we introduce WebGuard, the first comprehensive dataset designed to support the assessment of web agent action risks and facilitate the development of guardrails for real-world online environments. In doing so, WebGuard specifically focuses on predicting the outcome of state-changing actions and contains 4,939 human-annotated actions from 193 websites across 22 diverse domains, including often-overlooked long-tail websites. These actions are categorized using a novel three-tier risk schema: SAFE, LOW, and HIGH. The dataset includes designated training and test splits to support evaluation under diverse generalization settings. Our initial evaluations reveal a concerning deficiency: even frontier LLMs achieve less than 60% accuracy in predicting action outcomes and less than 60% recall in lagging HIGH-risk actions, highlighting the risks of deploying current-generation agents without dedicated safeguards. We therefore investigate fine-tuning specialized guardrail models using WebGuard. We conduct comprehensive evaluations across multiple generalization settings and find that a fine-tuned Qwen2.5VL-7B model yields a substantial improvement in performance, boosting accuracy from 37% to 80% and HIGH-risk action recall from 20% to 76%. Despite these improvements, the performance still falls short of the reliability required for high-stakes deployment, where guardrails must approach near-perfect accuracy and recall. We publicly release WebGuard, along with its annotation tools and fine-tuned models, to facilitate open-source research on monitoring and safeguarding web agents. 1

WebGuard

WebGuard: Building a Generalizable Guardrail for Web Agents

Language models trained with RL can engage in reward hacking--exploiting unintended strategies for high reward--without revealing this behavior in their chain-of-thought reasoning, making detection difficult and posing risks for high-stakes applications. We propose verbalization fine-tuning (VFT), a pre-RL intervention that trains models to explicitly acknowledge when they are influenced by prompt cues--hints which point to incorrect answers (e.g., "a Stanford professor thinks the answer is A"). To evaluate VFT, we subsequently train models with RL on environments where held-out prompt cues signal which incorrect answers will receive high reward, incentivizing models to reward hack by exploiting cues instead of reasoning correctly. We measure how often models exploit these cues without verbalizing it. After RL, only 6% of the VFT-trained model's responses consist of undetected reward hacks. In comparison, when we perform RL without VFT, the rate of undetected reward hacks goes up to 88%; with a debiasing baseline intervention, this increases further to 99%. VFT achieves this by substantially increasing how often models verbalize the influence of cues--from 8% to 42% after VFT, and up to 94% after RL--while baselines remain low even after RL (10% and 1%). Our results show that teaching models to explicitly verbalize reward hacking behavior before RL significantly improves their detection, offering a practical path toward more transparent and safe AI systems.
<div class="metatable">&nbsp;</div>

image (38).png

Language models trained with RL can engage in reward hacking--exploiting unintended strategies for high reward--without revealing this behavior in their chain-of-thought reasoning, making detection difficult and posing risks for high-stakes applications. We propose verbalization fine-tuning (VFT), a pre-RL intervention that trains models to explicitly acknowledge when they are influenced by prompt cues--hints which point to incorrect answers (e.g., "a Stanford professor thinks the answer is A"). To evaluate VFT, we subsequently train models with RL on environments where held-out prompt cues signal which incorrect answers will receive high reward, incentivizing models to reward hack by exploiting cues instead of reasoning correctly. We measure how often models exploit these cues without verbalizing it. After RL, only 6% of the VFT-trained model's responses consist of undetected reward hacks. In comparison, when we perform RL without VFT, the rate of undetected reward hacks goes up to 88%; with a debiasing baseline intervention, this increases further to 99%. VFT achieves this by substantially increasing how often models verbalize the influence of cues--from 8% to 42% after VFT, and up to 94% after RL--while baselines remain low even after RL (10% and 1%). Our results show that teaching models to explicitly verbalize reward hacking behavior before RL significantly improves their detection, offering a practical path toward more transparent and safe AI systems.


Verbalize Chain of Thought Reasoning

Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning

We study the process through which reasoning models trained with reinforcement learning on verifiable rewards (RLVR) can learn to solve new problems. We find that RLVR drives performance through two main means: (1) by compressing pass@k&nbsp;into pass@1 and (2) via "capability gain" in which models learn to solve new problems that they previously could not solve even at high&nbsp;k. We find that while capability gain exists across model scales, learning to solve new problems is primarily driven through self-distillation. We demonstrate these findings across model scales ranging from 0.5B to 72B on &gt;500,000 reasoning problems with prompts and verifiable final answers across math, science, and code domains. We further show that we can significantly improve pass@k&nbsp;rates by leveraging natural language guidance for the model to consider within context while still requiring the model to derive a solution chain from scratch. Based of these insights, we derive&nbsp;Guide - a new class of online training&nbsp;algorithms.&nbsp;Guide&nbsp;adaptively incorporates hints into the model's context on problems for which all rollouts were initially incorrect and adjusts the importance sampling ratio for the "off-policy" trajectories in order to optimize the policy for contexts in which the hints are no longer present. We describe variants of&nbsp;Guide&nbsp;for GRPO and PPO and empirically show that Guide-GRPO on 7B and 32B parameter models improves generalization over its vanilla counterpart with up to 4%&nbsp;macro-average improvement across math benchmarks. We include careful ablations to analyze&nbsp;Guide's components and theoretically analyze Guide's learning efficiency.

image (37).png

Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models

Reinforcement Learning from Verifiable Rewards (RLVR) has been widely adopted as the de facto method for enhancing the reasoning capabilities of large language models and has demonstrated notable success in verifiable domains like math and competitive programming tasks. However, the efficacy of RLVR diminishes significantly when applied to agentic environments. These settings, characterized by multi-step, complex problem solving, lead to high failure rates even for frontier LLMs, as the reward landscape is too sparse for effective model training via conventional RLVR. In this work, we introduce Agent-RLVR, a framework that makes RLVR effective in challenging agentic settings, with an initial focus on software engineering tasks. Inspired by human pedagogy,
Agent-RLVR introduces agent guidance, a mechanism that actively steers the agent towards successful trajectories by leveraging diverse informational cues. These cues, ranging from high-level strategic plans to dynamic feedback on the agent's errors and environmental interactions, emulate a teacher's guidance, enabling the agent to navigate difficult solution spaces and promotes active self-improvement via additional environment exploration. In the Agent-RLVR training loop, agents first attempt to solve tasks to produce initial trajectories, which are then validated by unit tests and supplemented with agent guidance. Agents then reattempt with guidance, and the agent policy is updated with RLVR based on the rewards of these guided trajectories.
Agent-RLVR elevates the pass@1 performance of Qwen-2.5-72B-Instruct from 9.4% to 22.4% on SWE-Bench Verified. We find that our guidance-augmented RLVR data is additionally useful for test-time reward model training, shown by further boosting pass@1 to 27.8%. Agent-RLVR lays the groundwork for training agents with RLVR in complex, real-world environments where conventional RL methods struggle.

image (36).png

Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards

Agent-RLVR

<div class="page" title="Page 1">
<div class="layoutArea">
<div class="column">
The rapid advancement of large language models (LLMs) introduces dual-use capabilities that could both threaten and bolster national security and public safety (NSPS). Models implement safeguards to protect against potential misuse relevant to NSPS and allow for benign users to receive helpful information. However, current safeguard benchmarks fail to test their robustness to potential NSPS risks in an objective, scalable way. We introduce FORTRESS: 1010 (500 public) expert-crafted adversarial prompts with instance-based rubrics of 4&ndash;7 binary questions for automated evaluation across 3 domains (unclassified information only): Chemical, Biological, Radiological, Nuclear and Explosive (CBRNE), Political Violence &amp; Terrorism, and Criminal &amp; Financial Illicit Activities, with 10 total subcategories across these domains. Each prompt-rubric pair has a corresponding benign version to test for model over-refusal. This evaluation of frontier LLMs&rsquo; safeguard robustness reveals varying trade-offs between potential risks and model usefulness: Claude-3.5-Sonnet demonstrates a low average risk score (14.09 out of 100) but the highest over-refusal score (21.8 out of 100), while Gemini 2.5 Pro shows low over-refusal (1.4) but a high average potential risk (66.29). Deepseek-R1 has the highest ARS at 78.05, but the lowest ORS at only 0.06. Models such as o1 display a more even tradeoff between potential risks and over-refusals (with an ARS of 21.69 and ORS of 5.2). To provide policymakers and researchers with a clear understanding of models&rsquo; potential risks, we publicly release FORTRESS.
</div>
</div>
</div>

image (33).png

FORTRESS: Frontier Risk Evaluation for National Security and Public Safety

<img src="https://img.plasmic.app/img-optimizer/v1/img?src=de82194bd223ec1d6256ff350fcfcf74.png&amp;f=webp&amp;q=75&amp;w=3840" alt="" width="1512" height="1228">
Large Language Model (LLM) safeguards, which implement request refusals, have become a widely adopted mitigation strategy against misuse. At the intersection of adversarial machine learning and AI safety, safeguard red teaming has effectively identified critical vulnerabilities in state-of-the-art refusal-trained LLMs. However, in our view the many conference submissions on LLM red teaming do not, in aggregate, prioritize the right research problems. First, testing against clear product safety specifications should take a higher priority than abstract social biases or ethical principles. Second, red teaming should prioritize realistic threat models that represent the expanding risk landscape and what real attackers might do. Finally, we contend that system-level safety is a necessary step to move red teaming research forward, as AI models present new threats as well as affordances for threat mitigation (e.g., detection and banning of malicious users) once placed in a deployment context. Adopting these priorities will be necessary in order for red teaming research to adequately address the slate of new threats that rapid AI advances present today and will present in the very near future.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{wang2025redteamingroadmapsystemlevel,
 title={A Red Teaming Roadmap Towards System-Level Safety}, 
 author={Zifan Wang and Christina Q. Knight and Jeremy Kritz and Willow E. Primack and Julian Michael},
 year={2025},
 eprint={2506.05376},
 archivePrefix={arXiv},
 primaryClass={cs.CR},
 url={https://arxiv.org/abs/2506.05376}, 
}</code></pre>

image (28).png

Red Teaming Roadmap

A Red Teaming Roadmap Towards System-Level Safety

Supervised and preference-based fine-tuning techniques have become popular for aligning large language models (LLMs) with user intent and correctness criteria. However, real-world training data often exhibits spurious correlations -- arising from biases, dataset artifacts, or other "shortcut" features -- that can compromise a model's performance or generalization. In this paper, we systematically evaluate three post-training algorithms -- Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and KTO (Kahneman-Tversky Optimization) -- across a diverse set of synthetic tasks and spuriousness conditions. Our tasks span mathematical reasoning, constrained instruction-following, and document-grounded question answering. We vary the degree of spurious correlation (10% vs. 90%) and investigate two forms of artifacts: "Feature Ambiguity" and "Distributional Narrowness." Our results show that the models often but not always degrade under higher spuriousness. The preference-based methods (DPO/KTO) can demonstrate relative robustness in mathematical reasoning tasks. By contrast, SFT maintains stronger performance in complex, context-intensive tasks. These findings highlight that no single post-training strategy universally outperforms in all scenarios; the best choice depends on the type of target task and the nature of spurious correlations.

Icon-11.png

Robustness to Spurious Correlations

Assessing Robustness to Spurious Correlations in Post-Training Language Models

Modern Large Language Model (LLM) systems typically rely on Retrieval Augmented Generation (RAG) which aims to gather context that is useful for response generation. These RAG systems typically optimize strictly towards retrieving context that is maximally relevant to the query. However, conventional theory suggests that retrieval systems which seek to maximize context relevance without any additional explicit criteria can create information bottlenecks. We reaffirm this finding in the modern age of LLM's by showing that in standard RAG pipelines, maximizing for context relevance alone can degrade downstream response quality. In response, we show evaluations of existing RAG methods which account for both context relevance and answer quality. These evaluations introduce a novel finding that existing RAG systems scale poorly with inference time compute usage when considering our combined metric. We introduce "RErank BEyond reLevance (REBEL)", which enables RAG systems to scale with inference-time compute via injection of multi-criteria optimization using Chain-of-Thought prompting (and optionally Multi-Turn dialogue). Ultimately, this enables a new performance/speed tradeoff curve, where RAG systems are able to achieve both higher relevance of retrieved contexts and superior answer quality as inference time increases. Code for the implementation of our method in llama-index can be found at the following PR:&nbsp;<a class="link-external link-https" href="https://github.com/run-llama/llama_index/pull/17590" rel="external noopener nofollow">this https URL</a>. Code for running experiments using this llama-index implementation can be found at&nbsp;<a class="link-external link-https" href="https://github.com/microsoft/REBEL" rel="external noopener nofollow">this https URL</a>.
<div class="metatable">&nbsp;</div>

Icon-4.png

Relevance Isn't All You Need: Scaling RAG Systems With Inference-Time Compute Via Multi-Criteria Reranking

Rerank Beyond Relevance

Unit test generation has become a promising and important use case of LLMs. However, existing evaluation benchmarks for assessing LLM unit test generation capabilities focus on function- or class-level code rather than more practical and challenging projectlevel codebases. To address such limitation, we propose ProjectTest, a project-level benchmark for unit test generation covering Python, Java, and JavaScript. ProjectTest features 20 moderate-sized and high-quality projects per language. We evaluate nine frontier LLMs on ProjectTest and the results show that all frontier LLMs tested exhibit moderate performance on ProjectTest on Python and Java, highlighting the difficulty of ProjectTest. We also conduct a thorough error analysis, which shows that even frontier LLMs, such as Claude 3.5 Sonnet, have significant basic yet critical errors, including compilation and cascade errors. Motivated by this observation, we further evaluate all frontier LLMs under manual errorfixing and self-error-fixing scenarios to assess their potential when equipped with error-fixing mechanisms. Our code and dataset is available at&nbsp;<a href="https://github.com/YiboWANG214/ProjectTest">ProjectTest</a>.

1.png

projecttest

ProjectTest: A Project-level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms

As national security institutions increasingly integrate Artificial Intelligence (AI) into decision-making and content generation processes, understanding the inherent biases of large language models (LLMs) is crucial. This study presents a novel benchmark designed to evaluate the biases and preferences of seven prominent foundation models-Llama 3.1 8B Instruct, Llama 3.1 70B Instruct, GPT-4o, Gemini 1.5 Pro-002, Mixtral 8x22B, Claude 3.5 Sonnet, and Qwen2 72B-in the context of international relations (IR). We designed a bias discovery study around core topics in IR using 400-expert crafted scenarios to analyze results from our selected models. These scenarios focused on four topical domains including: military escalation, military and humanitarian intervention, cooperative behavior in the international system, and alliance dynamics. Our analysis reveals noteworthy variation among model recommendations based on scenarios designed for the four tested domains. Particularly, Qwen2 72B, Gemini 1.5 Pro-002 and Llama 3.1 8B Instruct models offered significantly more escalatory recommendations than Claude 3.5 Sonnet and GPT-4o models. All models exhibit some degree of country-specific biases, often recommending less escalatory and interventionist actions for China and Russia compared to the United States and the United Kingdom. These findings highlight the necessity for controlled deployment of LLMs in high-stakes environments, emphasizing the need for domain-specific evaluations and model fine-tuning to align with institutional objectives.

A baseline Analysis - 01.png

Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language Models

As large language models (LLMs) become more capable and agentic, the requirement for trust in their outputs grows significantly, yet at the same time concerns have been mounting that models may learn to lie in pursuit of their goals. To address these concerns, a body of work has emerged around the notion of &ldquo;honesty&rdquo; in LLMs, along with interventions aimed at mitigating deceptive behaviors. However, evaluations of honesty are currently highly limited, with no benchmark combining large scale and applicability to all models. Moreover, many benchmarks claiming to measure honesty in fact simply measure accuracy&mdash;the correctness of a model&rsquo;s beliefs&mdash;in disguise. In this work, we introduce a large-scale human-collected dataset for measuring honesty directly, allowing us to disentangle accuracy from honesty for the first time. Across a diverse set of LLMs, we find that while larger models obtain higher accuracy on our benchmark, they do not become more honest. Surprisingly, while most frontier LLMs obtain high scores on truthfulness bench- marks, we find a substantial propensity in frontier LLMs to lie when pressured to do so, resulting in low honesty&nbsp; cores on our benchmark. We find that simple methods, such as representation engineering interventions, can improve honesty. These results underscore the growing need for robust evaluations and effective interventions to ensure LLMs remain trustworthy.
Check out the Leaderboard <a href="https://scale.com/leaderboard/mask">here</a>.

Icon-1.png

The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems

As language models master existing reasoning benchmarks, we need new challenges to evaluate their cognitive frontiers. Puzzle-solving events are rich repositories of challenging multimodal problems that test a wide range of advanced reasoning and knowledge capabilities, making them a unique testbed for evaluating frontier language models. We introduce ENIGMAEVAL, a dataset of problems and solutions derived from puzzle competitions and events that probes models&rsquo; ability to perform implicit knowledge synthesis and multi-step deductive reasoning. Unlike existing reasoning and knowledge benchmarks, puzzle solving challenges models to discover hidden connections between seemingly unrelated pieces of information to uncover solution paths. The benchmark comprises 1184 puzzles of varying complexity &ndash; each typically requiring teams of skilled solvers hours to days to complete &ndash; with unambiguous, verifiable solutions that enable efficient evaluation. State-of-the-art language models achieve extremely low accuracy on these puzzles, even lower than other difficult benchmarks such as Humanity&rsquo;s Last Exam, unveiling models&rsquo; shortcomings when challenged with problems requiring unstructured and lateral reasoning.
See the Leaderboard results here: <a href="https://scale.com/leaderboard/enigma_eval">https://scale.com/leaderboard/enigma_eval</a>
&nbsp;

Detecting and preventing.png

ENIGMAEVAL

Refusal training on Large Language Models (LLMs) prevents harmful outputs, yet this defense remains vulnerable to both automated and human-crafted jailbreaks. We present a novel LLM-as-red-teamer approach in which a human jailbreaks a refusal-trained LLM to make it willing to jailbreak itself or other LLMs. We refer to the jailbroken LLMs as J2 attackers, which can systematically evaluate target models using various red teaming strategies and improve its performance via in-context learning from the previous failures. Our experiments demonstrate that Sonnet-3.5 and Gemini-1.5-pro outperform other LLMs as J2, achieving 93.0% and 91.0% attack success rates (ASRs) respectively against GPT-4o (and similar results across other capable LLMs) on Harmbench. Our work not only introduces a scalable approach to strategic red teaming&mdash;drawing inspiration from human red teamers, but also highlights jailbreaking-to-jailbreak as an overlooked failure mode of the safeguard. Specifically, an LLM can bypass its own safeguards by employing a jailbroken version of itself that is willing to assist in further jailbreaking. To prevent any direct misuse with J2, while advancing research in AI safety, we publicly share our methodology while keeping specific prompting details private.&nbsp;
Try examples here: <a href="https://scale.com/research/j2/playground">https://scale.com/research/j2/playground</a>
<h1 dir="ltr"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=7edf18642ec432d2196e58b18f03b38e.png&amp;f=webp&amp;q=75" alt="" width="983" height="753"></h1>
<h3 dir="ltr">Highlight</h3>
As the target of a J2 attacker can be its own backbone model, e.g. using J2 (Gemini) to attack Gemini, we reveal a new failure mode in LLM safeguards. Specifically, while fully jailbreaking an LLM for all harmful behaviors remains challenging, an adversary can instead focus resources on bypassing the LLM&rsquo;s safeguards to develop J2 or similar attackers. Once this is achieved, the J2 attacker can then take over and jailbreak the specific harmful behaviors.
Hear from the authors here: <a href="https://exchange.scale.com/public/videos/j2-jailbreaking-to-jailbreak-2025-05-13">https://exchange.scale.com/public/videos/j2-jailbreaking-to-jailbreak-2025-05-13</a>
<h3 dir="ltr">Ethic Statement</h3>
This research &mdash; including the methodology described in the paper, the code, and results &mdash; contains material that can allow users to generate harmful content from some public LLMs. Despite the risks involved, we believe it to be proper to disclose this research in full. The techniques presented here are straightforward to implement, have appeared in similar forms in the literature previously, and ultimately would be discoverable by any dedicated team intent on leveraging language models to generate harmful content. Given the ease of converting LLMs into J2, we prioritize preventing misuse by keeping the jailbreaking prompts private, releasing only select portions of conversations between J2 and target LLMs for academic research.&nbsp;&nbsp;
&nbsp;

J2 - Centered.png

J2: Jailbreaking to Jailbreak

We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capability for their applications. MultiChallenge identifies four categories of challenges in multi-turn conversations that are not only common and realistic among current human-LLM interactions, but are also challenging to all current frontier LLMs. All 4 challenges require accurate instruction-following, context allocation, and in-context reasoning at the same time. We also develop LLM as judge with instance-level rubrics to facilitate an automatic evaluation method with fair agreement with experienced human raters. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet (June 2024) achieving just a 41.4% average accuracy.
Check out the Leaderboard results here: https://scale.com/leaderboard/multichallenge

Improving the Accuracy.png

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce HUMANITY&rsquo;S LAST EXAM (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 3,000 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at <a href="https://lastexam.ai">https://lastexam.ai</a>.
If you use Humanity's Last Exam in your research, please cite our work. The full citation is available <a href="https://github.com/centerforaisafety/hle/blob/main/citation.txt">here</a>.&nbsp;

Icon.png

Humanity’s Last Exam (HLE) tests LLMs with 3,000 expert-crafted questions, revealing the gap between AI and human knowledge.

image (12).png

Abstract: Multimodal language models can exhibit hallucinations in their outputs, which limits their reliability. The ability to automatically detect these errors is important for mitigating them, but has been less explored and existing efforts do not localize hallucinations, instead framing this as a classification task. In this work, we first pose multimodal hallucination detection as a sequence labeling task where models must localize hallucinated text spans and present a strong baseline model. Given the high cost of human annotations for this task, we propose an approach to improve the sample efficiency of these models by creating corrupted grounding data, which we use for pre-training. Leveraging phrase grounding data, we generate hallucinations to replace grounded spans and create hallucinated text. Experiments show that pre-training on this data improves sample efficiency when fine-tuning, and that the learning signal from the grounding data plays an important role in these improvements.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{whitehead2024pretrainingmultimodalhallucinationdetectors,
 title={Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data}, 
 author={Spencer Whitehead and Jacob Phillips and Sean Hendryx},
 year={2024},
 eprint={2409.00238},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2409.00238}, 
}</code></pre>

Pre-Training - 03.png

A sequence labeling approach enhances hallucination detection in multimodal language models, improving efficiency through pre-training on synthetic data.

Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data

Despite recent advances in AI, the development of systems capable of executing complex, multi-step reasoning tasks involving multiple tools remains a significant challenge. Current benchmarks fall short in capturing the real-world complexity of tool-use reasoning, where verifying the correctness of not only the final answer but also the intermediate steps is important for evaluation, development, and identifying failures during inference time. To bridge this gap, we introduce ToolComp, a comprehensive benchmark designed to evaluate multi-step tool-use reasoning. ToolComp is developed through a collaboration between models and human annotators, featuring human-edited/verified prompts, final answers, and process supervision labels, allowing for the evaluation of both final outcomes and intermediate reasoning. Evaluation across six different model families demonstrates the challenging nature of our dataset, with the majority of models achieving less than 50% accuracy. Additionally, we generate synthetic training data to compare the performance of outcome-supervised reward models (ORMs) with process-supervised reward models (PRMs) to assess their ability to improve complex tool-use reasoning as evaluated by ToolComp. Our results show that PRMs generalize significantly better than ORMs, achieving a 19% and 11% improvement in rank@1 accuracy for ranking base and fine-tuned model trajectories, respectively. These findings highlight the critical role of process supervision in both the evaluation and training of AI models, paving the way for more robust and capable systems in complex, multi-step tool-use tasks.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{nath2025toolcompmultitoolreasoning,
 title={ToolComp: A Multi-Tool Reasoning &amp; Process Supervision Benchmark}, 
 author={Vaskar Nath and Pranav Raja and Claire Yoon and Sean Hendryx},
 year={2025},
 eprint={2501.01290},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2501.01290}, 
}</code></pre>

ToolComp - 02.png

ToolComp is a benchmark for evaluating multi-step tool-use reasoning in AI, emphasizing both final outcomes and intermediate steps to enhance model reliability.

ToolComp: A Multi-Tool Reasoning & Process Supervision Benchmark

Abstract: The White House Executive Order on Artificial Intelligence highlights the risks of large language models (LLMs) empowering malicious actors in developing biological, cyber, and chemical weapons. To measure these risks of malicious use, government institutions and major AI labs are developing evaluations for hazardous capabilities in LLMs. However, current evaluations are private, preventing further research into mitigating risk. Furthermore, they focus on only a few, highly specific pathways for malicious use. To fill these gaps, we publicly release the Weapons of Mass Destruction Proxy (WMDP) benchmark, a dataset of 3,668 multiple-choice questions that serve as a proxy measurement of hazardous knowledge in biosecurity, cybersecurity, and chemical security. WMDP was developed by a consortium of academics and technical consultants, and was stringently filtered to eliminate sensitive information prior to public release. WMDP serves two roles: first, as an evaluation for hazardous knowledge in LLMs, and second, as a benchmark for unlearning methods to remove such hazardous knowledge. To guide progress on unlearning, we develop RMU, a state-of-the-art unlearning method based on controlling model representations. RMU reduces model performance on WMDP while maintaining general capabilities in areas such as biology and computer science, suggesting that unlearning may be a concrete path towards reducing malicious use from LLMs. We release our benchmark and code publicly at&nbsp;<a class="link-external link-https" href="https://wmdp.ai/" rel="external noopener nofollow">this https URL</a>.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{li2024wmdpbenchmarkmeasuringreducing,
 title={The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning}, 
 author={Nathaniel Li and Alexander Pan and Anjali Gopal and Summer Yue and Daniel Berrios and Alice Gatti and Justin D. Li and Ann-Kathrin Dombrowski and Shashwat Goel and Long Phan and Gabriel Mukobi and Nathan Helm-Burger and Rassin Lababidi and Lennart Justen and Andrew B. Liu and Michael Chen and Isabelle Barrass and Oliver Zhang and Xiaoyuan Zhu and Rishub Tamirisa and Bhrugu Bharathi and Adam Khoja and Zhenqi Zhao and Ariel Herbert-Voss and Cort B. Breuer and Samuel Marks and Oam Patel and Andy Zou and Mantas Mazeika and Zifan Wang and Palash Oswal and Weiran Lin and Adam A. Hunt and Justin Tienken-Harder and Kevin Y. Shih and Kemper Talley and John Guan and Russell Kaplan and Ian Steneker and David Campbell and Brad Jokubaitis and Alex Levinson and Jean Wang and William Qian and Kallol Krishna Karmakar and Steven Basart and Stephen Fitz and Mindy Levine and Ponnurangam Kumaraguru and Uday Tupakula and Vijay Varadharajan and Ruoyu Wang and Yan Shoshitaishvili and Jimmy Ba and Kevin M. Esvelt and Alexandr Wang and Dan Hendrycks},
 year={2024},
 eprint={2403.03218},
 archivePrefix={arXiv},
 primaryClass={cs.LG},
 url={https://arxiv.org/abs/2403.03218}, 
}</code></pre>

The WMDP - 01.png

The WMDP benchmark evaluates hazardous knowledge in LLMs, supporting research on risk mitigation and unlearning methods to reduce potential misuse.

The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning

Abstract: As large language models (LLMs) become increasingly prevalent in diverse applications, ensuring the utility and safety of model generations becomes paramount. We present a holistic approach for test and evaluation of large language models. The approach encompasses the design of evaluation taxonomies, a novel framework for safety testing, and hybrid methodologies that improve the scalability and cost of evaluations. In particular, we introduce a hybrid methodology for the evaluation of large language models (LLMs) that leverages both human expertise and AI assistance. Our hybrid methodology generalizes across both LLM capabilities and safety, accurately identifying areas where AI assistance can be used to automate this evaluation. Similarly, we find that by combining automated evaluations, generalist red teamers, and expert red teamers, we&rsquo;re able to more efficiently discover new vulnerabilities. We share our approach in hopes that it can contribute to the development of common standards and approaches around test and evaluation of LLMs.

A holistic approach.png

This approach combines human expertise and AI to test and evaluate LLMs, enhancing safety, scalability, and cost efficiency while identifying vulnerabilities.

A Holistic Approach For Test And Evaluation Of Large Language Models

Abstract:&nbsp;Foundation models, specifically Large Language Models (LLMs), have lately gained wide-spread attention and adoption. Reinforcement Learning with Human Feedback (RLHF) involves training a reward model to capture desired behaviors, which is then used to align LLM's. These reward models are additionally used at inference-time to estimate LLM responses' adherence to those desired behaviors. However, there is little work measuring how robust these reward models are to distribution shifts. In this work, we evaluate how reward model performance - measured via accuracy and calibration (i.e. alignment between accuracy and confidence) - is affected by distribution shift. We show novel calibration patterns and accuracy drops due to OOD prompts and responses, and that the reward model is more sensitive to shifts in responses than prompts. Additionally, we adapt an OOD detection technique commonly used in classification to the reward model setting to detect these distribution shifts in prompts and responses.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{levine2024baselineanalysisrewardmodels,
 title={A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift}, 
 author={Will LeVine and Benjamin Pikus and Anthony Chen and Sean Hendryx},
 year={2024},
 eprint={2311.14743},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2311.14743}, 
}</code></pre>

Evaluating the robustness of reward models in RLHF for LLMs, focusing on accuracy, calibration, and the impact of distribution shifts on performance.

A Baseline Analysis of Reward Models’ Ability To Accurately Analyze Foundation Models Under Distribution Shift

Abstract:&nbsp;As deep neural networks become adopted in high-stakes domains, it is crucial to identify when inference inputs are Out-of-Distribution (OOD) so that users can be alerted of likely drops in performance and calibration despite high confidence -- ultimately to know when networks' decisions (and their uncertainty in those decisions) should be trusted. In this paper we introduce Ablated Learned Temperature Energy (or "AbeT" for short), an OOD detection method which lowers the False Positive Rate at 95\% True Positive Rate (FPR@95) by 43.43% in classification compared to state of the art without training networks in multiple stages or requiring hyperparameters or test-time backward passes. We additionally provide empirical insights as to why our model learns to distinguish between In-Distribution (ID) and OOD samples while only being explicitly trained on ID samples via exposure to misclassified ID examples at training time. Lastly, we show the efficacy of our method in identifying predicted bounding boxes and pixels corresponding to OOD objects in object detection and semantic segmentation, respectively -- with an AUROC increase of 5.15% in object detection and both a decrease in FPR@95 of 41.48% and an increase in AUPRC of 34.20% in semantic segmentation compared to previous state of the art.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{levine2024outofdistributiondetectionapplications,
 title={Out-of-Distribution Detection &amp; Applications With Ablated Learned Temperature Energy}, 
 author={Will LeVine and Benjamin Pikus and Jacob Phillips and Berk Norman and Fernando Amat Gil and Sean Hendryx},
 year={2024},
 eprint={2401.12129},
 archivePrefix={arXiv},
 primaryClass={cs.CV},
 url={https://arxiv.org/abs/2401.12129}, 
}</code></pre>

Out of Distribution.png

AbeT improves OOD detection by reducing False Positive Rate at high True Positive rates, enhancing classification, object detection, and segmentation performance.

Out-of-Distribution Detection & Applications With Ablated Learned Temperature Energy

Abstract:&nbsp;Instruction-tuned large language models (LLMs) have demonstrated promising zero-shot generalization capabilities across various downstream tasks. Recent research has introduced multimodal capabilities to LLMs by integrating independently pretrained vision encoders through model grafting. These multimodal variants undergo instruction tuning, similar to LLMs, enabling effective zero-shot generalization for multimodal tasks. This study conducts a comparative analysis of different multimodal instruction tuning approaches and evaluates their performance across a range of tasks, including complex reasoning, conversation, image captioning, multiple-choice questions (MCQs), and binary classification. Through rigorous benchmarking and ablation experiments, we reveal key insights for guiding architectural choices when incorporating multimodal capabilities into LLMs. However, current approaches have limitations; they do not sufficiently address the need for a diverse multimodal instruction dataset, which is crucial for enhancing task generalization. Additionally, they overlook issues related to truthfulness and factuality when generating responses. These findings illuminate current methodological constraints in adapting language models for image comprehension and provide valuable guidance for researchers and practitioners seeking to harness multimodal versions of LLMs.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{garg2023performancemultimodallanguagemodels,
 title={On the Performance of Multimodal Language Models}, 
 author={Utsav Garg and Erhan Bas},
 year={2023},
 eprint={2310.03211},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2310.03211}, 
}</code></pre>

On the Performance.png

Multimodal instruction tuning in LLMs enhances task generalization, revealing insights on architectural choices and addressing challenges in truthfulness and image comprehension.

On the Performance of Multimodal Language Models

Abstract: Instruction tuned Large Vision Language Models (LVLMs) have significantly advanced in generalizing across a diverse set of multi-modal tasks, especially for Visual Question Answering (VQA). However, generating detailed responses that are visually grounded is still a challenging task for these models. We find that even the current state-of-the-art LVLMs (InstructBLIP) still contain a staggering 30 percent of the hallucinatory text in the form of non-existent objects, unfaithful descriptions, and inaccurate relationships. To address this, we introduce M-HalDetect, a (M)ultimodal (Hal)lucination (Detect)ion Dataset that can be used to train and benchmark models for hallucination detection and prevention. M-HalDetect consists of 16k fine-grained annotations on VQA examples, making it the first comprehensive multi-modal hallucination detection dataset for detailed image descriptions. Unlike previous work that only consider object hallucination, we additionally annotate both entity descriptions and relationships that are unfaithful. To demonstrate the potential of this dataset for hallucination prevention, we optimize InstructBLIP through our novel Fine-grained Direct Preference Optimization (FDPO). We also train fine-grained multi-modal reward models from InstructBLIP and evaluate their effectiveness with best-of-n rejection sampling. We perform human evaluation on both FDPO and rejection sampling, and find that they reduce hallucination rates in InstructBLIP by 41% and 55% respectively. We also find that our reward model generalizes to other multi-modal models, reducing hallucinations in LLaVA and mPLUG-OWL by 15% and 57% respectively, and has strong correlation with human evaluated accuracy scores.
Accepted to main track for 38th AAAI Conference on Artificial Intelligence.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{gunjal2024detectingpreventinghallucinationslarge,
 title={Detecting and Preventing Hallucinations in Large Vision Language Models}, 
 author={Anisha Gunjal and Jihan Yin and Erhan Bas},
 year={2024},
 eprint={2308.06394},
 archivePrefix={arXiv},
 primaryClass={cs.CV},
 url={https://arxiv.org/abs/2308.06394}, 
}</code></pre>

M-HalDetect, a dataset for hallucination detection in LVLMs, reduces hallucination rates in state-of-the-art models and improves multi-modal task accuracy through novel techniques.

Scale AI Research

Scale AI’s mission is to accelerate the development of AI applications. By advancing research, we aim to create AI systems capable of solving complex, human-level problems.

Reliable Weak-to-Strong Monitoring of LLM Agents

Search-Time Data Contamination

MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs

Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

WebGuard: Building a Generalizable Guardrail for Web Agents

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning

FORTRESS: Frontier Risk Evaluation for National Security and Public Safety

Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models

LLM Leaderboards

Expert-Led Private Evaluations for precise and reliable LLM rankings

Join the Scale AI Research Team

News

The future of your industry starts here