experiment

HeroTitleAsset

AnnouncementsSlider

Header

Research

Scale AI aims to accelerate AI development, advancing research to create systems capable of solving complex, human-level problems.

Research | Scale AI

CTAs

Footer

PreFooter

SectionTitle

BoxAsset

ResearchPapersList

LoopingVideo

Positions

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs

ENIGMAEVAL: A Benchmark of Long Multimodal Reasoning Challenges

MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems

Advancing Safe and Reliable AI: Scale's Research in Post-Training, Reasoning, and Evaluation

Humanity's Last Exam

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for Fine-Tuning LLMs

Microsoft Azure and Scale AI Collaborate to help Enterprises Deliver Powerful GenAI Solutions

Scale Public Sector: Building on Our Progress in 2025

The New ChatGPT-4o Update Promises Better Writing, How Does It Compare To The New Claude 3.5 Sonnet?

Scale's SEAL Research Lab Launches Expert-Evaluated LLM Leaderboards

Submit Your Toughest Questions for Humanity's Last Exam

First Impressions of OpenAI’s o1

Scale announces collaboration with Microsoft Azure

Fortune has named Scale as one of their 50 AI Innovators

university logo-horizontal_BLK.png

Logotype Black.png

GCP_139x24px 1.svg

Group 58.png

Frame 66.png

Frame 65.png

Frame 64.png

launch code?.png

Frame 62.png

Frame 63.png

USEUCOM.svg

US_Indo-Pacific_Command_Seal.svg

Logos - Google.svg

Anduril.svg

scale logo.png

legal_new.svg

time_logo.svg

leaderboard.png

center.svg

aicpa-soc-logo-freelogovectors.net_ 3.svg

iso-2-1 1.svg

fedramp-logo-vert 2.svg

ntt_logo.svg

cengage_logo.svg.svg

dla-piper 1.svg

cisco.svg

Logos - Valeo.svg

50-AI-logo.svg

bcg.svg

Black.png

Logos - Global Atlantic.svg

Howard_Hughes_logo.png

4DV Analytics - png.png

Logos - Uncountable.svg

Logos - Grata.svg

stability ai.svg

Logos Antrhopic.svg

image.jpg

Logos - Brex.svg

Logos - Nuro.svg

openai.svg

kaleido_ai_black.svg

Luminar Logo White Horizontal Vector 4.svg

Language models trained with RL can engage in reward hacking--exploiting unintended strategies for high reward--without revealing this behavior in their chain-of-thought reasoning, making detection difficult and posing risks for high-stakes applications. We propose verbalization fine-tuning (VFT), a pre-RL intervention that trains models to explicitly acknowledge when they are influenced by prompt cues--hints which point to incorrect answers (e.g., "a Stanford professor thinks the answer is A"). To evaluate VFT, we subsequently train models with RL on environments where held-out prompt cues signal which incorrect answers will receive high reward, incentivizing models to reward hack by exploiting cues instead of reasoning correctly. We measure how often models exploit these cues without verbalizing it. After RL, only 6% of the VFT-trained model's responses consist of undetected reward hacks. In comparison, when we perform RL without VFT, the rate of undetected reward hacks goes up to 88%; with a debiasing baseline intervention, this increases further to 99%. VFT achieves this by substantially increasing how often models verbalize the influence of cues--from 8% to 42% after VFT, and up to 94% after RL--while baselines remain low even after RL (10% and 1%). Our results show that teaching models to explicitly verbalize reward hacking behavior before RL significantly improves their detection, offering a practical path toward more transparent and safe AI systems.
<div class="metatable">&nbsp;</div>

image (38).png

Language models trained with RL can engage in reward hacking--exploiting unintended strategies for high reward--without revealing this behavior in their chain-of-thought reasoning, making detection difficult and posing risks for high-stakes applications. We propose verbalization fine-tuning (VFT), a pre-RL intervention that trains models to explicitly acknowledge when they are influenced by prompt cues--hints which point to incorrect answers (e.g., "a Stanford professor thinks the answer is A"). To evaluate VFT, we subsequently train models with RL on environments where held-out prompt cues signal which incorrect answers will receive high reward, incentivizing models to reward hack by exploiting cues instead of reasoning correctly. We measure how often models exploit these cues without verbalizing it. After RL, only 6% of the VFT-trained model's responses consist of undetected reward hacks. In comparison, when we perform RL without VFT, the rate of undetected reward hacks goes up to 88%; with a debiasing baseline intervention, this increases further to 99%. VFT achieves this by substantially increasing how often models verbalize the influence of cues--from 8% to 42% after VFT, and up to 94% after RL--while baselines remain low even after RL (10% and 1%). Our results show that teaching models to explicitly verbalize reward hacking behavior before RL significantly improves their detection, offering a practical path toward more transparent and safe AI systems.


Verbalize Chain of Thought Reasoning

Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning

We study the process through which reasoning models trained with reinforcement learning on verifiable rewards (RLVR) can learn to solve new problems. We find that RLVR drives performance through two main means: (1) by compressing pass@k&nbsp;into pass@1 and (2) via "capability gain" in which models learn to solve new problems that they previously could not solve even at high&nbsp;k. We find that while capability gain exists across model scales, learning to solve new problems is primarily driven through self-distillation. We demonstrate these findings across model scales ranging from 0.5B to 72B on &gt;500,000 reasoning problems with prompts and verifiable final answers across math, science, and code domains. We further show that we can significantly improve pass@k&nbsp;rates by leveraging natural language guidance for the model to consider within context while still requiring the model to derive a solution chain from scratch. Based of these insights, we derive&nbsp;Guide - a new class of online training&nbsp;algorithms.&nbsp;Guide&nbsp;adaptively incorporates hints into the model's context on problems for which all rollouts were initially incorrect and adjusts the importance sampling ratio for the "off-policy" trajectories in order to optimize the policy for contexts in which the hints are no longer present. We describe variants of&nbsp;Guide&nbsp;for GRPO and PPO and empirically show that Guide-GRPO on 7B and 32B parameter models improves generalization over its vanilla counterpart with up to 4%&nbsp;macro-average improvement across math benchmarks. We include careful ablations to analyze&nbsp;Guide's components and theoretically analyze Guide's learning efficiency.

image (37).png

Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models

Reinforcement Learning from Verifiable Rewards (RLVR) has been widely adopted as the de facto method for enhancing the reasoning capabilities of large language models and has demonstrated notable success in verifiable domains like math and competitive programming tasks. However, the efficacy of RLVR diminishes significantly when applied to agentic environments. These settings, characterized by multi-step, complex problem solving, lead to high failure rates even for frontier LLMs, as the reward landscape is too sparse for effective model training via conventional RLVR. In this work, we introduce Agent-RLVR, a framework that makes RLVR effective in challenging agentic settings, with an initial focus on software engineering tasks. Inspired by human pedagogy, Agent-RLVR introduces agent guidance, a mechanism that actively steers the agent towards successful trajectories by leveraging diverse informational cues. These cues, ranging from high-level strategic plans to dynamic feedback on the agent's errors and environmental interactions, emulate a teacher's guidance, enabling the agent to navigate difficult solution spaces and promotes active self-improvement via additional environment exploration. In the Agent-RLVR training loop, agents first attempt to solve tasks to produce initial trajectories, which are then validated by unit tests and supplemented with agent guidance. Agents then reattempt with guidance, and the agent policy is updated with RLVR based on the rewards of these guided trajectories. Agent-RLVR elevates the pass@1 performance of Qwen-2.5-72B-Instruct from 9.4% to 22.4% on SWE-Bench Verified. We find that our guidance-augmented RLVR data is additionally useful for test-time reward model training, shown by further boosting pass@1 to 27.8%. Agent-RLVR lays the groundwork for training agents with RLVR in complex, real-world environments where conventional RL methods struggle.

image (36).png

Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards

Agent-RLVR

<div class="page" title="Page 1">
<div class="layoutArea">
<div class="column">
The rapid advancement of large language models (LLMs) introduces dual-use capabilities that could both threaten and bolster national security and public safety (NSPS). Models implement safeguards to protect against potential misuse relevant to NSPS and allow for benign users to receive helpful information. However, current safeguard benchmarks fail to test their robustness to potential NSPS risks in an objective, scalable way. We introduce FORTRESS: 1010 (500 public) expert-crafted adversarial prompts with instance-based rubrics of 4&ndash;7 binary questions for automated evaluation across 3 domains (unclassified information only): Chemical, Biological, Radiological, Nuclear and Explosive (CBRNE), Political Violence &amp; Terrorism, and Criminal &amp; Financial Illicit Activities, with 10 total subcategories across these domains. Each prompt-rubric pair has a corresponding benign version to test for model over-refusal. This evaluation of frontier LLMs&rsquo; safeguard robustness reveals varying trade-offs between potential risks and model usefulness: Claude-3.5-Sonnet demonstrates a low average risk score (14.09 out of 100) but the highest over-refusal score (21.8 out of 100), while Gemini 2.5 Pro shows low over-refusal (1.4) but a high average potential risk (66.29). Deepseek-R1 has the highest ARS at 78.05, but the lowest ORS at only 0.06. Models such as o1 display a more even tradeoff between potential risks and over-refusals (with an ARS of 21.69 and ORS of 5.2). To provide policymakers and researchers with a clear understanding of models&rsquo; potential risks, we publicly release FORTRESS.
</div>
</div>
</div>

image (33).png

FORTRESS: Frontier Risk Evaluation for National Security and Public Safety

<img src="https://img.plasmic.app/img-optimizer/v1/img?src=de82194bd223ec1d6256ff350fcfcf74.png&amp;f=webp&amp;q=75&amp;w=3840" alt="" width="1512" height="1228">
Large Language Model (LLM) safeguards, which implement request refusals, have become a widely adopted mitigation strategy against misuse. At the intersection of adversarial machine learning and AI safety, safeguard red teaming has effectively identified critical vulnerabilities in state-of-the-art refusal-trained LLMs. However, in our view the many conference submissions on LLM red teaming do not, in aggregate, prioritize the right research problems. First, testing against clear product safety specifications should take a higher priority than abstract social biases or ethical principles. Second, red teaming should prioritize realistic threat models that represent the expanding risk landscape and what real attackers might do. Finally, we contend that system-level safety is a necessary step to move red teaming research forward, as AI models present new threats as well as affordances for threat mitigation (e.g., detection and banning of malicious users) once placed in a deployment context. Adopting these priorities will be necessary in order for red teaming research to adequately address the slate of new threats that rapid AI advances present today and will present in the very near future.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{wang2025redteamingroadmapsystemlevel,
 title={A Red Teaming Roadmap Towards System-Level Safety}, 
 author={Zifan Wang and Christina Q. Knight and Jeremy Kritz and Willow E. Primack and Julian Michael},
 year={2025},
 eprint={2506.05376},
 archivePrefix={arXiv},
 primaryClass={cs.CR},
 url={https://arxiv.org/abs/2506.05376}, 
}</code></pre>

image (28).png

Red Teaming Roadmap

A Red Teaming Roadmap Towards System-Level Safety

Supervised and preference-based fine-tuning techniques have become popular for aligning large language models (LLMs) with user intent and correctness criteria. However, real-world training data often exhibits spurious correlations -- arising from biases, dataset artifacts, or other "shortcut" features -- that can compromise a model's performance or generalization. In this paper, we systematically evaluate three post-training algorithms -- Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and KTO (Kahneman-Tversky Optimization) -- across a diverse set of synthetic tasks and spuriousness conditions. Our tasks span mathematical reasoning, constrained instruction-following, and document-grounded question answering. We vary the degree of spurious correlation (10% vs. 90%) and investigate two forms of artifacts: "Feature Ambiguity" and "Distributional Narrowness." Our results show that the models often but not always degrade under higher spuriousness. The preference-based methods (DPO/KTO) can demonstrate relative robustness in mathematical reasoning tasks. By contrast, SFT maintains stronger performance in complex, context-intensive tasks. These findings highlight that no single post-training strategy universally outperforms in all scenarios; the best choice depends on the type of target task and the nature of spurious correlations.

Icon-11.png

Robustness to Spurious Correlations

Assessing Robustness to Spurious Correlations in Post-Training Language Models

Modern Large Language Model (LLM) systems typically rely on Retrieval Augmented Generation (RAG) which aims to gather context that is useful for response generation. These RAG systems typically optimize strictly towards retrieving context that is maximally relevant to the query. However, conventional theory suggests that retrieval systems which seek to maximize context relevance without any additional explicit criteria can create information bottlenecks. We reaffirm this finding in the modern age of LLM's by showing that in standard RAG pipelines, maximizing for context relevance alone can degrade downstream response quality. In response, we show evaluations of existing RAG methods which account for both context relevance and answer quality. These evaluations introduce a novel finding that existing RAG systems scale poorly with inference time compute usage when considering our combined metric. We introduce "RErank BEyond reLevance (REBEL)", which enables RAG systems to scale with inference-time compute via injection of multi-criteria optimization using Chain-of-Thought prompting (and optionally Multi-Turn dialogue). Ultimately, this enables a new performance/speed tradeoff curve, where RAG systems are able to achieve both higher relevance of retrieved contexts and superior answer quality as inference time increases. Code for the implementation of our method in llama-index can be found at the following PR:&nbsp;<a class="link-external link-https" href="https://github.com/run-llama/llama_index/pull/17590" rel="external noopener nofollow">this https URL</a>. Code for running experiments using this llama-index implementation can be found at&nbsp;<a class="link-external link-https" href="https://github.com/microsoft/REBEL" rel="external noopener nofollow">this https URL</a>.
<div class="metatable">&nbsp;</div>

Icon-4.png

Relevance Isn't All You Need: Scaling RAG Systems With Inference-Time Compute Via Multi-Criteria Reranking

Rerank Beyond Relevance

Unit test generation has become a promising and important use case of LLMs. However, existing evaluation benchmarks for assessing LLM unit test generation capabilities focus on function- or class-level code rather than more practical and challenging projectlevel codebases. To address such limitation, we propose ProjectTest, a project-level benchmark for unit test generation covering Python, Java, and JavaScript. ProjectTest features 20 moderate-sized and high-quality projects per language. We evaluate nine frontier LLMs on ProjectTest and the results show that all frontier LLMs tested exhibit moderate performance on ProjectTest on Python and Java, highlighting the difficulty of ProjectTest. We also conduct a thorough error analysis, which shows that even frontier LLMs, such as Claude 3.5 Sonnet, have significant basic yet critical errors, including compilation and cascade errors. Motivated by this observation, we further evaluate all frontier LLMs under manual errorfixing and self-error-fixing scenarios to assess their potential when equipped with error-fixing mechanisms. Our code and dataset is available at&nbsp;<a href="https://github.com/YiboWANG214/ProjectTest">ProjectTest</a>.

1.png

projecttest

ProjectTest: A Project-level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms

As national security institutions increasingly integrate Artificial Intelligence (AI) into decision-making and content generation processes, understanding the inherent biases of large language models (LLMs) is crucial. This study presents a novel benchmark designed to evaluate the biases and preferences of seven prominent foundation models-Llama 3.1 8B Instruct, Llama 3.1 70B Instruct, GPT-4o, Gemini 1.5 Pro-002, Mixtral 8x22B, Claude 3.5 Sonnet, and Qwen2 72B-in the context of international relations (IR). We designed a bias discovery study around core topics in IR using 400-expert crafted scenarios to analyze results from our selected models. These scenarios focused on four topical domains including: military escalation, military and humanitarian intervention, cooperative behavior in the international system, and alliance dynamics. Our analysis reveals noteworthy variation among model recommendations based on scenarios designed for the four tested domains. Particularly, Qwen2 72B, Gemini 1.5 Pro-002 and Llama 3.1 8B Instruct models offered significantly more escalatory recommendations than Claude 3.5 Sonnet and GPT-4o models. All models exhibit some degree of country-specific biases, often recommending less escalatory and interventionist actions for China and Russia compared to the United States and the United Kingdom. These findings highlight the necessity for controlled deployment of LLMs in high-stakes environments, emphasizing the need for domain-specific evaluations and model fine-tuning to align with institutional objectives.

A baseline Analysis - 01.png

Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language Models

As large language models (LLMs) become more capable and agentic, the requirement for trust in their outputs grows significantly, yet at the same time concerns have been mounting that models may learn to lie in pursuit of their goals. To address these concerns, a body of work has emerged around the notion of &ldquo;honesty&rdquo; in LLMs, along with interventions aimed at mitigating deceptive behaviors. However, evaluations of honesty are currently highly limited, with no benchmark combining large scale and applicability to all models. Moreover, many benchmarks claiming to measure honesty in fact simply measure accuracy&mdash;the correctness of a model&rsquo;s beliefs&mdash;in disguise. In this work, we introduce a large-scale human-collected dataset for measuring honesty directly, allowing us to disentangle accuracy from honesty for the first time. Across a diverse set of LLMs, we find that while larger models obtain higher accuracy on our benchmark, they do not become more honest. Surprisingly, while most frontier LLMs obtain high scores on truthfulness bench- marks, we find a substantial propensity in frontier LLMs to lie when pressured to do so, resulting in low honesty&nbsp; cores on our benchmark. We find that simple methods, such as representation engineering interventions, can improve honesty. These results underscore the growing need for robust evaluations and effective interventions to ensure LLMs remain trustworthy.
Check out the Leaderboard <a href="https://scale.com/leaderboard/mask">here</a>.

Icon-1.png

The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems

As language models master existing reasoning benchmarks, we need new challenges to evaluate their cognitive frontiers. Puzzle-solving events are rich repositories of challenging multimodal problems that test a wide range of advanced reasoning and knowledge capabilities, making them a unique testbed for evaluating frontier language models. We introduce ENIGMAEVAL, a dataset of problems and solutions derived from puzzle competitions and events that probes models&rsquo; ability to perform implicit knowledge synthesis and multi-step deductive reasoning. Unlike existing reasoning and knowledge benchmarks, puzzle solving challenges models to discover hidden connections between seemingly unrelated pieces of information to uncover solution paths. The benchmark comprises 1184 puzzles of varying complexity &ndash; each typically requiring teams of skilled solvers hours to days to complete &ndash; with unambiguous, verifiable solutions that enable efficient evaluation. State-of-the-art language models achieve extremely low accuracy on these puzzles, even lower than other difficult benchmarks such as Humanity&rsquo;s Last Exam, unveiling models&rsquo; shortcomings when challenged with problems requiring unstructured and lateral reasoning.
See the Leaderboard results here: <a href="https://scale.com/leaderboard/enigma_eval">https://scale.com/leaderboard/enigma_eval</a>
&nbsp;

Detecting and preventing.png

ENIGMAEVAL

Refusal training on Large Language Models (LLMs) prevents harmful outputs, yet this defense remains vulnerable to both automated and human-crafted jailbreaks. We present a novel LLM-as-red-teamer approach in which a human jailbreaks a refusal-trained LLM to make it willing to jailbreak itself or other LLMs. We refer to the jailbroken LLMs as J2 attackers, which can systematically evaluate target models using various red teaming strategies and improve its performance via in-context learning from the previous failures. Our experiments demonstrate that Sonnet-3.5 and Gemini-1.5-pro outperform other LLMs as J2, achieving 93.0% and 91.0% attack success rates (ASRs) respectively against GPT-4o (and similar results across other capable LLMs) on Harmbench. Our work not only introduces a scalable approach to strategic red teaming&mdash;drawing inspiration from human red teamers, but also highlights jailbreaking-to-jailbreak as an overlooked failure mode of the safeguard. Specifically, an LLM can bypass its own safeguards by employing a jailbroken version of itself that is willing to assist in further jailbreaking. To prevent any direct misuse with J2, while advancing research in AI safety, we publicly share our methodology while keeping specific prompting details private.&nbsp;
Try examples here: <a href="https://scale.com/research/j2/playground">https://scale.com/research/j2/playground</a>
<h1 dir="ltr"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=7edf18642ec432d2196e58b18f03b38e.png&amp;f=webp&amp;q=75" alt="" width="983" height="753"></h1>
<h3 dir="ltr">Highlight</h3>
As the target of a J2 attacker can be its own backbone model, e.g. using J2 (Gemini) to attack Gemini, we reveal a new failure mode in LLM safeguards. Specifically, while fully jailbreaking an LLM for all harmful behaviors remains challenging, an adversary can instead focus resources on bypassing the LLM&rsquo;s safeguards to develop J2 or similar attackers. Once this is achieved, the J2 attacker can then take over and jailbreak the specific harmful behaviors.
Hear from the authors here: <a href="https://exchange.scale.com/public/videos/j2-jailbreaking-to-jailbreak-2025-05-13">https://exchange.scale.com/public/videos/j2-jailbreaking-to-jailbreak-2025-05-13</a>
<h3 dir="ltr">Ethic Statement</h3>
This research &mdash; including the methodology described in the paper, the code, and results &mdash; contains material that can allow users to generate harmful content from some public LLMs. Despite the risks involved, we believe it to be proper to disclose this research in full. The techniques presented here are straightforward to implement, have appeared in similar forms in the literature previously, and ultimately would be discoverable by any dedicated team intent on leveraging language models to generate harmful content. Given the ease of converting LLMs into J2, we prioritize preventing misuse by keeping the jailbreaking prompts private, releasing only select portions of conversations between J2 and target LLMs for academic research.&nbsp;&nbsp;
&nbsp;

J2 - Centered.png

J2: Jailbreaking to Jailbreak

We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capability for their applications. MultiChallenge identifies four categories of challenges in multi-turn conversations that are not only common and realistic among current human-LLM interactions, but are also challenging to all current frontier LLMs. All 4 challenges require accurate instruction-following, context allocation, and in-context reasoning at the same time. We also develop LLM as judge with instance-level rubrics to facilitate an automatic evaluation method with fair agreement with experienced human raters. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet (June 2024) achieving just a 41.4% average accuracy.
Check out the Leaderboard results here: https://scale.com/leaderboard/multichallenge

Improving the Accuracy.png

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce HUMANITY&rsquo;S LAST EXAM (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 3,000 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at <a href="https://lastexam.ai">https://lastexam.ai</a>.
If you use Humanity's Last Exam in your research, please cite our work. The full citation is available <a href="https://github.com/centerforaisafety/hle/blob/main/citation.txt">here</a>.&nbsp;

Icon.png

Humanity’s Last Exam (HLE) tests LLMs with 3,000 expert-crafted questions, revealing the gap between AI and human knowledge.

image (12).png

Abstract: Multimodal language models can exhibit hallucinations in their outputs, which limits their reliability. The ability to automatically detect these errors is important for mitigating them, but has been less explored and existing efforts do not localize hallucinations, instead framing this as a classification task. In this work, we first pose multimodal hallucination detection as a sequence labeling task where models must localize hallucinated text spans and present a strong baseline model. Given the high cost of human annotations for this task, we propose an approach to improve the sample efficiency of these models by creating corrupted grounding data, which we use for pre-training. Leveraging phrase grounding data, we generate hallucinations to replace grounded spans and create hallucinated text. Experiments show that pre-training on this data improves sample efficiency when fine-tuning, and that the learning signal from the grounding data plays an important role in these improvements.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{whitehead2024pretrainingmultimodalhallucinationdetectors,
 title={Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data}, 
 author={Spencer Whitehead and Jacob Phillips and Sean Hendryx},
 year={2024},
 eprint={2409.00238},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2409.00238}, 
}</code></pre>

Pre-Training - 03.png

A sequence labeling approach enhances hallucination detection in multimodal language models, improving efficiency through pre-training on synthetic data.

Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data

Despite recent advances in AI, the development of systems capable of executing complex, multi-step reasoning tasks involving multiple tools remains a significant challenge. Current benchmarks fall short in capturing the real-world complexity of tool-use reasoning, where verifying the correctness of not only the final answer but also the intermediate steps is important for evaluation, development, and identifying failures during inference time. To bridge this gap, we introduce ToolComp, a comprehensive benchmark designed to evaluate multi-step tool-use reasoning. ToolComp is developed through a collaboration between models and human annotators, featuring human-edited/verified prompts, final answers, and process supervision labels, allowing for the evaluation of both final outcomes and intermediate reasoning. Evaluation across six different model families demonstrates the challenging nature of our dataset, with the majority of models achieving less than 50% accuracy. Additionally, we generate synthetic training data to compare the performance of outcome-supervised reward models (ORMs) with process-supervised reward models (PRMs) to assess their ability to improve complex tool-use reasoning as evaluated by ToolComp. Our results show that PRMs generalize significantly better than ORMs, achieving a 19% and 11% improvement in rank@1 accuracy for ranking base and fine-tuned model trajectories, respectively. These findings highlight the critical role of process supervision in both the evaluation and training of AI models, paving the way for more robust and capable systems in complex, multi-step tool-use tasks.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{nath2025toolcompmultitoolreasoning,
 title={ToolComp: A Multi-Tool Reasoning &amp; Process Supervision Benchmark}, 
 author={Vaskar Nath and Pranav Raja and Claire Yoon and Sean Hendryx},
 year={2025},
 eprint={2501.01290},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2501.01290}, 
}</code></pre>

ToolComp - 02.png

ToolComp is a benchmark for evaluating multi-step tool-use reasoning in AI, emphasizing both final outcomes and intermediate steps to enhance model reliability.

ToolComp: A Multi-Tool Reasoning & Process Supervision Benchmark

Abstract: The White House Executive Order on Artificial Intelligence highlights the risks of large language models (LLMs) empowering malicious actors in developing biological, cyber, and chemical weapons. To measure these risks of malicious use, government institutions and major AI labs are developing evaluations for hazardous capabilities in LLMs. However, current evaluations are private, preventing further research into mitigating risk. Furthermore, they focus on only a few, highly specific pathways for malicious use. To fill these gaps, we publicly release the Weapons of Mass Destruction Proxy (WMDP) benchmark, a dataset of 3,668 multiple-choice questions that serve as a proxy measurement of hazardous knowledge in biosecurity, cybersecurity, and chemical security. WMDP was developed by a consortium of academics and technical consultants, and was stringently filtered to eliminate sensitive information prior to public release. WMDP serves two roles: first, as an evaluation for hazardous knowledge in LLMs, and second, as a benchmark for unlearning methods to remove such hazardous knowledge. To guide progress on unlearning, we develop RMU, a state-of-the-art unlearning method based on controlling model representations. RMU reduces model performance on WMDP while maintaining general capabilities in areas such as biology and computer science, suggesting that unlearning may be a concrete path towards reducing malicious use from LLMs. We release our benchmark and code publicly at&nbsp;<a class="link-external link-https" href="https://wmdp.ai/" rel="external noopener nofollow">this https URL</a>.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{li2024wmdpbenchmarkmeasuringreducing,
 title={The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning}, 
 author={Nathaniel Li and Alexander Pan and Anjali Gopal and Summer Yue and Daniel Berrios and Alice Gatti and Justin D. Li and Ann-Kathrin Dombrowski and Shashwat Goel and Long Phan and Gabriel Mukobi and Nathan Helm-Burger and Rassin Lababidi and Lennart Justen and Andrew B. Liu and Michael Chen and Isabelle Barrass and Oliver Zhang and Xiaoyuan Zhu and Rishub Tamirisa and Bhrugu Bharathi and Adam Khoja and Zhenqi Zhao and Ariel Herbert-Voss and Cort B. Breuer and Samuel Marks and Oam Patel and Andy Zou and Mantas Mazeika and Zifan Wang and Palash Oswal and Weiran Lin and Adam A. Hunt and Justin Tienken-Harder and Kevin Y. Shih and Kemper Talley and John Guan and Russell Kaplan and Ian Steneker and David Campbell and Brad Jokubaitis and Alex Levinson and Jean Wang and William Qian and Kallol Krishna Karmakar and Steven Basart and Stephen Fitz and Mindy Levine and Ponnurangam Kumaraguru and Uday Tupakula and Vijay Varadharajan and Ruoyu Wang and Yan Shoshitaishvili and Jimmy Ba and Kevin M. Esvelt and Alexandr Wang and Dan Hendrycks},
 year={2024},
 eprint={2403.03218},
 archivePrefix={arXiv},
 primaryClass={cs.LG},
 url={https://arxiv.org/abs/2403.03218}, 
}</code></pre>

The WMDP - 01.png

The WMDP benchmark evaluates hazardous knowledge in LLMs, supporting research on risk mitigation and unlearning methods to reduce potential misuse.

The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning

Abstract: As large language models (LLMs) become increasingly prevalent in diverse applications, ensuring the utility and safety of model generations becomes paramount. We present a holistic approach for test and evaluation of large language models. The approach encompasses the design of evaluation taxonomies, a novel framework for safety testing, and hybrid methodologies that improve the scalability and cost of evaluations. In particular, we introduce a hybrid methodology for the evaluation of large language models (LLMs) that leverages both human expertise and AI assistance. Our hybrid methodology generalizes across both LLM capabilities and safety, accurately identifying areas where AI assistance can be used to automate this evaluation. Similarly, we find that by combining automated evaluations, generalist red teamers, and expert red teamers, we&rsquo;re able to more efficiently discover new vulnerabilities. We share our approach in hopes that it can contribute to the development of common standards and approaches around test and evaluation of LLMs.

A holistic approach.png

This approach combines human expertise and AI to test and evaluate LLMs, enhancing safety, scalability, and cost efficiency while identifying vulnerabilities.

A Holistic Approach For Test And Evaluation Of Large Language Models

Abstract:&nbsp;Foundation models, specifically Large Language Models (LLMs), have lately gained wide-spread attention and adoption. Reinforcement Learning with Human Feedback (RLHF) involves training a reward model to capture desired behaviors, which is then used to align LLM's. These reward models are additionally used at inference-time to estimate LLM responses' adherence to those desired behaviors. However, there is little work measuring how robust these reward models are to distribution shifts. In this work, we evaluate how reward model performance - measured via accuracy and calibration (i.e. alignment between accuracy and confidence) - is affected by distribution shift. We show novel calibration patterns and accuracy drops due to OOD prompts and responses, and that the reward model is more sensitive to shifts in responses than prompts. Additionally, we adapt an OOD detection technique commonly used in classification to the reward model setting to detect these distribution shifts in prompts and responses.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{levine2024baselineanalysisrewardmodels,
 title={A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift}, 
 author={Will LeVine and Benjamin Pikus and Anthony Chen and Sean Hendryx},
 year={2024},
 eprint={2311.14743},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2311.14743}, 
}</code></pre>

Evaluating the robustness of reward models in RLHF for LLMs, focusing on accuracy, calibration, and the impact of distribution shifts on performance.

A Baseline Analysis of Reward Models’ Ability To Accurately Analyze Foundation Models Under Distribution Shift

Abstract:&nbsp;As deep neural networks become adopted in high-stakes domains, it is crucial to identify when inference inputs are Out-of-Distribution (OOD) so that users can be alerted of likely drops in performance and calibration despite high confidence -- ultimately to know when networks' decisions (and their uncertainty in those decisions) should be trusted. In this paper we introduce Ablated Learned Temperature Energy (or "AbeT" for short), an OOD detection method which lowers the False Positive Rate at 95\% True Positive Rate (FPR@95) by 43.43% in classification compared to state of the art without training networks in multiple stages or requiring hyperparameters or test-time backward passes. We additionally provide empirical insights as to why our model learns to distinguish between In-Distribution (ID) and OOD samples while only being explicitly trained on ID samples via exposure to misclassified ID examples at training time. Lastly, we show the efficacy of our method in identifying predicted bounding boxes and pixels corresponding to OOD objects in object detection and semantic segmentation, respectively -- with an AUROC increase of 5.15% in object detection and both a decrease in FPR@95 of 41.48% and an increase in AUPRC of 34.20% in semantic segmentation compared to previous state of the art.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{levine2024outofdistributiondetectionapplications,
 title={Out-of-Distribution Detection &amp; Applications With Ablated Learned Temperature Energy}, 
 author={Will LeVine and Benjamin Pikus and Jacob Phillips and Berk Norman and Fernando Amat Gil and Sean Hendryx},
 year={2024},
 eprint={2401.12129},
 archivePrefix={arXiv},
 primaryClass={cs.CV},
 url={https://arxiv.org/abs/2401.12129}, 
}</code></pre>

Out of Distribution.png

AbeT improves OOD detection by reducing False Positive Rate at high True Positive rates, enhancing classification, object detection, and segmentation performance.

Out-of-Distribution Detection & Applications With Ablated Learned Temperature Energy

Abstract:&nbsp;Instruction-tuned large language models (LLMs) have demonstrated promising zero-shot generalization capabilities across various downstream tasks. Recent research has introduced multimodal capabilities to LLMs by integrating independently pretrained vision encoders through model grafting. These multimodal variants undergo instruction tuning, similar to LLMs, enabling effective zero-shot generalization for multimodal tasks. This study conducts a comparative analysis of different multimodal instruction tuning approaches and evaluates their performance across a range of tasks, including complex reasoning, conversation, image captioning, multiple-choice questions (MCQs), and binary classification. Through rigorous benchmarking and ablation experiments, we reveal key insights for guiding architectural choices when incorporating multimodal capabilities into LLMs. However, current approaches have limitations; they do not sufficiently address the need for a diverse multimodal instruction dataset, which is crucial for enhancing task generalization. Additionally, they overlook issues related to truthfulness and factuality when generating responses. These findings illuminate current methodological constraints in adapting language models for image comprehension and provide valuable guidance for researchers and practitioners seeking to harness multimodal versions of LLMs.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{garg2023performancemultimodallanguagemodels,
 title={On the Performance of Multimodal Language Models}, 
 author={Utsav Garg and Erhan Bas},
 year={2023},
 eprint={2310.03211},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2310.03211}, 
}</code></pre>

On the Performance.png

Multimodal instruction tuning in LLMs enhances task generalization, revealing insights on architectural choices and addressing challenges in truthfulness and image comprehension.

On the Performance of Multimodal Language Models

Abstract: Instruction tuned Large Vision Language Models (LVLMs) have significantly advanced in generalizing across a diverse set of multi-modal tasks, especially for Visual Question Answering (VQA). However, generating detailed responses that are visually grounded is still a challenging task for these models. We find that even the current state-of-the-art LVLMs (InstructBLIP) still contain a staggering 30 percent of the hallucinatory text in the form of non-existent objects, unfaithful descriptions, and inaccurate relationships. To address this, we introduce M-HalDetect, a (M)ultimodal (Hal)lucination (Detect)ion Dataset that can be used to train and benchmark models for hallucination detection and prevention. M-HalDetect consists of 16k fine-grained annotations on VQA examples, making it the first comprehensive multi-modal hallucination detection dataset for detailed image descriptions. Unlike previous work that only consider object hallucination, we additionally annotate both entity descriptions and relationships that are unfaithful. To demonstrate the potential of this dataset for hallucination prevention, we optimize InstructBLIP through our novel Fine-grained Direct Preference Optimization (FDPO). We also train fine-grained multi-modal reward models from InstructBLIP and evaluate their effectiveness with best-of-n rejection sampling. We perform human evaluation on both FDPO and rejection sampling, and find that they reduce hallucination rates in InstructBLIP by 41% and 55% respectively. We also find that our reward model generalizes to other multi-modal models, reducing hallucinations in LLaVA and mPLUG-OWL by 15% and 57% respectively, and has strong correlation with human evaluated accuracy scores.
Accepted to main track for 38th AAAI Conference on Artificial Intelligence.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{gunjal2024detectingpreventinghallucinationslarge,
 title={Detecting and Preventing Hallucinations in Large Vision Language Models}, 
 author={Anisha Gunjal and Jihan Yin and Erhan Bas},
 year={2024},
 eprint={2308.06394},
 archivePrefix={arXiv},
 primaryClass={cs.CV},
 url={https://arxiv.org/abs/2308.06394}, 
}</code></pre>

M-HalDetect, a dataset for hallucination detection in LVLMs, reduces hallucination rates in state-of-the-art models and improves multi-modal task accuracy through novel techniques.

Detecting and Preventing Hallucinations in Large Vision Language Models

Abstract:&nbsp;Calibration of deep learning models is crucial to their trustworthiness and safe&nbsp;usage, and as such, has been extensively studied in supervised classification models, with methods rafted to decrease miscalibration. However, there has yet to be a comprehensive study of the calibration of vision-language models that are&nbsp;used for zero-shot inference, like CLIP. We measure calibration across relevant&nbsp;variables like prompt, dataset, and architecture, and find that zero-shot inference&nbsp;with CLIP is miscalibrated. Furthermore, we propose a modified version of temperature scaling that is&nbsp; ligned with the common use cases of CLIP as a zero-shot inference model, and show that a single learned temperature generalizes for each&nbsp;specific CLIP model (defined by a chosen pre-training dataset and architecture)&nbsp;across inference dataset and prompt choice.
Accepted to workshop on the pitfalls of limited data and computation for Trustworthy ML, ICLR 2023.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{levine2023enablingcalibrationzeroshotinference,
 title={Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models}, 
 author={Will LeVine and Benjamin Pikus and Pranav Raja and Fernando Amat Gil},
 year={2023},
 eprint={2303.12748},
 archivePrefix={arXiv},
 primaryClass={cs.CV},
 url={https://arxiv.org/abs/2303.12748}, 
}</code></pre>
&nbsp;

Enabling Calibration.png

This study evaluates and improves the calibration of vision-language models like CLIP for zero-shot inference, proposing a modified temperature scaling method for better generalization.

Enabling Calibration In The Zero-shot Inference Of Large Vision-Language Models

Abstract:&nbsp;As foundation models continue to exponentially scale in size, efficient methods of adaptation become increasingly critical. Parameter-efficient fine-tuning (PEFT), a recent class of techniques that require only modifying a small percentage of the model parameters, is currently the most popular method for adapting large language models (LLMs). Several PEFT techniques have recently been proposed with varying tradeoffs. We provide a comprehensive and uniform benchmark of various PEFT techniques across a representative LLM, the FLAN-T5 model, and evaluate model performance across different data scales of classification and generation datasets. Based on this, we provide a framework for choosing the optimal fine-tuning techniques given the task type and data availability. Contrary to popular belief, we also empirically prove that PEFT techniques converge slower than full tuning in low data scenarios, and posit the amount of data required for PEFT methods to both perform well and converge efficiently. Lastly, we further optimize these PEFT techniques by selectively choosing which parts of the model to train, and find that these techniques can be applied with significantly fewer parameters while maintaining and even improving performance.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{pu2023empiricalanalysisstrengthsweaknesses,
 title={Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs}, 
 author={George Pu and Anirudh Jain and Jihan Yin and Russell Kaplan},
 year={2023},
 eprint={2304.14999},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2304.14999}, 
}</code></pre>

Empirical Analysis - 01.png

Benchmarking parameter-efficient fine-tuning (PEFT) techniques for large language models, evaluating their performance across different datasets, and offering a framework for optimal adaptation.

Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs

Abstract:&nbsp;While prior research has proposed a plethora of methods that build neural classifiers robust against adversarial robustness, practitioners are still reluctant to adopt them due to their unacceptably severe clean accuracy penalties. This paper significantly alleviates this accuracy-robustness trade-off by mixing the output probabilities of a standard classifier and a robust classifier, where the standard network is optimized for clean accuracy and is not robust in general. We show that the robust base classifier's confidence difference for correct and incorrect examples is the key to this improvement. In addition to providing intuitions and empirical evidence, we theoretically certify the robustness of the mixed classifier under realistic assumptions. Furthermore, we adapt an adversarial input detector into a mixing network that adaptively adjusts the mixture of the two base models, further reducing the accuracy penalty of achieving robustness. The proposed flexible method, termed "adaptive smoothing", can work in conjunction with existing or even future methods that improve clean accuracy, robustness, or adversary detection. Our empirical evaluation considers strong attack methods, including AutoAttack and adaptive attack. On the CIFAR-100 dataset, our method achieves an 85.21% clean accuracy while maintaining a 38.72%&nbsp;ℓ&infin;-AutoAttacked (ϵ=8/255) accuracy, becoming the second most robust method on the RobustBench CIFAR-100 benchmark as of submission, while improving the clean accuracy by ten percentage points compared with all listed models. The code that implements our method is available at&nbsp;<a class="link-external link-https" href="https://github.com/Bai-YT/AdaptiveSmoothing" rel="external noopener nofollow">this https URL</a>.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{bai2024improvingaccuracyrobustnesstradeoffclassifiers,
 title={Improving the Accuracy-Robustness Trade-Off of Classifiers via Adaptive Smoothing}, 
 author={Yatong Bai and Brendon G. Anderson and Aerin Kim and Somayeh Sojoudi},
 year={2024},
 eprint={2301.12554},
 archivePrefix={arXiv},
 primaryClass={cs.LG},
 url={https://arxiv.org/abs/2301.12554}, 
}</code></pre>

Adaptive smoothing improves adversarial robustness in classifiers, achieving 85.21% clean accuracy and 38.72% ℓ∞-AutoAttacked accuracy on CIFAR-100.

Improving the Accuracy-Robustness Trade-Off of Classifiers via Adaptive Smoothing

In this paper, we introduce DEBAGREEMENT, a dataset of 42,894 commentreply pairs from the popular discussion website reddit, annotated with agree, neutral or disagree labels. We collect data from five forums on reddit: r/BlackLivesMatter, r/Brexit, r/climate, r/democrats, r/Republican. For each forum, we select comment pairs such that they form altogether a user interaction graph. DEBAGREEMENT presents a challenge for Natural Language Processing (NLP) systems, as it contains slang, sarcasm and topic-specific jokes, often present in online exchanges. We evaluate the performance of state-of-the-art language models on a (dis)agreement detection task, and investigate the use of contextual information available (graph, authorship, and temporal information). Since recent research has shown that context, such as social context or knowledge graph information, enables language models to better perform on downstream NLP tasks, DEBAGREEMENT provides novel opportunities for combining graph-based and text-based machine learning techniques to detect (dis)agreements online.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@inproceedings{
pougu{\'e}-biyong2021debagreement,
title={{DEBAGREEMENT}: A comment-reply dataset for (dis)agreement detection in online debates},
author={John Pougu{\'e}-Biyong and Valentina Semenova and Alexandre Matton and Rachel Han and Aerin Kim and Renaud Lambiotte and Doyne Farmer},
booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)},
year={2021},
url={https://openreview.net/forum?id=udVUN__gFO}
}</code></pre>

Debagreement - 01.png

DEBAGREEMENT is a dataset of 42,894 comment-reply pairs from Reddit, annotated for agreement detection, incorporating graph-based and text-based techniques.

DEBAGREEMENT: A comment-reply dataset for (dis)agreement detection in online debates

Abstract:&nbsp;Although state-of-the-art object detection methods have shown compelling performance, models often are not robust to adversarial attacks and out-of-distribution data. We introduce a new dataset, Natural Adversarial Objects (NAO), to evaluate the robustness of object detection models. NAO contains 7,934 images and 9,943 objects that are unmodified and representative of real-world scenarios, but cause state-of-the-art detection models to misclassify with high confidence. The mean average precision (mAP) of EfficientDet-D7 drops 74.5% when evaluated on NAO compared to the standard MSCOCO validation set. Moreover, by comparing a variety of object detection architectures, we find that better performance on MSCOCO validation set does not necessarily translate to better performance on NAO, suggesting that robustness cannot be simply achieved by training a more accurate model. We further investigate why examples in NAO are difficult to detect and classify. Experiments of shuffling image patches reveal that models are overly sensitive to local texture. Additionally, using integrated gradients and background replacement, we find that the detection model is reliant on pixel information within the bounding box, and insensitive to the background context when predicting class labels. NAO can be downloaded at&nbsp;<a class="link-external link-https" href="https://drive.google.com/drive/folders/15P8sOWoJku6SSEiHLEts86ORfytGezi8" rel="external noopener nofollow">this https URL</a>.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{lau2021naturaladversarialobjects,
 title={Natural Adversarial Objects}, 
 author={Felix Lau and Nishant Subramani and Sasha Harrison and Aerin Kim and Elliot Branson and Rosanne Liu},
 year={2021},
 eprint={2111.04204},
 archivePrefix={arXiv},
 primaryClass={cs.CV},
 url={https://arxiv.org/abs/2111.04204}, 
}</code></pre>

Natural Adversarial Objects.png

NAO is a dataset of 7,934 images and 9,943 objects designed to evaluate object detection models' robustness, causing significant misclassification.

Natural Adversarial Objects

Abstract: How does the accuracy of deep neural network models trained to classify clinical images of skin conditions vary across skin color? While recent studies demonstrate computer vision models can serve as a useful decision support tool in healthcare and provide dermatologist-level classification on a number of specific tasks, darker skin is underrepresented in the data. Most publicly available data sets do not include Fitzpatrick skin type labels. We annotate 16,577 clinical images sourced from two dermatology atlases with Fitzpatrick skin type labels and open-source these annotations. Based on these labels, we find that there are significantly more images of light skin types than dark skin types in this dataset. We train a deep neural network model to classify 114 skin conditions and find that the model is most accurate on skin types similar to those it was trained on. In addition, we evaluate how an algorithmic approach to identifying skin tones, individual typology angle, compares with Fitzpatrick skin type labels annotated by a team of human labelers.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{groh2021evaluatingdeepneuralnetworks,
 title={Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset}, 
 author={Matthew Groh and Caleb Harris and Luis Soenksen and Felix Lau and Rachel Han and Aerin Kim and Arash Koochek and Omar Badri},
 year={2021},
 eprint={2104.09957},
 archivePrefix={arXiv},
 primaryClass={cs.CV},
 url={https://arxiv.org/abs/2104.09957}, 
}</code></pre>

Evaluating deep neural networks.png

Study analyzes deep neural networks' accuracy on skin condition images, revealing biases with underrepresentation of dark skin types in clinical datasets.

Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset

Abstract:&nbsp;Attaching attributes (such as color, shape, state, action) to object categories is an important computer vision problem. Attribute prediction has seen exciting recent progress and is often formulated as a multi-label classification problem. Yet significant challenges remain in: 1) predicting diverse attributes over multiple categories, 2) modeling attributes-category dependency, 3) capturing both global and local scene context, and 4) predicting attributes of objects with low pixel-count. To address these issues, we propose a novel multi-category attribute prediction deep architecture named GlideNet, which contains three distinct feature extractors. A global feature extractor recognizes what objects are present in a scene, whereas a local one focuses on the area surrounding the object of interest. Meanwhile, an intrinsic feature extractor uses an extension of standard convolution dubbed Informed Convolution to retrieve features of objects with low pixel-count. GlideNet uses gating mechanisms with binary masks and its self-learned category embedding to combine the dense embeddings. Collectively, the Global-Local-Intrinsic blocks comprehend the scene's global context while attending to the characteristics of the local object of interest. Finally, using the combined features, an interpreter predicts the attributes, and the length of the output is determined by the category, thereby removing unnecessary attributes. GlideNet can achieve compelling results on two recent and challenging datasets -- VAW and CAR -- for large-scale attribute prediction. For instance, it obtains more than 5\% gain over state of the art in the mean recall (mR) metric. GlideNet's advantages are especially apparent when predicting attributes of objects with low pixel counts as well as attributes that demand global context understanding. Finally, we show that GlideNet excels in training starved real-world scenarios.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{metwaly2022glidenetgloballocalintrinsic,
 title={GlideNet: Global, Local and Intrinsic based Dense Embedding NETwork for Multi-category Attributes Prediction}, 
 author={Kareem Metwaly and Aerin Kim and Elliot Branson and Vishal Monga},
 year={2022},
 eprint={2203.03079},
 archivePrefix={arXiv},
 primaryClass={cs.CV},
 url={https://arxiv.org/abs/2203.03079}, 
}</code></pre>

GlideNet.png

GlideNet enhances multi-category attribute prediction by combining global, local, and intrinsic feature extraction, achieving 5% gain in large-scale datasets

GlideNet: Global, Local and Intrinsic based Dense Embedding NETwork for Multi-category Attributes Prediction

Abstract: Self-driving vehicles are the future of transportation. With current advancements in this field, the world is getting closer to safe roads with almost zero probability of having accidents and eliminating human errors. However, there is still plenty of research and development necessary to reach a level of robustness. One important aspect is to understand a scene fully including all details. As some characteristics (attributes) of objects in a scene (drivers' behavior for instance) could be imperative for correct decision making. However, current algorithms suffer from low-quality datasets with such rich attributes. Therefore, in this paper, we present a new dataset for attributes recognition -- Cityscapes Attributes Recognition (CAR). The new dataset extends the well-known dataset Cityscapes by adding an additional yet important annotation layer of attributes of objects in each image. Currently, we have annotated more than 32k instances of various categories (Vehicles, Pedestrians, etc.). The dataset has a structured and tailored taxonomy where each category has its own set of possible attributes. The tailored taxonomy focuses on attributes that is of most beneficent for developing better self-driving algorithms that depend on accurate computer vision and scene comprehension. We have also created an API for the dataset to ease the usage of CAR. The API can be accessed through&nbsp;<a class="link-external link-https" href="https://github.com/kareem-metwaly/CAR-API" rel="external noopener nofollow">this https URL</a>.
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{metwaly2021carcityscapesattributes,
 title={CAR -- Cityscapes Attributes Recognition A Multi-category Attributes Dataset for Autonomous Vehicles}, 
 author={Kareem Metwaly and Aerin Kim and Elliot Branson and Vishal Monga},
 year={2021},
 eprint={2111.08243},
 archivePrefix={arXiv},
 primaryClass={cs.CV},
 url={https://arxiv.org/abs/2111.08243}, 
}</code></pre>

Cityscapes - 01.png

Cityscapes Attributes Recognition (CAR) enhances self-driving algorithms with 32k+ annotated instances, adding vital object attributes for improved scene comprehension.

Products

Enterprise

Government

Resources

Customers

Leaderboards →

Scale AI Research

Scale AI’s mission is to accelerate the development of AI applications. By advancing research, we aim to create AI systems capable of solving complex, human-level problems.

Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning

FORTRESS: Frontier Risk Evaluation for National Security and Public Safety

Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models

Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards

A Red Teaming Roadmap Towards System-Level Safety

Assessing Robustness to Spurious Correlations in Post-Training Language Models

Relevance Isn't All You Need: Scaling RAG Systems With Inference-Time Compute Via Multi-Criteria Reranking

Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language Models

The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems

LLM Leaderboards

Expert-Led Private Evaluations for precise and reliable LLM rankings

Join the Scale AI Research Team

News

The future of your industry starts here