experiment

HeroTitleAsset

AnnouncementsSlider

Header

Research

Scale AI aims to accelerate AI development, advancing research to create systems capable of solving complex, human-level problems.

Research | Scale AI

CTAs

Footer

PreFooter

SectionTitle

BoxAsset

ResearchPapersList

LoopingVideo

Positions

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs

ENIGMAEVAL: A Benchmark of Long Multimodal Reasoning Challenges

MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems

Advancing Safe and Reliable AI: Scale's Research in Post-Training, Reasoning, and Evaluation

Humanity's Last Exam

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for Fine-Tuning LLMs

Microsoft Azure and Scale AI Collaborate to help Enterprises Deliver Powerful GenAI Solutions

Scale Public Sector: Building on Our Progress in 2025

The New ChatGPT-4o Update Promises Better Writing, How Does It Compare To The New Claude 3.5 Sonnet?

Scale's SEAL Research Lab Launches Expert-Evaluated LLM Leaderboards

Submit Your Toughest Questions for Humanity's Last Exam

First Impressions of OpenAI’s o1

Scale announces collaboration with Microsoft Azure

Fortune has named Scale as one of their 50 AI Innovators

university logo-horizontal_BLK.png

Logotype Black.png

GCP_139x24px 1.svg

Group 58.png

Frame 66.png

Frame 65.png

Frame 64.png

launch code?.png

Frame 62.png

Frame 63.png

USEUCOM.svg

US_Indo-Pacific_Command_Seal.svg

Logos - Google.svg

Anduril.svg

scale logo.png

legal_new.svg

time_logo.svg

leaderboard.png

center.svg

aicpa-soc-logo-freelogovectors.net_ 3.svg

iso-2-1 1.svg

fedramp-logo-vert 2.svg

ntt_logo.svg

cengage_logo.svg.svg

dla-piper 1.svg

cisco.svg

Logos - Valeo.svg

50-AI-logo.svg

bcg.svg

Black.png

Logos - Global Atlantic.svg

Howard_Hughes_logo.png

4DV Analytics - png.png

Logos - Uncountable.svg

Logos - Grata.svg

stability ai.svg

Logos Antrhopic.svg

image.jpg

Logos - Brex.svg

Logos - Nuro.svg

openai.svg

kaleido_ai_black.svg

Luminar Logo White Horizontal Vector 4.svg

<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=de82194bd223ec1d6256ff350fcfcf74.png&amp;f=webp&amp;q=75&amp;w=3840" alt="" width="1512" height="1228"></p>
<p>Large Language Model (LLM) safeguards, which implement request refusals, have become a widely adopted mitigation strategy against misuse. At the intersection of adversarial machine learning and AI safety, safeguard red teaming has effectively identified critical vulnerabilities in state-of-the-art refusal-trained LLMs. However, in our view the many conference submissions on LLM red teaming do not, in aggregate, prioritize the right research problems. First, testing against clear product safety specifications should take a higher priority than abstract social biases or ethical principles. Second, red teaming should prioritize realistic threat models that represent the expanding risk landscape and what real attackers might do. Finally, we contend that system-level safety is a necessary step to move red teaming research forward, as AI models present new threats as well as affordances for threat mitigation (e.g., detection and banning of malicious users) once placed in a deployment context. Adopting these priorities will be necessary in order for red teaming research to adequately address the slate of new threats that rapid AI advances present today and will present in the very near future.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{wang2025redteamingroadmapsystemlevel,
      title={A Red Teaming Roadmap Towards System-Level Safety}, 
      author={Zifan Wang and Christina Q. Knight and Jeremy Kritz and Willow E. Primack and Julian Michael},
      year={2025},
      eprint={2506.05376},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2506.05376}, 
}</code></pre>

image (28).png

Red Teaming Roadmap

A Red Teaming Roadmap Towards System-Level Safety

<p>Supervised and preference-based fine-tuning techniques have become popular for aligning large language models (LLMs) with user intent and correctness criteria. However, real-world training data often exhibits spurious correlations -- arising from biases, dataset artifacts, or other "shortcut" features -- that can compromise a model's performance or generalization. In this paper, we systematically evaluate three post-training algorithms -- Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and KTO (Kahneman-Tversky Optimization) -- across a diverse set of synthetic tasks and spuriousness conditions. Our tasks span mathematical reasoning, constrained instruction-following, and document-grounded question answering. We vary the degree of spurious correlation (10% vs. 90%) and investigate two forms of artifacts: "Feature Ambiguity" and "Distributional Narrowness." Our results show that the models often but not always degrade under higher spuriousness. The preference-based methods (DPO/KTO) can demonstrate relative robustness in mathematical reasoning tasks. By contrast, SFT maintains stronger performance in complex, context-intensive tasks. These findings highlight that no single post-training strategy universally outperforms in all scenarios; the best choice depends on the type of target task and the nature of spurious correlations.</p>

Icon-11.png

Robustness to Spurious Correlations

Assessing Robustness to Spurious Correlations in Post-Training Language Models

<p>Modern Large Language Model (LLM) systems typically rely on Retrieval Augmented Generation (RAG) which aims to gather context that is useful for response generation. These RAG systems typically optimize strictly towards retrieving context that is maximally relevant to the query. However, conventional theory suggests that retrieval systems which seek to maximize context relevance without any additional explicit criteria can create information bottlenecks. We reaffirm this finding in the modern age of LLM's by showing that in standard RAG pipelines, maximizing for context relevance alone can degrade downstream response quality. In response, we show evaluations of existing RAG methods which account for both context relevance and answer quality. These evaluations introduce a novel finding that existing RAG systems scale poorly with inference time compute usage when considering our combined metric. We introduce "RErank BEyond reLevance (REBEL)", which enables RAG systems to scale with inference-time compute via injection of multi-criteria optimization using Chain-of-Thought prompting (and optionally Multi-Turn dialogue). Ultimately, this enables a new performance/speed tradeoff curve, where RAG systems are able to achieve both higher relevance of retrieved contexts and superior answer quality as inference time increases. Code for the implementation of our method in llama-index can be found at the following PR:&nbsp;<a class="link-external link-https" href="https://github.com/run-llama/llama_index/pull/17590" rel="external noopener nofollow">this https URL</a>. Code for running experiments using this llama-index implementation can be found at&nbsp;<a class="link-external link-https" href="https://github.com/microsoft/REBEL" rel="external noopener nofollow">this https URL</a>.</p>
<div class="metatable">&nbsp;</div>

Icon-4.png

Relevance Isn't All You Need: Scaling RAG Systems With Inference-Time Compute Via Multi-Criteria Reranking

Rerank Beyond Relevance

<p>Unit test generation has become a promising and important use case of LLMs. However, existing evaluation benchmarks for assessing LLM unit test generation capabilities focus on function- or class-level code rather than more practical and challenging projectlevel codebases. To address such limitation, we propose ProjectTest, a project-level benchmark for unit test generation covering Python, Java, and JavaScript. ProjectTest features 20 moderate-sized and high-quality projects per language. We evaluate nine frontier LLMs on ProjectTest and the results show that all frontier LLMs tested exhibit moderate performance on ProjectTest on Python and Java, highlighting the difficulty of ProjectTest. We also conduct a thorough error analysis, which shows that even frontier LLMs, such as Claude 3.5 Sonnet, have significant basic yet critical errors, including compilation and cascade errors. Motivated by this observation, we further evaluate all frontier LLMs under manual errorfixing and self-error-fixing scenarios to assess their potential when equipped with error-fixing mechanisms. Our code and dataset is available at&nbsp;<a href="https://github.com/YiboWANG214/ProjectTest">ProjectTest</a>.</p>

1.png

projecttest

ProjectTest: A Project-level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms

<p>As national security institutions increasingly integrate Artificial Intelligence (AI) into decision-making and content generation processes, understanding the inherent biases of large language models (LLMs) is crucial. This study presents a novel benchmark designed to evaluate the biases and preferences of seven prominent foundation models-Llama 3.1 8B Instruct, Llama 3.1 70B Instruct, GPT-4o, Gemini 1.5 Pro-002, Mixtral 8x22B, Claude 3.5 Sonnet, and Qwen2 72B-in the context of international relations (IR). We designed a bias discovery study around core topics in IR using 400-expert crafted scenarios to analyze results from our selected models. These scenarios focused on four topical domains including: military escalation, military and humanitarian intervention, cooperative behavior in the international system, and alliance dynamics. Our analysis reveals noteworthy variation among model recommendations based on scenarios designed for the four tested domains. Particularly, Qwen2 72B, Gemini 1.5 Pro-002 and Llama 3.1 8B Instruct models offered significantly more escalatory recommendations than Claude 3.5 Sonnet and GPT-4o models. All models exhibit some degree of country-specific biases, often recommending less escalatory and interventionist actions for China and Russia compared to the United States and the United Kingdom. These findings highlight the necessity for controlled deployment of LLMs in high-stakes environments, emphasizing the need for domain-specific evaluations and model fine-tuning to align with institutional objectives.</p>

A baseline Analysis - 01.png

Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language Models

<p style="text-align: left;">As large language models (LLMs) become more capable and agentic, the requirement for trust in their outputs grows significantly, yet at the same time concerns have been mounting that models may learn to lie in pursuit of their goals. To address these concerns, a body of work has emerged around the notion of &ldquo;honesty&rdquo; in LLMs, along with interventions aimed at mitigating deceptive behaviors. However, evaluations of honesty are currently highly limited, with no benchmark combining large scale and applicability to all models. Moreover, many benchmarks claiming to measure honesty in fact simply measure accuracy&mdash;the correctness of a model&rsquo;s beliefs&mdash;in disguise. In this work, we introduce a large-scale human-collected dataset for measuring honesty directly, allowing us to disentangle accuracy from honesty for the first time. Across a diverse set of LLMs, we find that while larger models obtain higher accuracy on our benchmark, they do not become more honest. Surprisingly, while most frontier LLMs obtain high scores on truthfulness bench- marks, we find a substantial propensity in frontier LLMs to lie when pressured to do so, resulting in low honesty&nbsp; cores on our benchmark. We find that simple methods, such as representation engineering interventions, can improve honesty. These results underscore the growing need for robust evaluations and effective interventions to ensure LLMs remain trustworthy.</p>
<p style="text-align: left;">Check out the Leaderboard <a href="https://scale.com/leaderboard/mask">here</a>.</p>

Icon-1.png

The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems

<p>As language models master existing reasoning benchmarks, we need new challenges to evaluate their cognitive frontiers. Puzzle-solving events are rich repositories of challenging multimodal problems that test a wide range of advanced reasoning and knowledge capabilities, making them a unique testbed for evaluating frontier language models. We introduce ENIGMAEVAL, a dataset of problems and solutions derived from puzzle competitions and events that probes models&rsquo; ability to perform implicit knowledge synthesis and multi-step deductive reasoning. Unlike existing reasoning and knowledge benchmarks, puzzle solving challenges models to discover hidden connections between seemingly unrelated pieces of information to uncover solution paths. The benchmark comprises 1184 puzzles of varying complexity &ndash; each typically requiring teams of skilled solvers hours to days to complete &ndash; with unambiguous, verifiable solutions that enable efficient evaluation. State-of-the-art language models achieve extremely low accuracy on these puzzles, even lower than other difficult benchmarks such as Humanity&rsquo;s Last Exam, unveiling models&rsquo; shortcomings when challenged with problems requiring unstructured and lateral reasoning.</p>
<p>See the Leaderboard results here: <a href="https://scale.com/leaderboard/enigma_eval">https://scale.com/leaderboard/enigma_eval</a></p>
<p>&nbsp;</p>

Detecting and preventing.png

ENIGMAEVAL

<p>Refusal training on Large Language Models (LLMs) prevents harmful outputs, yet this defense remains vulnerable to both automated and human-crafted jailbreaks. We present a novel LLM-as-red-teamer approach in which a human jailbreaks a refusal-trained LLM to make it willing to jailbreak itself or other LLMs. We refer to the jailbroken LLMs as <em>J<sub>2</sub></em> attackers, which can systematically evaluate target models using various red teaming strategies and improve its performance via in-context learning from the previous failures. Our experiments demonstrate that Sonnet-3.5 and Gemini-1.5-pro outperform other LLMs as <em>J<sub>2</sub></em>, achieving 93.0% and 91.0% attack success rates (ASRs) respectively against GPT-4o (and similar results across other capable LLMs) on Harmbench. Our work not only introduces a scalable approach to strategic red teaming&mdash;drawing inspiration from human red teamers, but also highlights jailbreaking-to-jailbreak as an overlooked failure mode of the safeguard. Specifically, an LLM can bypass its own safeguards by employing a jailbroken version of itself that is willing to assist in further jailbreaking. To prevent any direct misuse with <em>J<sub>2</sub></em>, while advancing research in AI safety, we publicly share our methodology while keeping specific prompting details private.&nbsp;</p>
<p>Try examples here: <span style="text-decoration: underline;"><strong><a href="https://scale.com/research/j2/playground">https://scale.com/research/j2/playground</a></strong></span></p>
<h1 dir="ltr"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=7edf18642ec432d2196e58b18f03b38e.png&amp;f=webp&amp;q=75" alt="" width="983" height="753"></h1>
<h3 dir="ltr">Highlight</h3>
<p dir="ltr">As the target of a <em>J<sub>2</sub></em> attacker can be its own backbone model, e.g. using <em>J<sub>2 </sub></em>(Gemini) to attack Gemini, we reveal a new failure mode in LLM safeguards. Specifically, while fully jailbreaking an LLM for all harmful behaviors remains challenging, an adversary can instead focus resources on bypassing the LLM&rsquo;s safeguards to develop <em>J<sub>2</sub></em> or similar attackers. Once this is achieved, the <em>J<sub>2</sub> </em>attacker can then take over and jailbreak the specific harmful behaviors.</p>
<h3 dir="ltr">Ethic Statement</h3>
<p dir="ltr">This research &mdash; including the methodology described in the paper, the code, and results &mdash; contains material that can allow users to generate harmful content from some public LLMs. Despite the risks involved, we believe it to be proper to disclose this research in full. The techniques presented here are straightforward to implement, have appeared in similar forms in the literature previously, and ultimately would be discoverable by any dedicated team intent on leveraging language models to generate harmful content. Given the ease of converting LLMs into <em>J<sub>2</sub></em>, we prioritize preventing misuse by keeping the jailbreaking prompts private, releasing only select portions of conversations between <em>J<sub>2</sub></em> and target LLMs for academic research.&nbsp;&nbsp;</p>
<p>&nbsp;</p>

J2 - Centered.png

J2: Jailbreaking to Jailbreak

<p>We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capability for their applications. MultiChallenge identifies four categories of challenges in multi-turn conversations that are not only common and realistic among current human-LLM interactions, but are also challenging to all current frontier LLMs. All 4 challenges require accurate instruction-following, context allocation, and in-context reasoning at the same time. We also develop LLM as judge with instance-level rubrics to facilitate an automatic evaluation method with fair agreement with experienced human raters. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet (June 2024) achieving just a 41.4% average accuracy.</p>
<p>Check out the Leaderboard results here: https://scale.com/leaderboard/multichallenge</p>

Improving the Accuracy.png

<p>Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce HUMANITY&rsquo;S LAST EXAM (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 3,000 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at <a href="https://lastexam.ai">https://lastexam.ai</a>.</p>
<p>If you use Humanity's Last Exam in your research, please cite our work. The full citation is available <a href="https://github.com/centerforaisafety/hle/blob/main/citation.txt">here</a>.&nbsp;</p>

Icon.png

Humanity’s Last Exam (HLE) tests LLMs with 3,000 expert-crafted questions, revealing the gap between AI and human knowledge.

image (12).png

<p><strong>Abstract:</strong> Multimodal language models can exhibit hallucinations in their outputs, which limits their reliability. The ability to automatically detect these errors is important for mitigating them, but has been less explored and existing efforts do not localize hallucinations, instead framing this as a classification task. In this work, we first pose multimodal hallucination detection as a sequence labeling task where models must localize hallucinated text spans and present a strong baseline model. Given the high cost of human annotations for this task, we propose an approach to improve the sample efficiency of these models by creating corrupted grounding data, which we use for pre-training. Leveraging phrase grounding data, we generate hallucinations to replace grounded spans and create hallucinated text. Experiments show that pre-training on this data improves sample efficiency when fine-tuning, and that the learning signal from the grounding data plays an important role in these improvements.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{whitehead2024pretrainingmultimodalhallucinationdetectors,
      title={Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data}, 
      author={Spencer Whitehead and Jacob Phillips and Sean Hendryx},
      year={2024},
      eprint={2409.00238},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2409.00238}, 
}</code></pre>

Pre-Training - 03.png

A sequence labeling approach enhances hallucination detection in multimodal language models, improving efficiency through pre-training on synthetic data.

Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data

<p>Despite recent advances in AI, the development of systems capable of executing complex, multi-step reasoning tasks involving multiple tools remains a significant challenge. Current benchmarks fall short in capturing the real-world complexity of tool-use reasoning, where verifying the correctness of not only the final answer but also the intermediate steps is important for evaluation, development, and identifying failures during inference time. To bridge this gap, we introduce ToolComp, a comprehensive benchmark designed to evaluate multi-step tool-use reasoning. ToolComp is developed through a collaboration between models and human annotators, featuring human-edited/verified prompts, final answers, and process supervision labels, allowing for the evaluation of both final outcomes and intermediate reasoning. Evaluation across six different model families demonstrates the challenging nature of our dataset, with the majority of models achieving less than 50% accuracy. Additionally, we generate synthetic training data to compare the performance of outcome-supervised reward models (ORMs) with process-supervised reward models (PRMs) to assess their ability to improve complex tool-use reasoning as evaluated by ToolComp. Our results show that PRMs generalize significantly better than ORMs, achieving a 19% and 11% improvement in rank@1 accuracy for ranking base and fine-tuned model trajectories, respectively. These findings highlight the critical role of process supervision in both the evaluation and training of AI models, paving the way for more robust and capable systems in complex, multi-step tool-use tasks.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{nath2025toolcompmultitoolreasoning,
      title={ToolComp: A Multi-Tool Reasoning &amp; Process Supervision Benchmark}, 
      author={Vaskar Nath and Pranav Raja and Claire Yoon and Sean Hendryx},
      year={2025},
      eprint={2501.01290},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2501.01290}, 
}</code></pre>

ToolComp - 02.png

ToolComp is a benchmark for evaluating multi-step tool-use reasoning in AI, emphasizing both final outcomes and intermediate steps to enhance model reliability.

ToolComp: A Multi-Tool Reasoning & Process Supervision Benchmark

<p><strong>Abstract:</strong> The White House Executive Order on Artificial Intelligence highlights the risks of large language models (LLMs) empowering malicious actors in developing biological, cyber, and chemical weapons. To measure these risks of malicious use, government institutions and major AI labs are developing evaluations for hazardous capabilities in LLMs. However, current evaluations are private, preventing further research into mitigating risk. Furthermore, they focus on only a few, highly specific pathways for malicious use. To fill these gaps, we publicly release the Weapons of Mass Destruction Proxy (WMDP) benchmark, a dataset of 3,668 multiple-choice questions that serve as a proxy measurement of hazardous knowledge in biosecurity, cybersecurity, and chemical security. WMDP was developed by a consortium of academics and technical consultants, and was stringently filtered to eliminate sensitive information prior to public release. WMDP serves two roles: first, as an evaluation for hazardous knowledge in LLMs, and second, as a benchmark for unlearning methods to remove such hazardous knowledge. To guide progress on unlearning, we develop RMU, a state-of-the-art unlearning method based on controlling model representations. RMU reduces model performance on WMDP while maintaining general capabilities in areas such as biology and computer science, suggesting that unlearning may be a concrete path towards reducing malicious use from LLMs. We release our benchmark and code publicly at&nbsp;<a class="link-external link-https" href="https://wmdp.ai/" rel="external noopener nofollow">this https URL</a>.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{li2024wmdpbenchmarkmeasuringreducing,
      title={The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning}, 
      author={Nathaniel Li and Alexander Pan and Anjali Gopal and Summer Yue and Daniel Berrios and Alice Gatti and Justin D. Li and Ann-Kathrin Dombrowski and Shashwat Goel and Long Phan and Gabriel Mukobi and Nathan Helm-Burger and Rassin Lababidi and Lennart Justen and Andrew B. Liu and Michael Chen and Isabelle Barrass and Oliver Zhang and Xiaoyuan Zhu and Rishub Tamirisa and Bhrugu Bharathi and Adam Khoja and Zhenqi Zhao and Ariel Herbert-Voss and Cort B. Breuer and Samuel Marks and Oam Patel and Andy Zou and Mantas Mazeika and Zifan Wang and Palash Oswal and Weiran Lin and Adam A. Hunt and Justin Tienken-Harder and Kevin Y. Shih and Kemper Talley and John Guan and Russell Kaplan and Ian Steneker and David Campbell and Brad Jokubaitis and Alex Levinson and Jean Wang and William Qian and Kallol Krishna Karmakar and Steven Basart and Stephen Fitz and Mindy Levine and Ponnurangam Kumaraguru and Uday Tupakula and Vijay Varadharajan and Ruoyu Wang and Yan Shoshitaishvili and Jimmy Ba and Kevin M. Esvelt and Alexandr Wang and Dan Hendrycks},
      year={2024},
      eprint={2403.03218},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2403.03218}, 
}</code></pre>

The WMDP - 01.png

The WMDP benchmark evaluates hazardous knowledge in LLMs, supporting research on risk mitigation and unlearning methods to reduce potential misuse.

The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning

<p><strong>Abstract:</strong> As large language models (LLMs) become increasingly prevalent in diverse applications, ensuring the utility and safety of model generations becomes paramount. We present a holistic approach for test and evaluation of large language models. The approach encompasses the design of evaluation taxonomies, a novel framework for safety testing, and hybrid methodologies that improve the scalability and cost of evaluations. In particular, we introduce a hybrid methodology for the evaluation of large language models (LLMs) that leverages both human expertise and AI assistance. Our hybrid methodology generalizes across both LLM capabilities and safety, accurately identifying areas where AI assistance can be used to automate this evaluation. Similarly, we find that by combining automated evaluations, generalist red teamers, and expert red teamers, we&rsquo;re able to more efficiently discover new vulnerabilities. We share our approach in hopes that it can contribute to the development of common standards and approaches around test and evaluation of LLMs.</p>

A holistic approach.png

This approach combines human expertise and AI to test and evaluate LLMs, enhancing safety, scalability, and cost efficiency while identifying vulnerabilities.

A Holistic Approach For Test And Evaluation Of Large Language Models

<p><strong>Abstract:</strong>&nbsp;Foundation models, specifically Large Language Models (LLMs), have lately gained wide-spread attention and adoption. Reinforcement Learning with Human Feedback (RLHF) involves training a reward model to capture desired behaviors, which is then used to align LLM's. These reward models are additionally used at inference-time to estimate LLM responses' adherence to those desired behaviors. However, there is little work measuring how robust these reward models are to distribution shifts. In this work, we evaluate how reward model performance - measured via accuracy and calibration (i.e. alignment between accuracy and confidence) - is affected by distribution shift. We show novel calibration patterns and accuracy drops due to OOD prompts and responses, and that the reward model is more sensitive to shifts in responses than prompts. Additionally, we adapt an OOD detection technique commonly used in classification to the reward model setting to detect these distribution shifts in prompts and responses.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{levine2024baselineanalysisrewardmodels,
      title={A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift}, 
      author={Will LeVine and Benjamin Pikus and Anthony Chen and Sean Hendryx},
      year={2024},
      eprint={2311.14743},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2311.14743}, 
}</code></pre>

Evaluating the robustness of reward models in RLHF for LLMs, focusing on accuracy, calibration, and the impact of distribution shifts on performance.

A Baseline Analysis of Reward Models’ Ability To Accurately Analyze Foundation Models Under Distribution Shift

<p><strong>Abstract:&nbsp;</strong>As deep neural networks become adopted in high-stakes domains, it is crucial to identify when inference inputs are Out-of-Distribution (OOD) so that users can be alerted of likely drops in performance and calibration despite high confidence -- ultimately to know when networks' decisions (and their uncertainty in those decisions) should be trusted. In this paper we introduce Ablated Learned Temperature Energy (or "AbeT" for short), an OOD detection method which lowers the False Positive Rate at 95\% True Positive Rate (FPR@95) by 43.43% in classification compared to state of the art without training networks in multiple stages or requiring hyperparameters or test-time backward passes. We additionally provide empirical insights as to why our model learns to distinguish between In-Distribution (ID) and OOD samples while only being explicitly trained on ID samples via exposure to misclassified ID examples at training time. Lastly, we show the efficacy of our method in identifying predicted bounding boxes and pixels corresponding to OOD objects in object detection and semantic segmentation, respectively -- with an AUROC increase of 5.15% in object detection and both a decrease in FPR@95 of 41.48% and an increase in AUPRC of 34.20% in semantic segmentation compared to previous state of the art.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{levine2024outofdistributiondetectionapplications,
      title={Out-of-Distribution Detection &amp; Applications With Ablated Learned Temperature Energy}, 
      author={Will LeVine and Benjamin Pikus and Jacob Phillips and Berk Norman and Fernando Amat Gil and Sean Hendryx},
      year={2024},
      eprint={2401.12129},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2401.12129}, 
}</code></pre>

Out of Distribution.png

AbeT improves OOD detection by reducing False Positive Rate at high True Positive rates, enhancing classification, object detection, and segmentation performance.

Out-of-Distribution Detection & Applications With Ablated Learned Temperature Energy

<p><strong>Abstract:&nbsp;</strong>Instruction-tuned large language models (LLMs) have demonstrated promising zero-shot generalization capabilities across various downstream tasks. Recent research has introduced multimodal capabilities to LLMs by integrating independently pretrained vision encoders through model grafting. These multimodal variants undergo instruction tuning, similar to LLMs, enabling effective zero-shot generalization for multimodal tasks. This study conducts a comparative analysis of different multimodal instruction tuning approaches and evaluates their performance across a range of tasks, including complex reasoning, conversation, image captioning, multiple-choice questions (MCQs), and binary classification. Through rigorous benchmarking and ablation experiments, we reveal key insights for guiding architectural choices when incorporating multimodal capabilities into LLMs. However, current approaches have limitations; they do not sufficiently address the need for a diverse multimodal instruction dataset, which is crucial for enhancing task generalization. Additionally, they overlook issues related to truthfulness and factuality when generating responses. These findings illuminate current methodological constraints in adapting language models for image comprehension and provide valuable guidance for researchers and practitioners seeking to harness multimodal versions of LLMs.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{garg2023performancemultimodallanguagemodels,
      title={On the Performance of Multimodal Language Models}, 
      author={Utsav Garg and Erhan Bas},
      year={2023},
      eprint={2310.03211},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2310.03211}, 
}</code></pre>

On the Performance.png

Multimodal instruction tuning in LLMs enhances task generalization, revealing insights on architectural choices and addressing challenges in truthfulness and image comprehension.

On the Performance of Multimodal Language Models

<p><strong>Abstract</strong>: Instruction tuned Large Vision Language Models (LVLMs) have significantly advanced in generalizing across a diverse set of multi-modal tasks, especially for Visual Question Answering (VQA). However, generating detailed responses that are visually grounded is still a challenging task for these models. We find that even the current state-of-the-art LVLMs (InstructBLIP) still contain a staggering 30 percent of the hallucinatory text in the form of non-existent objects, unfaithful descriptions, and inaccurate relationships. To address this, we introduce M-HalDetect, a (M)ultimodal (Hal)lucination (Detect)ion Dataset that can be used to train and benchmark models for hallucination detection and prevention. M-HalDetect consists of 16k fine-grained annotations on VQA examples, making it the first comprehensive multi-modal hallucination detection dataset for detailed image descriptions. Unlike previous work that only consider object hallucination, we additionally annotate both entity descriptions and relationships that are unfaithful. To demonstrate the potential of this dataset for hallucination prevention, we optimize InstructBLIP through our novel Fine-grained Direct Preference Optimization (FDPO). We also train fine-grained multi-modal reward models from InstructBLIP and evaluate their effectiveness with best-of-n rejection sampling. We perform human evaluation on both FDPO and rejection sampling, and find that they reduce hallucination rates in InstructBLIP by 41% and 55% respectively. We also find that our reward model generalizes to other multi-modal models, reducing hallucinations in LLaVA and mPLUG-OWL by 15% and 57% respectively, and has strong correlation with human evaluated accuracy scores.</p>
<p>Accepted to main track for 38th AAAI Conference on Artificial Intelligence.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{gunjal2024detectingpreventinghallucinationslarge,
      title={Detecting and Preventing Hallucinations in Large Vision Language Models}, 
      author={Anisha Gunjal and Jihan Yin and Erhan Bas},
      year={2024},
      eprint={2308.06394},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2308.06394}, 
}</code></pre>

M-HalDetect, a dataset for hallucination detection in LVLMs, reduces hallucination rates in state-of-the-art models and improves multi-modal task accuracy through novel techniques.

Detecting and Preventing Hallucinations in Large Vision Language Models

<p><strong>Abstract:&nbsp;</strong>Calibration of deep learning models is crucial to their trustworthiness and safe&nbsp;usage, and as such, has been extensively studied in supervised classification models, with methods rafted to decrease miscalibration. However, there has yet to be a comprehensive study of the calibration of vision-language models that are&nbsp;used for zero-shot inference, like CLIP. We measure calibration across relevant&nbsp;variables like prompt, dataset, and architecture, and find that zero-shot inference&nbsp;with CLIP is miscalibrated. Furthermore, we propose a modified version of temperature scaling that is&nbsp; ligned with the common use cases of CLIP as a zero-shot inference model, and show that a single learned temperature generalizes for each&nbsp;specific CLIP model (defined by a chosen pre-training dataset and architecture)&nbsp;across inference dataset and prompt choice.</p>
<p>Accepted to workshop on the pitfalls of limited data and computation for Trustworthy ML, ICLR 2023.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{levine2023enablingcalibrationzeroshotinference,
      title={Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models}, 
      author={Will LeVine and Benjamin Pikus and Pranav Raja and Fernando Amat Gil},
      year={2023},
      eprint={2303.12748},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2303.12748}, 
}</code></pre>
<p>&nbsp;</p>

Enabling Calibration.png

This study evaluates and improves the calibration of vision-language models like CLIP for zero-shot inference, proposing a modified temperature scaling method for better generalization.

Enabling Calibration In The Zero-shot Inference Of Large Vision-Language Models

<p><strong>Abstract:</strong>&nbsp;As foundation models continue to exponentially scale in size, efficient methods of adaptation become increasingly critical. Parameter-efficient fine-tuning (PEFT), a recent class of techniques that require only modifying a small percentage of the model parameters, is currently the most popular method for adapting large language models (LLMs). Several PEFT techniques have recently been proposed with varying tradeoffs. We provide a comprehensive and uniform benchmark of various PEFT techniques across a representative LLM, the FLAN-T5 model, and evaluate model performance across different data scales of classification and generation datasets. Based on this, we provide a framework for choosing the optimal fine-tuning techniques given the task type and data availability. Contrary to popular belief, we also empirically prove that PEFT techniques converge slower than full tuning in low data scenarios, and posit the amount of data required for PEFT methods to both perform well and converge efficiently. Lastly, we further optimize these PEFT techniques by selectively choosing which parts of the model to train, and find that these techniques can be applied with significantly fewer parameters while maintaining and even improving performance.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{pu2023empiricalanalysisstrengthsweaknesses,
      title={Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs}, 
      author={George Pu and Anirudh Jain and Jihan Yin and Russell Kaplan},
      year={2023},
      eprint={2304.14999},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2304.14999}, 
}</code></pre>

Empirical Analysis - 01.png

Benchmarking parameter-efficient fine-tuning (PEFT) techniques for large language models, evaluating their performance across different datasets, and offering a framework for optimal adaptation.

Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs

<p><strong>Abstract:&nbsp;</strong>While prior research has proposed a plethora of methods that build neural classifiers robust against adversarial robustness, practitioners are still reluctant to adopt them due to their unacceptably severe clean accuracy penalties. This paper significantly alleviates this accuracy-robustness trade-off by mixing the output probabilities of a standard classifier and a robust classifier, where the standard network is optimized for clean accuracy and is not robust in general. We show that the robust base classifier's confidence difference for correct and incorrect examples is the key to this improvement. In addition to providing intuitions and empirical evidence, we theoretically certify the robustness of the mixed classifier under realistic assumptions. Furthermore, we adapt an adversarial input detector into a mixing network that adaptively adjusts the mixture of the two base models, further reducing the accuracy penalty of achieving robustness. The proposed flexible method, termed "adaptive smoothing", can work in conjunction with existing or even future methods that improve clean accuracy, robustness, or adversary detection. Our empirical evaluation considers strong attack methods, including AutoAttack and adaptive attack. On the CIFAR-100 dataset, our method achieves an 85.21% clean accuracy while maintaining a 38.72%&nbsp;<span id="MathJax-Element-1-Frame" class="MathJax" tabindex="0"><span id="MathJax-Span-1" class="math"><span id="MathJax-Span-2" class="mrow"><span id="MathJax-Span-3" class="msubsup"><span id="MathJax-Span-4" class="mi">ℓ</span><span id="MathJax-Span-5" class="mi">&infin;</span></span></span></span></span>-AutoAttacked (<span id="MathJax-Element-2-Frame" class="MathJax" tabindex="0"><span id="MathJax-Span-6" class="math"><span id="MathJax-Span-7" class="mrow"><span id="MathJax-Span-8" class="mi">ϵ</span><span id="MathJax-Span-9" class="mo">=</span><span id="MathJax-Span-10" class="mn">8</span><span id="MathJax-Span-11" class="texatom"><span id="MathJax-Span-12" class="mrow"><span id="MathJax-Span-13" class="mo">/</span></span></span><span id="MathJax-Span-14" class="mn">255</span></span></span></span>) accuracy, becoming the second most robust method on the RobustBench CIFAR-100 benchmark as of submission, while improving the clean accuracy by ten percentage points compared with all listed models. The code that implements our method is available at&nbsp;<a class="link-external link-https" href="https://github.com/Bai-YT/AdaptiveSmoothing" rel="external noopener nofollow">this https URL</a>.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{bai2024improvingaccuracyrobustnesstradeoffclassifiers,
      title={Improving the Accuracy-Robustness Trade-Off of Classifiers via Adaptive Smoothing}, 
      author={Yatong Bai and Brendon G. Anderson and Aerin Kim and Somayeh Sojoudi},
      year={2024},
      eprint={2301.12554},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2301.12554}, 
}</code></pre>

Adaptive smoothing improves adversarial robustness in classifiers, achieving 85.21% clean accuracy and 38.72% ℓ∞-AutoAttacked accuracy on CIFAR-100.

Improving the Accuracy-Robustness Trade-Off of Classifiers via Adaptive Smoothing

<p>In this paper, we introduce DEBAGREEMENT, a dataset of 42,894 commentreply pairs from the popular discussion website reddit, annotated with agree, neutral or disagree labels. We collect data from five forums on reddit: r/BlackLivesMatter, r/Brexit, r/climate, r/democrats, r/Republican. For each forum, we select comment pairs such that they form altogether a user interaction graph. DEBAGREEMENT presents a challenge for Natural Language Processing (NLP) systems, as it contains slang, sarcasm and topic-specific jokes, often present in online exchanges. We evaluate the performance of state-of-the-art language models on a (dis)agreement detection task, and investigate the use of contextual information available (graph, authorship, and temporal information). Since recent research has shown that context, such as social context or knowledge graph information, enables language models to better perform on downstream NLP tasks, DEBAGREEMENT provides novel opportunities for combining graph-based and text-based machine learning techniques to detect (dis)agreements online.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@inproceedings{
pougu{\'e}-biyong2021debagreement,
title={{DEBAGREEMENT}: A comment-reply dataset for (dis)agreement detection in online debates},
author={John Pougu{\'e}-Biyong and Valentina Semenova and Alexandre Matton and Rachel Han and Aerin Kim and Renaud Lambiotte and Doyne Farmer},
booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)},
year={2021},
url={https://openreview.net/forum?id=udVUN__gFO}
}</code></pre>

Debagreement - 01.png

DEBAGREEMENT is a dataset of 42,894 comment-reply pairs from Reddit, annotated for agreement detection, incorporating graph-based and text-based techniques.

DEBAGREEMENT: A comment-reply dataset for (dis)agreement detection in online debates

<p><strong>Abstract:&nbsp;</strong>Although state-of-the-art object detection methods have shown compelling performance, models often are not robust to adversarial attacks and out-of-distribution data. We introduce a new dataset, Natural Adversarial Objects (NAO), to evaluate the robustness of object detection models. NAO contains 7,934 images and 9,943 objects that are unmodified and representative of real-world scenarios, but cause state-of-the-art detection models to misclassify with high confidence. The mean average precision (mAP) of EfficientDet-D7 drops 74.5% when evaluated on NAO compared to the standard MSCOCO validation set.<br><br>Moreover, by comparing a variety of object detection architectures, we find that better performance on MSCOCO validation set does not necessarily translate to better performance on NAO, suggesting that robustness cannot be simply achieved by training a more accurate model.<br><br>We further investigate why examples in NAO are difficult to detect and classify. Experiments of shuffling image patches reveal that models are overly sensitive to local texture. Additionally, using integrated gradients and background replacement, we find that the detection model is reliant on pixel information within the bounding box, and insensitive to the background context when predicting class labels. NAO can be downloaded at&nbsp;<a class="link-external link-https" href="https://drive.google.com/drive/folders/15P8sOWoJku6SSEiHLEts86ORfytGezi8" rel="external noopener nofollow">this https URL</a>.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{lau2021naturaladversarialobjects,
      title={Natural Adversarial Objects}, 
      author={Felix Lau and Nishant Subramani and Sasha Harrison and Aerin Kim and Elliot Branson and Rosanne Liu},
      year={2021},
      eprint={2111.04204},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2111.04204}, 
}</code></pre>

Natural Adversarial Objects.png

NAO is a dataset of 7,934 images and 9,943 objects designed to evaluate object detection models' robustness, causing significant misclassification.

Natural Adversarial Objects

<p><strong>Abstract</strong>: How does the accuracy of deep neural network models trained to classify clinical images of skin conditions vary across skin color? While recent studies demonstrate computer vision models can serve as a useful decision support tool in healthcare and provide dermatologist-level classification on a number of specific tasks, darker skin is underrepresented in the data. Most publicly available data sets do not include Fitzpatrick skin type labels. We annotate 16,577 clinical images sourced from two dermatology atlases with Fitzpatrick skin type labels and open-source these annotations. Based on these labels, we find that there are significantly more images of light skin types than dark skin types in this dataset. We train a deep neural network model to classify 114 skin conditions and find that the model is most accurate on skin types similar to those it was trained on. In addition, we evaluate how an algorithmic approach to identifying skin tones, individual typology angle, compares with Fitzpatrick skin type labels annotated by a team of human labelers.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{groh2021evaluatingdeepneuralnetworks,
      title={Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset}, 
      author={Matthew Groh and Caleb Harris and Luis Soenksen and Felix Lau and Rachel Han and Aerin Kim and Arash Koochek and Omar Badri},
      year={2021},
      eprint={2104.09957},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2104.09957}, 
}</code></pre>

Evaluating deep neural networks.png

Study analyzes deep neural networks' accuracy on skin condition images, revealing biases with underrepresentation of dark skin types in clinical datasets.

Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset

<p><strong>Abstract:&nbsp;</strong>Attaching attributes (such as color, shape, state, action) to object categories is an important computer vision problem. Attribute prediction has seen exciting recent progress and is often formulated as a multi-label classification problem. Yet significant challenges remain in: 1) predicting diverse attributes over multiple categories, 2) modeling attributes-category dependency, 3) capturing both global and local scene context, and 4) predicting attributes of objects with low pixel-count. To address these issues, we propose a novel multi-category attribute prediction deep architecture named GlideNet, which contains three distinct feature extractors. A global feature extractor recognizes what objects are present in a scene, whereas a local one focuses on the area surrounding the object of interest. Meanwhile, an intrinsic feature extractor uses an extension of standard convolution dubbed Informed Convolution to retrieve features of objects with low pixel-count. GlideNet uses gating mechanisms with binary masks and its self-learned category embedding to combine the dense embeddings. Collectively, the Global-Local-Intrinsic blocks comprehend the scene's global context while attending to the characteristics of the local object of interest. Finally, using the combined features, an interpreter predicts the attributes, and the length of the output is determined by the category, thereby removing unnecessary attributes. GlideNet can achieve compelling results on two recent and challenging datasets -- VAW and CAR -- for large-scale attribute prediction. For instance, it obtains more than 5\% gain over state of the art in the mean recall (mR) metric. GlideNet's advantages are especially apparent when predicting attributes of objects with low pixel counts as well as attributes that demand global context understanding. Finally, we show that GlideNet excels in training starved real-world scenarios.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{metwaly2022glidenetgloballocalintrinsic,
      title={GlideNet: Global, Local and Intrinsic based Dense Embedding NETwork for Multi-category Attributes Prediction}, 
      author={Kareem Metwaly and Aerin Kim and Elliot Branson and Vishal Monga},
      year={2022},
      eprint={2203.03079},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2203.03079}, 
}</code></pre>

GlideNet.png

GlideNet enhances multi-category attribute prediction by combining global, local, and intrinsic feature extraction, achieving 5% gain in large-scale datasets

GlideNet: Global, Local and Intrinsic based Dense Embedding NETwork for Multi-category Attributes Prediction

<p><strong>Abstract</strong>: Self-driving vehicles are the future of transportation. With current advancements in this field, the world is getting closer to safe roads with almost zero probability of having accidents and eliminating human errors. However, there is still plenty of research and development necessary to reach a level of robustness. One important aspect is to understand a scene fully including all details. As some characteristics (attributes) of objects in a scene (drivers' behavior for instance) could be imperative for correct decision making. However, current algorithms suffer from low-quality datasets with such rich attributes. Therefore, in this paper, we present a new dataset for attributes recognition -- Cityscapes Attributes Recognition (CAR). The new dataset extends the well-known dataset Cityscapes by adding an additional yet important annotation layer of attributes of objects in each image. Currently, we have annotated more than 32k instances of various categories (Vehicles, Pedestrians, etc.). The dataset has a structured and tailored taxonomy where each category has its own set of possible attributes. The tailored taxonomy focuses on attributes that is of most beneficent for developing better self-driving algorithms that depend on accurate computer vision and scene comprehension. We have also created an API for the dataset to ease the usage of CAR. The API can be accessed through&nbsp;<a class="link-external link-https" href="https://github.com/kareem-metwaly/CAR-API" rel="external noopener nofollow">this https URL</a>.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{metwaly2021carcityscapesattributes,
      title={CAR -- Cityscapes Attributes Recognition A Multi-category Attributes Dataset for Autonomous Vehicles}, 
      author={Kareem Metwaly and Aerin Kim and Elliot Branson and Vishal Monga},
      year={2021},
      eprint={2111.08243},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2111.08243}, 
}</code></pre>

Cityscapes - 01.png

Cityscapes Attributes Recognition (CAR) enhances self-driving algorithms with 32k+ annotated instances, adding vital object attributes for improved scene comprehension.

CAR – Cityscapes Attributes Recognition A Multi-category Attributes Dataset for Autonomous Vehicles

<p><strong>Abstract:</strong> High-quality labeled datasets play a crucial role in fueling the development of machine learning (ML), and in particular the development of deep learning (DL). However, since the emergence of the ImageNet dataset and the AlexNet model in 2012, the size of new open-source labeled vision datasets has remained roughly constant. Consequently, only a minority of publications in the computer vision community tackle supervised learning on datasets that are orders of magnitude larger than Imagenet.</p>
<p>In this paper, we survey computer vision research domains that study the effects of such large datasets on model performance across different vision tasks. We summarize the community&rsquo;s current understanding of those effects, and highlight some open questions related to training with massive datasets. In particular, we tackle: (a) The largest datasets currently used in computer vision research and the interesting takeaways from training on such datasets; (b) The effectiveness of pretraining on large datasets; (c) Recent advancements and hurdles facing synthetic datasets; (d) An overview of double descent and sample nonmonotonicity phenomena; and finally, (e) A brief discussion of lifelong/continual learning and how it fares compared to learning from huge labeled datasets in an offline setting. Overall, our findings are that research on optimization for deep learning focuses on perfecting the training routine and thus making DL models less data hungry, while research on synthetic datasets aims to offset the cost of data labeling. However, for the time being, acquiring non-synthetic labeled data remains indispensable to boost performance.</p>
<p><sup>1</sup>Scale AI, CA, USA, <sup>2</sup>Department of Mathematics, University of Maryland, MD, USA.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{emam2021statedatacomputervision,
      title={On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models}, 
      author={Zeyad Emam and Andrew Kondrich and Sasha Harrison and Felix Lau and Yushi Wang and Aerin Kim and Elliot Branson},
      year={2021},
      eprint={2108.00114},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2108.00114}, 
}</code></pre>

On the State.png

High-quality labeled datasets are vital for ML and DL, yet the size of open-source vision datasets has stagnated since ImageNet, limiting supervised learning progress.

On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models

<p><strong>Abstract: </strong>Documents are a core part of many businesses in many fields such as law, finance,&nbsp;and technology among others. Automatic understanding of documents such as&nbsp;invoices, contracts, and resumes is lucrative, opening up many new avenues of&nbsp;business. The fields of natural language processing and computer vision have seen&nbsp;tremendous progress through the development of deep learning such that these&nbsp;methods have started to become infused in contemporary document understanding&nbsp;systems. In this survey paper, we review different techniques for document understanding for documents written in English and consolidate methodologies present&nbsp;in literature to act as a jumping-off point for researchers exploring this area.</p>
<p>This paper has been accepted to the 34th NeurIPS 2020 Workshop: ML Retrospectives, Surveys &amp; Meta-analyses (ML-RSA).</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{subramani2021surveydeeplearningapproaches,
      title={A Survey of Deep Learning Approaches for OCR and Document Understanding}, 
      author={Nishant Subramani and Alexandre Matton and Malcolm Greaves and Adrian Lam},
      year={2021},
      eprint={2011.13534},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2011.13534}, 
}</code></pre>

A Survey of Deep - 02.png

Reviewing deep learning techniques for automatic document understanding, covering invoices, contracts, and resumes, to guide future research in NLP and computer vision.

A Survey of Deep Learning Approaches for OCR and Document Understanding

<p>As large language models (LLMs) are applied to more use cases, creating high-quality, task-specific datasets for fine-tuning becomes a bottleneck for model improvement. Using high-quality human data has been the most common approach to unlock model performance, but it is prohibitively expensive in many scenarios. Several alternative methods have also emerged, such as generating synthetic or hybrid data, but the effectiveness of these approaches remains unclear, especially in resource-constrained scenarios and tasks that are not easily verified. To investigate this, we group various synthetic data generation strategies into three representative categories &ndash; Answer Augmentation, Question Rephrase, and New Question &ndash; and study the performance of student LLMs trained under various constraints, namely seed instruction set size and query budget. We demonstrate that these strategies are not equally effective across settings. Notably, the optimal data generation strategy depends strongly on the ratio between the available teacher query budget and the size of the seed instruction set. When this ratio is low, generating new answers to existing questions proves most effective, but generating new questions becomes optimal as this ratio increases. Across all tasks, we find that the choice of augmentation method and other design choices matter substantially more in low to mid-data regimes than in high-data regimes. We provide a practical framework for selecting the appropriate augmentation method across settings, taking into account additional factors such as the scalability of each method, the importance of verifying synthetic data, and using different LLMs for synthetic data generation.</p>
<h3>Method</h3>
<p>In this work, we examine techniques in synthetic data that were initially introduced in the math reasoning domains and extend the analysis of these strategies to a more diverse collection of tasks and constraints (Setlur et al., 2024; Yu et al., 2023; Li et al., 2024). We choose supervised finetuning (SFT) as the learning objective for the student model, which requires a dataset consisting of instruction-response pairs. We identify three main types of synthetic data generation strategies pertinent to an SFT instruction set &ndash; Answer Augmentation, Question Rephrasing and New Question &ndash; that are in essence, either the transformations of the seed instructions by an augmenter LLM, generation of corresponding responses by a teacher LLM or both.</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=39260c0163d4676d6d4267b23ffa4f61.png&amp;f=webp&amp;q=75&amp;w=3840" alt="" width="1503" height="935"></p>
<p>In the above figure, we present an overview of our data generation methods. Given a seed instruction set, we have 3 different methods to create instruction-response pairs for fine-tuning our student model. We use an example seed instruction from the ARC-C training set with synthetic instructions and responses generated with Llama 3.1 70b Instruct.</p>
<p>For all experiments, we assess the effectiveness of each synthetic data generation strategy by comparing the accuracy of fine-tuned student model on the evaluation split of the dataset used in the experiment. First, we understand the effectiveness of each data generation strategy through scalability and data constraints. Next, we investigate the cost-effectiveness of creating new instructions and responses. Finally, we cover ablation studies focused on understanding the impact of different teacher models, verification of responses, and a different student model.</p>
<p><strong>This paper&nbsp;<em>Balancing Cost and Effectiveness of Synthetic Data&nbsp;Generation Strategies for LLMs</em> (<a href="https://arxiv.org/pdf/2409.19759">arxiv</a>), has been accepted to the <a href="https://sites.google.com/view/neurips2024-ftw">Fine-Tuning in Machine Learning (FITML) Workshop at NeurIPS 2024</a>.</strong></p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-markup"><code>@article{chan2024balancing,
  title={Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs},
  author={Chan, Yung-Chieh and Pu, George and Shanker, Apaar and Suresh, Parth and Jenks, Penn and Heyer, John and Denton, Sam},
  journal={arXiv preprint arXiv:2409.19759},
  year={2024}
}</code></pre>

Legal-Balance-v2 1.png

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for Fine-Tuning LLMs 

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs

Open Graph.png

Synthetic Data Generation Strategies for LLM Fine-Tuning

<p>The Superficial Alignment Hypothesis suggests that most of a language model's knowledge and skills come from its initial training, while post-training is about giving a model the right style and format. This study revisits this hypothesis by empirically studying the scaling behavior of post-training with&nbsp;increasing finetuning examples and evaluating them using objective task-specific standardized benchmarks for skills like math, coding, and reasoning.</p>
<p>The findings show that, similar to pre-training scaling laws, post-training task performance scales as a power law against the number of finetuning examples. For complex tasks like math and reasoning,a handful of post-training examples merely align the model stylistically but do not saturate performance on the benchmarks; more extensive examples lead to meaningful improvements in domains like multi-hop reasoning. This indicates that models aren&rsquo;t strictly limited by their initial training knowledge. With the right post-training, models can learn and apply new information to better tackle complex questions.&nbsp;</p>
<div class="c-pillow_file__description"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=2b88f5e0c1122df86e8052689d8e5522.png&amp;f=webp&amp;q=75" alt="" width="2293" height="717"></div>
<p><em>Performance improvements as finetuning data is scaled up, for models in the sub-10 Billion parameter range. The points are fitted with a power law curve of the form P &prop; D<sup>1/b</sup> . Model performance consistently scales in a power law fashion, across model families.</em></p>
<p>These findings suggest that evaluating models using a single preference vote can be misleading for&nbsp; reasoning tasks, especially those with an objective component. A holistic evaluation program covering both human preference alignment and rigorous reasoning benchmarks is required to&nbsp; valuate general purpose frontier models. This suggests the Superficial Alignment Hypothesis may oversimplify how language models learn and develop skills.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-markup"><code>@misc{raghavendra2024revisitingsuperficialalignmenthypothesis,
      title={Revisiting the Superficial Alignment Hypothesis}, 
      author={Mohit Raghavendra and Vaskar Nath and Sean Hendryx},
      year={2024},
      eprint={2410.03717},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.03717}, 
}</code></pre>
<p dir="ltr">&nbsp;</p>
<p>&nbsp;</p>

Shedding new light on the Superficial Alignment Hypothesis, and understanding of model's abilities and knowledge.

Products

Enterprise

Government

Resources

Customers

Leaderboards →

Scale AI Research

Scale AI’s mission is to accelerate the development of AI applications. By advancing research, we aim to create AI systems capable of solving complex, human-level problems.

A Red Teaming Roadmap Towards System-Level Safety

Assessing Robustness to Spurious Correlations in Post-Training Language Models

Relevance Isn't All You Need: Scaling RAG Systems With Inference-Time Compute Via Multi-Criteria Reranking

Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language Models

The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems

ENIGMAEVAL: A Benchmark of Long Multimodal Reasoning Challenges

J2: Jailbreaking to Jailbreak

ProjectTest: A Project-level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms

MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs

LLM Leaderboards

Expert-Led Private Evaluations for precise and reliable LLM rankings

Join the Scale AI Research Team

News

The future of your industry starts here.