experiment

HeroTitleAsset

AnnouncementsSlider

Header

Research

Scale AI aims to accelerate AI development, advancing research to create systems capable of solving complex, human-level problems.

Research | Scale AI

CTAs

Footer

PreFooter

SectionTitle

BoxAsset

ResearchPapersList

LoopingVideo

Positions

NextImage

AnnouncementContextProvider

Screen

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Advancing Safe and Reliable AI: Scale's Research in Post-Training, Reasoning, and Evaluation

Humanity's Last Exam

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for Fine-Tuning LLMs

Microsoft Azure and Scale AI Collaborate to help Enterprises Deliver Powerful GenAI Solutions

Scale Public Sector: Building on Our Progress in 2025

The New ChatGPT-4o Update Promises Better Writing, How Does It Compare To The New Claude 3.5 Sonnet?

Scale's SEAL Research Lab Launches Expert-Evaluated LLM Leaderboards

Submit Your Toughest Questions for Humanity's Last Exam

First Impressions of OpenAI’s o1

Scale announces collaboration with Microsoft Azure

Fortune has named Scale as one of their 50 AI Innovators

scale logo.png

legal_new.svg

TIME_logo.png

leaderboard.png

center.svg

aicpa-soc-logo-freelogovectors.net_ 3.svg

iso-2-1 1.svg

fedramp-logo-vert 2.svg

ntt_logo.svg

cengage_logo.svg.svg

DLA_Piper_Logo 1.svg

Cisco_logo 1.svg

Logos - Valeo.svg

50-AI-logo.svg

bcg.svg

Black.png

Logos - Global Atlantic.svg

Howard_Hughes_logo.png

4DV Analytics - png.png

Logos - Uncountable.svg

Logos - Grata.svg

stability ai.svg

Logos Antrhopic.svg

Logos - Brex.svg

image.jpg

Logos - Nuro.svg

openai.svg

kaleido_ai_black.svg

Luminar Logo White Horizontal Vector 4.svg

<p>As language models master existing reasoning benchmarks, we need new challenges to evaluate their cognitive frontiers. Puzzle-solving events are rich repositories of challenging multimodal problems that test a wide range of advanced reasoning and knowledge capabilities, making them a unique testbed for evaluating frontier language models. We introduce ENIGMAEVAL, a dataset of problems and solutions derived from puzzle competitions and events that probes models&rsquo; ability to perform implicit knowledge synthesis and multi-step deductive reasoning. Unlike existing reasoning and knowledge benchmarks, puzzle solving challenges models to discover hidden connections between seemingly unrelated pieces of information to uncover solution paths. The benchmark comprises 1184 puzzles of varying complexity &ndash; each typically requiring teams of skilled solvers hours to days to complete &ndash; with unambiguous, verifiable solutions that enable efficient evaluation. State-of-the-art language models achieve extremely low accuracy on these puzzles, even lower than other difficult benchmarks such as Humanity&rsquo;s Last Exam, unveiling models&rsquo; shortcomings when challenged with problems requiring unstructured and lateral reasoning.</p>
<p>See the Leaderboard results here: <a href="https://scale.com/leaderboard/enigma_eval">https://scale.com/leaderboard/enigma_eval</a></p>
<p>&nbsp;</p>

Detecting and preventing.png

ENIGMAEVAL

ENIGMAEVAL: A Benchmark of Long Multimodal Reasoning Challenges

<p>Refusal training on Large Language Models (LLMs) prevents harmful outputs, yet this defense remains vulnerable to both automated and human-crafted jailbreaks. We present a novel LLM-as-red-teamer approach in which a human jailbreaks a refusal-trained LLM to make it willing to jailbreak itself or other LLMs. We refer to the jailbroken LLMs as <em>J<sub>2</sub></em> attackers, which can systematically evaluate target models using various red teaming strategies and improve its performance via in-context learning from the previous failures. Our experiments demonstrate that Sonnet-3.5 and Gemini-1.5-pro outperform other LLMs as <em>J<sub>2</sub></em>, achieving 93.0% and 91.0% attack success rates (ASRs) respectively against GPT-4o (and similar results across other capable LLMs) on Harmbench. Our work not only introduces a scalable approach to strategic red teaming&mdash;drawing inspiration from human red teamers, but also highlights jailbreaking-to-jailbreak as an overlooked failure mode of the safeguard. Specifically, an LLM can bypass its own safeguards by employing a jailbroken version of itself that is willing to assist in further jailbreaking. To prevent any direct misuse with <em>J<sub>2</sub></em>, while advancing research in AI safety, we publicly share our methodology while keeping specific prompting details private.&nbsp;</p>
<p>Try examples here: <span style="text-decoration: underline;"><strong><a href="https://scale.com/research/j2/playground">https://scale.com/research/j2/playground</a></strong></span></p>
<h1 dir="ltr"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=7edf18642ec432d2196e58b18f03b38e.png&amp;f=webp&amp;q=75" alt="" width="983" height="753"></h1>
<h1 dir="ltr">Highlight</h1>
<p dir="ltr">As the target of a <em>J<sub>2</sub></em> attacker can be its own backbone model, e.g. using <em>J<sub>2 </sub></em>(Gemini) to attack Gemini, we reveal a new failure mode in LLM safeguards. Specifically, while fully jailbreaking an LLM for all harmful behaviors remains challenging, an adversary can instead focus resources on bypassing the LLM&rsquo;s safeguards to develop <em>J<sub>2</sub></em> or similar attackers. Once this is achieved, the <em>J<sub>2</sub> </em>attacker can then take over and jailbreak the specific harmful behaviors.</p>
<h1 dir="ltr">Ethic Statement</h1>
<p dir="ltr">This research &mdash; including the methodology described in the paper, the code, and results &mdash; contains material that can allow users to generate harmful content from some public LLMs. Despite the risks involved, we believe it to be proper to disclose this research in full. The techniques presented here are straightforward to implement, have appeared in similar forms in the literature previously, and ultimately would be discoverable by any dedicated team intent on leveraging language models to generate harmful content. Given the ease of converting LLMs into <em>J<sub>2</sub></em>, we prioritize preventing misuse by keeping the jailbreaking prompts private, releasing only select portions of conversations between <em>J<sub>2</sub></em> and target LLMs for academic research.&nbsp;&nbsp;</p>
<p>&nbsp;</p>

J2 - Centered.png

J2: Jailbreaking to Jailbreak

<p>We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capability for their applications. MultiChallenge identifies four categories of challenges in multi-turn conversations that are not only common and realistic among current human-LLM interactions, but are also challenging to all current frontier LLMs. All 4 challenges require accurate instruction-following, context allocation, and in-context reasoning at the same time. We also develop LLM as judge with instance-level rubrics to facilitate an automatic evaluation method with fair agreement with experienced human raters. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet (June 2024) achieving just a 41.4% average accuracy.</p>

Improving the Accuracy.png

MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs

<p>Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce HUMANITY&rsquo;S LAST EXAM (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 3,000 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at <a href="https://lastexam.ai">https://lastexam.ai</a>.</p>
<p>If you use Humanity's Last Exam in your research, please cite our work. The full citation is available <a href="https://github.com/centerforaisafety/hle/blob/main/citation.txt">here</a>.&nbsp;</p>

Icon.png

Humanity’s Last Exam (HLE) tests LLMs with 3,000 expert-crafted questions, revealing the gap between AI and human knowledge.

image (12).png

<p><strong>Abstract:</strong> Multimodal language models can exhibit hallucinations in their outputs, which limits their reliability. The ability to automatically detect these errors is important for mitigating them, but has been less explored and existing efforts do not localize hallucinations, instead framing this as a classification task. In this work, we first pose multimodal hallucination detection as a sequence labeling task where models must localize hallucinated text spans and present a strong baseline model. Given the high cost of human annotations for this task, we propose an approach to improve the sample efficiency of these models by creating corrupted grounding data, which we use for pre-training. Leveraging phrase grounding data, we generate hallucinations to replace grounded spans and create hallucinated text. Experiments show that pre-training on this data improves sample efficiency when fine-tuning, and that the learning signal from the grounding data plays an important role in these improvements.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{whitehead2024pretrainingmultimodalhallucinationdetectors,
      title={Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data}, 
      author={Spencer Whitehead and Jacob Phillips and Sean Hendryx},
      year={2024},
      eprint={2409.00238},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2409.00238}, 
}</code></pre>

Pre-Training - 03.png

A sequence labeling approach enhances hallucination detection in multimodal language models, improving efficiency through pre-training on synthetic data.

Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data

<p>Despite recent advances in AI, the development of systems capable of executing complex, multi-step reasoning tasks involving multiple tools remains a significant challenge. Current benchmarks fall short in capturing the real-world complexity of tool-use reasoning, where verifying the correctness of not only the final answer but also the intermediate steps is important for evaluation, development, and identifying failures during inference time. To bridge this gap, we introduce ToolComp, a comprehensive benchmark designed to evaluate multi-step tool-use reasoning. ToolComp is developed through a collaboration between models and human annotators, featuring human-edited/verified prompts, final answers, and process supervision labels, allowing for the evaluation of both final outcomes and intermediate reasoning. Evaluation across six different model families demonstrates the challenging nature of our dataset, with the majority of models achieving less than 50% accuracy. Additionally, we generate synthetic training data to compare the performance of outcome-supervised reward models (ORMs) with process-supervised reward models (PRMs) to assess their ability to improve complex tool-use reasoning as evaluated by ToolComp. Our results show that PRMs generalize significantly better than ORMs, achieving a 19% and 11% improvement in rank@1 accuracy for ranking base and fine-tuned model trajectories, respectively. These findings highlight the critical role of process supervision in both the evaluation and training of AI models, paving the way for more robust and capable systems in complex, multi-step tool-use tasks.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{nath2025toolcompmultitoolreasoning,
      title={ToolComp: A Multi-Tool Reasoning &amp; Process Supervision Benchmark}, 
      author={Vaskar Nath and Pranav Raja and Claire Yoon and Sean Hendryx},
      year={2025},
      eprint={2501.01290},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2501.01290}, 
}</code></pre>

ToolComp - 02.png

ToolComp is a benchmark for evaluating multi-step tool-use reasoning in AI, emphasizing both final outcomes and intermediate steps to enhance model reliability.

ToolComp: A Multi-Tool Reasoning & Process Supervision Benchmark

<p><strong>Abstract:</strong> The White House Executive Order on Artificial Intelligence highlights the risks of large language models (LLMs) empowering malicious actors in developing biological, cyber, and chemical weapons. To measure these risks of malicious use, government institutions and major AI labs are developing evaluations for hazardous capabilities in LLMs. However, current evaluations are private, preventing further research into mitigating risk. Furthermore, they focus on only a few, highly specific pathways for malicious use. To fill these gaps, we publicly release the Weapons of Mass Destruction Proxy (WMDP) benchmark, a dataset of 3,668 multiple-choice questions that serve as a proxy measurement of hazardous knowledge in biosecurity, cybersecurity, and chemical security. WMDP was developed by a consortium of academics and technical consultants, and was stringently filtered to eliminate sensitive information prior to public release. WMDP serves two roles: first, as an evaluation for hazardous knowledge in LLMs, and second, as a benchmark for unlearning methods to remove such hazardous knowledge. To guide progress on unlearning, we develop RMU, a state-of-the-art unlearning method based on controlling model representations. RMU reduces model performance on WMDP while maintaining general capabilities in areas such as biology and computer science, suggesting that unlearning may be a concrete path towards reducing malicious use from LLMs. We release our benchmark and code publicly at&nbsp;<a class="link-external link-https" href="https://wmdp.ai/" rel="external noopener nofollow">this https URL</a>.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{li2024wmdpbenchmarkmeasuringreducing,
      title={The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning}, 
      author={Nathaniel Li and Alexander Pan and Anjali Gopal and Summer Yue and Daniel Berrios and Alice Gatti and Justin D. Li and Ann-Kathrin Dombrowski and Shashwat Goel and Long Phan and Gabriel Mukobi and Nathan Helm-Burger and Rassin Lababidi and Lennart Justen and Andrew B. Liu and Michael Chen and Isabelle Barrass and Oliver Zhang and Xiaoyuan Zhu and Rishub Tamirisa and Bhrugu Bharathi and Adam Khoja and Zhenqi Zhao and Ariel Herbert-Voss and Cort B. Breuer and Samuel Marks and Oam Patel and Andy Zou and Mantas Mazeika and Zifan Wang and Palash Oswal and Weiran Lin and Adam A. Hunt and Justin Tienken-Harder and Kevin Y. Shih and Kemper Talley and John Guan and Russell Kaplan and Ian Steneker and David Campbell and Brad Jokubaitis and Alex Levinson and Jean Wang and William Qian and Kallol Krishna Karmakar and Steven Basart and Stephen Fitz and Mindy Levine and Ponnurangam Kumaraguru and Uday Tupakula and Vijay Varadharajan and Ruoyu Wang and Yan Shoshitaishvili and Jimmy Ba and Kevin M. Esvelt and Alexandr Wang and Dan Hendrycks},
      year={2024},
      eprint={2403.03218},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2403.03218}, 
}</code></pre>

The WMDP - 01.png

The WMDP benchmark evaluates hazardous knowledge in LLMs, supporting research on risk mitigation and unlearning methods to reduce potential misuse.

The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning

<p><strong>Abstract:</strong> As large language models (LLMs) become increasingly prevalent in diverse applications, ensuring the utility and safety of model generations becomes paramount. We present a holistic approach for test and evaluation of large language models. The approach encompasses the design of evaluation taxonomies, a novel framework for safety testing, and hybrid methodologies that improve the scalability and cost of evaluations. In particular, we introduce a hybrid methodology for the evaluation of large language models (LLMs) that leverages both human expertise and AI assistance. Our hybrid methodology generalizes across both LLM capabilities and safety, accurately identifying areas where AI assistance can be used to automate this evaluation. Similarly, we find that by combining automated evaluations, generalist red teamers, and expert red teamers, we&rsquo;re able to more efficiently discover new vulnerabilities. We share our approach in hopes that it can contribute to the development of common standards and approaches around test and evaluation of LLMs.</p>

A holistic approach.png

This approach combines human expertise and AI to test and evaluate LLMs, enhancing safety, scalability, and cost efficiency while identifying vulnerabilities.

A Holistic Approach For Test And Evaluation Of Large Language Models

<p><strong>Abstract:</strong>&nbsp;Foundation models, specifically Large Language Models (LLMs), have lately gained wide-spread attention and adoption. Reinforcement Learning with Human Feedback (RLHF) involves training a reward model to capture desired behaviors, which is then used to align LLM's. These reward models are additionally used at inference-time to estimate LLM responses' adherence to those desired behaviors. However, there is little work measuring how robust these reward models are to distribution shifts. In this work, we evaluate how reward model performance - measured via accuracy and calibration (i.e. alignment between accuracy and confidence) - is affected by distribution shift. We show novel calibration patterns and accuracy drops due to OOD prompts and responses, and that the reward model is more sensitive to shifts in responses than prompts. Additionally, we adapt an OOD detection technique commonly used in classification to the reward model setting to detect these distribution shifts in prompts and responses.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{levine2024baselineanalysisrewardmodels,
      title={A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift}, 
      author={Will LeVine and Benjamin Pikus and Anthony Chen and Sean Hendryx},
      year={2024},
      eprint={2311.14743},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2311.14743}, 
}</code></pre>

A baseline Analysis - 01.png

Evaluating the robustness of reward models in RLHF for LLMs, focusing on accuracy, calibration, and the impact of distribution shifts on performance.

A Baseline Analysis of Reward Models’ Ability To Accurately Analyze Foundation Models Under Distribution Shift

<p><strong>Abstract:&nbsp;</strong>As deep neural networks become adopted in high-stakes domains, it is crucial to identify when inference inputs are Out-of-Distribution (OOD) so that users can be alerted of likely drops in performance and calibration despite high confidence -- ultimately to know when networks' decisions (and their uncertainty in those decisions) should be trusted. In this paper we introduce Ablated Learned Temperature Energy (or "AbeT" for short), an OOD detection method which lowers the False Positive Rate at 95\% True Positive Rate (FPR@95) by 43.43% in classification compared to state of the art without training networks in multiple stages or requiring hyperparameters or test-time backward passes. We additionally provide empirical insights as to why our model learns to distinguish between In-Distribution (ID) and OOD samples while only being explicitly trained on ID samples via exposure to misclassified ID examples at training time. Lastly, we show the efficacy of our method in identifying predicted bounding boxes and pixels corresponding to OOD objects in object detection and semantic segmentation, respectively -- with an AUROC increase of 5.15% in object detection and both a decrease in FPR@95 of 41.48% and an increase in AUPRC of 34.20% in semantic segmentation compared to previous state of the art.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{levine2024outofdistributiondetectionapplications,
      title={Out-of-Distribution Detection &amp; Applications With Ablated Learned Temperature Energy}, 
      author={Will LeVine and Benjamin Pikus and Jacob Phillips and Berk Norman and Fernando Amat Gil and Sean Hendryx},
      year={2024},
      eprint={2401.12129},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2401.12129}, 
}</code></pre>

Out of Distribution.png

AbeT improves OOD detection by reducing False Positive Rate at high True Positive rates, enhancing classification, object detection, and segmentation performance.

Out-of-Distribution Detection & Applications With Ablated Learned Temperature Energy

<p><strong>Abstract:&nbsp;</strong>Instruction-tuned large language models (LLMs) have demonstrated promising zero-shot generalization capabilities across various downstream tasks. Recent research has introduced multimodal capabilities to LLMs by integrating independently pretrained vision encoders through model grafting. These multimodal variants undergo instruction tuning, similar to LLMs, enabling effective zero-shot generalization for multimodal tasks. This study conducts a comparative analysis of different multimodal instruction tuning approaches and evaluates their performance across a range of tasks, including complex reasoning, conversation, image captioning, multiple-choice questions (MCQs), and binary classification. Through rigorous benchmarking and ablation experiments, we reveal key insights for guiding architectural choices when incorporating multimodal capabilities into LLMs. However, current approaches have limitations; they do not sufficiently address the need for a diverse multimodal instruction dataset, which is crucial for enhancing task generalization. Additionally, they overlook issues related to truthfulness and factuality when generating responses. These findings illuminate current methodological constraints in adapting language models for image comprehension and provide valuable guidance for researchers and practitioners seeking to harness multimodal versions of LLMs.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{garg2023performancemultimodallanguagemodels,
      title={On the Performance of Multimodal Language Models}, 
      author={Utsav Garg and Erhan Bas},
      year={2023},
      eprint={2310.03211},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2310.03211}, 
}</code></pre>

On the Performance.png

Multimodal instruction tuning in LLMs enhances task generalization, revealing insights on architectural choices and addressing challenges in truthfulness and image comprehension.

On the Performance of Multimodal Language Models

<p><strong>Abstract</strong>: Instruction tuned Large Vision Language Models (LVLMs) have significantly advanced in generalizing across a diverse set of multi-modal tasks, especially for Visual Question Answering (VQA). However, generating detailed responses that are visually grounded is still a challenging task for these models. We find that even the current state-of-the-art LVLMs (InstructBLIP) still contain a staggering 30 percent of the hallucinatory text in the form of non-existent objects, unfaithful descriptions, and inaccurate relationships. To address this, we introduce M-HalDetect, a (M)ultimodal (Hal)lucination (Detect)ion Dataset that can be used to train and benchmark models for hallucination detection and prevention. M-HalDetect consists of 16k fine-grained annotations on VQA examples, making it the first comprehensive multi-modal hallucination detection dataset for detailed image descriptions. Unlike previous work that only consider object hallucination, we additionally annotate both entity descriptions and relationships that are unfaithful. To demonstrate the potential of this dataset for hallucination prevention, we optimize InstructBLIP through our novel Fine-grained Direct Preference Optimization (FDPO). We also train fine-grained multi-modal reward models from InstructBLIP and evaluate their effectiveness with best-of-n rejection sampling. We perform human evaluation on both FDPO and rejection sampling, and find that they reduce hallucination rates in InstructBLIP by 41% and 55% respectively. We also find that our reward model generalizes to other multi-modal models, reducing hallucinations in LLaVA and mPLUG-OWL by 15% and 57% respectively, and has strong correlation with human evaluated accuracy scores.</p>
<p>Accepted to main track for 38th AAAI Conference on Artificial Intelligence.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{gunjal2024detectingpreventinghallucinationslarge,
      title={Detecting and Preventing Hallucinations in Large Vision Language Models}, 
      author={Anisha Gunjal and Jihan Yin and Erhan Bas},
      year={2024},
      eprint={2308.06394},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2308.06394}, 
}</code></pre>

M-HalDetect, a dataset for hallucination detection in LVLMs, reduces hallucination rates in state-of-the-art models and improves multi-modal task accuracy through novel techniques.

Detecting and Preventing Hallucinations in Large Vision Language Models

<p><strong>Abstract:&nbsp;</strong>Calibration of deep learning models is crucial to their trustworthiness and safe&nbsp;usage, and as such, has been extensively studied in supervised classification models, with methods rafted to decrease miscalibration. However, there has yet to be a comprehensive study of the calibration of vision-language models that are&nbsp;used for zero-shot inference, like CLIP. We measure calibration across relevant&nbsp;variables like prompt, dataset, and architecture, and find that zero-shot inference&nbsp;with CLIP is miscalibrated. Furthermore, we propose a modified version of temperature scaling that is&nbsp; ligned with the common use cases of CLIP as a zero-shot inference model, and show that a single learned temperature generalizes for each&nbsp;specific CLIP model (defined by a chosen pre-training dataset and architecture)&nbsp;across inference dataset and prompt choice.</p>
<p>Accepted to workshop on the pitfalls of limited data and computation for Trustworthy ML, ICLR 2023.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{levine2023enablingcalibrationzeroshotinference,
      title={Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models}, 
      author={Will LeVine and Benjamin Pikus and Pranav Raja and Fernando Amat Gil},
      year={2023},
      eprint={2303.12748},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2303.12748}, 
}</code></pre>
<p>&nbsp;</p>

Enabling Calibration.png

This study evaluates and improves the calibration of vision-language models like CLIP for zero-shot inference, proposing a modified temperature scaling method for better generalization.

Enabling Calibration In The Zero-shot Inference Of Large Vision-Language Models

<p><strong>Abstract:</strong>&nbsp;As foundation models continue to exponentially scale in size, efficient methods of adaptation become increasingly critical. Parameter-efficient fine-tuning (PEFT), a recent class of techniques that require only modifying a small percentage of the model parameters, is currently the most popular method for adapting large language models (LLMs). Several PEFT techniques have recently been proposed with varying tradeoffs. We provide a comprehensive and uniform benchmark of various PEFT techniques across a representative LLM, the FLAN-T5 model, and evaluate model performance across different data scales of classification and generation datasets. Based on this, we provide a framework for choosing the optimal fine-tuning techniques given the task type and data availability. Contrary to popular belief, we also empirically prove that PEFT techniques converge slower than full tuning in low data scenarios, and posit the amount of data required for PEFT methods to both perform well and converge efficiently. Lastly, we further optimize these PEFT techniques by selectively choosing which parts of the model to train, and find that these techniques can be applied with significantly fewer parameters while maintaining and even improving performance.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{pu2023empiricalanalysisstrengthsweaknesses,
      title={Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs}, 
      author={George Pu and Anirudh Jain and Jihan Yin and Russell Kaplan},
      year={2023},
      eprint={2304.14999},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2304.14999}, 
}</code></pre>

Empirical Analysis - 01.png

Benchmarking parameter-efficient fine-tuning (PEFT) techniques for large language models, evaluating their performance across different datasets, and offering a framework for optimal adaptation.

Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs

<p><strong>Abstract:&nbsp;</strong>While prior research has proposed a plethora of methods that build neural classifiers robust against adversarial robustness, practitioners are still reluctant to adopt them due to their unacceptably severe clean accuracy penalties. This paper significantly alleviates this accuracy-robustness trade-off by mixing the output probabilities of a standard classifier and a robust classifier, where the standard network is optimized for clean accuracy and is not robust in general. We show that the robust base classifier's confidence difference for correct and incorrect examples is the key to this improvement. In addition to providing intuitions and empirical evidence, we theoretically certify the robustness of the mixed classifier under realistic assumptions. Furthermore, we adapt an adversarial input detector into a mixing network that adaptively adjusts the mixture of the two base models, further reducing the accuracy penalty of achieving robustness. The proposed flexible method, termed "adaptive smoothing", can work in conjunction with existing or even future methods that improve clean accuracy, robustness, or adversary detection. Our empirical evaluation considers strong attack methods, including AutoAttack and adaptive attack. On the CIFAR-100 dataset, our method achieves an 85.21% clean accuracy while maintaining a 38.72%&nbsp;<span id="MathJax-Element-1-Frame" class="MathJax" tabindex="0"><span id="MathJax-Span-1" class="math"><span id="MathJax-Span-2" class="mrow"><span id="MathJax-Span-3" class="msubsup"><span id="MathJax-Span-4" class="mi">ℓ</span><span id="MathJax-Span-5" class="mi">&infin;</span></span></span></span></span>-AutoAttacked (<span id="MathJax-Element-2-Frame" class="MathJax" tabindex="0"><span id="MathJax-Span-6" class="math"><span id="MathJax-Span-7" class="mrow"><span id="MathJax-Span-8" class="mi">ϵ</span><span id="MathJax-Span-9" class="mo">=</span><span id="MathJax-Span-10" class="mn">8</span><span id="MathJax-Span-11" class="texatom"><span id="MathJax-Span-12" class="mrow"><span id="MathJax-Span-13" class="mo">/</span></span></span><span id="MathJax-Span-14" class="mn">255</span></span></span></span>) accuracy, becoming the second most robust method on the RobustBench CIFAR-100 benchmark as of submission, while improving the clean accuracy by ten percentage points compared with all listed models. The code that implements our method is available at&nbsp;<a class="link-external link-https" href="https://github.com/Bai-YT/AdaptiveSmoothing" rel="external noopener nofollow">this https URL</a>.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{bai2024improvingaccuracyrobustnesstradeoffclassifiers,
      title={Improving the Accuracy-Robustness Trade-Off of Classifiers via Adaptive Smoothing}, 
      author={Yatong Bai and Brendon G. Anderson and Aerin Kim and Somayeh Sojoudi},
      year={2024},
      eprint={2301.12554},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2301.12554}, 
}</code></pre>

Adaptive smoothing improves adversarial robustness in classifiers, achieving 85.21% clean accuracy and 38.72% ℓ∞-AutoAttacked accuracy on CIFAR-100.

Improving the Accuracy-Robustness Trade-Off of Classifiers via Adaptive Smoothing

<p>In this paper, we introduce DEBAGREEMENT, a dataset of 42,894 commentreply pairs from the popular discussion website reddit, annotated with agree, neutral or disagree labels. We collect data from five forums on reddit: r/BlackLivesMatter, r/Brexit, r/climate, r/democrats, r/Republican. For each forum, we select comment pairs such that they form altogether a user interaction graph. DEBAGREEMENT presents a challenge for Natural Language Processing (NLP) systems, as it contains slang, sarcasm and topic-specific jokes, often present in online exchanges. We evaluate the performance of state-of-the-art language models on a (dis)agreement detection task, and investigate the use of contextual information available (graph, authorship, and temporal information). Since recent research has shown that context, such as social context or knowledge graph information, enables language models to better perform on downstream NLP tasks, DEBAGREEMENT provides novel opportunities for combining graph-based and text-based machine learning techniques to detect (dis)agreements online.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@inproceedings{
pougu{\'e}-biyong2021debagreement,
title={{DEBAGREEMENT}: A comment-reply dataset for (dis)agreement detection in online debates},
author={John Pougu{\'e}-Biyong and Valentina Semenova and Alexandre Matton and Rachel Han and Aerin Kim and Renaud Lambiotte and Doyne Farmer},
booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)},
year={2021},
url={https://openreview.net/forum?id=udVUN__gFO}
}</code></pre>

Debagreement - 01.png

DEBAGREEMENT is a dataset of 42,894 comment-reply pairs from Reddit, annotated for agreement detection, incorporating graph-based and text-based techniques.

DEBAGREEMENT: A comment-reply dataset for (dis)agreement detection in online debates

<p><strong>Abstract:&nbsp;</strong>Although state-of-the-art object detection methods have shown compelling performance, models often are not robust to adversarial attacks and out-of-distribution data. We introduce a new dataset, Natural Adversarial Objects (NAO), to evaluate the robustness of object detection models. NAO contains 7,934 images and 9,943 objects that are unmodified and representative of real-world scenarios, but cause state-of-the-art detection models to misclassify with high confidence. The mean average precision (mAP) of EfficientDet-D7 drops 74.5% when evaluated on NAO compared to the standard MSCOCO validation set.<br><br>Moreover, by comparing a variety of object detection architectures, we find that better performance on MSCOCO validation set does not necessarily translate to better performance on NAO, suggesting that robustness cannot be simply achieved by training a more accurate model.<br><br>We further investigate why examples in NAO are difficult to detect and classify. Experiments of shuffling image patches reveal that models are overly sensitive to local texture. Additionally, using integrated gradients and background replacement, we find that the detection model is reliant on pixel information within the bounding box, and insensitive to the background context when predicting class labels. NAO can be downloaded at&nbsp;<a class="link-external link-https" href="https://drive.google.com/drive/folders/15P8sOWoJku6SSEiHLEts86ORfytGezi8" rel="external noopener nofollow">this https URL</a>.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{lau2021naturaladversarialobjects,
      title={Natural Adversarial Objects}, 
      author={Felix Lau and Nishant Subramani and Sasha Harrison and Aerin Kim and Elliot Branson and Rosanne Liu},
      year={2021},
      eprint={2111.04204},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2111.04204}, 
}</code></pre>

Natural Adversarial Objects.png

NAO is a dataset of 7,934 images and 9,943 objects designed to evaluate object detection models' robustness, causing significant misclassification.

Natural Adversarial Objects

<p><strong>Abstract</strong>: How does the accuracy of deep neural network models trained to classify clinical images of skin conditions vary across skin color? While recent studies demonstrate computer vision models can serve as a useful decision support tool in healthcare and provide dermatologist-level classification on a number of specific tasks, darker skin is underrepresented in the data. Most publicly available data sets do not include Fitzpatrick skin type labels. We annotate 16,577 clinical images sourced from two dermatology atlases with Fitzpatrick skin type labels and open-source these annotations. Based on these labels, we find that there are significantly more images of light skin types than dark skin types in this dataset. We train a deep neural network model to classify 114 skin conditions and find that the model is most accurate on skin types similar to those it was trained on. In addition, we evaluate how an algorithmic approach to identifying skin tones, individual typology angle, compares with Fitzpatrick skin type labels annotated by a team of human labelers.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{groh2021evaluatingdeepneuralnetworks,
      title={Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset}, 
      author={Matthew Groh and Caleb Harris and Luis Soenksen and Felix Lau and Rachel Han and Aerin Kim and Arash Koochek and Omar Badri},
      year={2021},
      eprint={2104.09957},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2104.09957}, 
}</code></pre>

Evaluating deep neural networks.png

Study analyzes deep neural networks' accuracy on skin condition images, revealing biases with underrepresentation of dark skin types in clinical datasets.

Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset

<p><strong>Abstract:&nbsp;</strong>Attaching attributes (such as color, shape, state, action) to object categories is an important computer vision problem. Attribute prediction has seen exciting recent progress and is often formulated as a multi-label classification problem. Yet significant challenges remain in: 1) predicting diverse attributes over multiple categories, 2) modeling attributes-category dependency, 3) capturing both global and local scene context, and 4) predicting attributes of objects with low pixel-count. To address these issues, we propose a novel multi-category attribute prediction deep architecture named GlideNet, which contains three distinct feature extractors. A global feature extractor recognizes what objects are present in a scene, whereas a local one focuses on the area surrounding the object of interest. Meanwhile, an intrinsic feature extractor uses an extension of standard convolution dubbed Informed Convolution to retrieve features of objects with low pixel-count. GlideNet uses gating mechanisms with binary masks and its self-learned category embedding to combine the dense embeddings. Collectively, the Global-Local-Intrinsic blocks comprehend the scene's global context while attending to the characteristics of the local object of interest. Finally, using the combined features, an interpreter predicts the attributes, and the length of the output is determined by the category, thereby removing unnecessary attributes. GlideNet can achieve compelling results on two recent and challenging datasets -- VAW and CAR -- for large-scale attribute prediction. For instance, it obtains more than 5\% gain over state of the art in the mean recall (mR) metric. GlideNet's advantages are especially apparent when predicting attributes of objects with low pixel counts as well as attributes that demand global context understanding. Finally, we show that GlideNet excels in training starved real-world scenarios.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{metwaly2022glidenetgloballocalintrinsic,
      title={GlideNet: Global, Local and Intrinsic based Dense Embedding NETwork for Multi-category Attributes Prediction}, 
      author={Kareem Metwaly and Aerin Kim and Elliot Branson and Vishal Monga},
      year={2022},
      eprint={2203.03079},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2203.03079}, 
}</code></pre>

GlideNet.png

GlideNet enhances multi-category attribute prediction by combining global, local, and intrinsic feature extraction, achieving 5% gain in large-scale datasets

GlideNet: Global, Local and Intrinsic based Dense Embedding NETwork for Multi-category Attributes Prediction

<p><strong>Abstract</strong>: Self-driving vehicles are the future of transportation. With current advancements in this field, the world is getting closer to safe roads with almost zero probability of having accidents and eliminating human errors. However, there is still plenty of research and development necessary to reach a level of robustness. One important aspect is to understand a scene fully including all details. As some characteristics (attributes) of objects in a scene (drivers' behavior for instance) could be imperative for correct decision making. However, current algorithms suffer from low-quality datasets with such rich attributes. Therefore, in this paper, we present a new dataset for attributes recognition -- Cityscapes Attributes Recognition (CAR). The new dataset extends the well-known dataset Cityscapes by adding an additional yet important annotation layer of attributes of objects in each image. Currently, we have annotated more than 32k instances of various categories (Vehicles, Pedestrians, etc.). The dataset has a structured and tailored taxonomy where each category has its own set of possible attributes. The tailored taxonomy focuses on attributes that is of most beneficent for developing better self-driving algorithms that depend on accurate computer vision and scene comprehension. We have also created an API for the dataset to ease the usage of CAR. The API can be accessed through&nbsp;<a class="link-external link-https" href="https://github.com/kareem-metwaly/CAR-API" rel="external noopener nofollow">this https URL</a>.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{metwaly2021carcityscapesattributes,
      title={CAR -- Cityscapes Attributes Recognition A Multi-category Attributes Dataset for Autonomous Vehicles}, 
      author={Kareem Metwaly and Aerin Kim and Elliot Branson and Vishal Monga},
      year={2021},
      eprint={2111.08243},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2111.08243}, 
}</code></pre>

Cityscapes - 01.png

Cityscapes Attributes Recognition (CAR) enhances self-driving algorithms with 32k+ annotated instances, adding vital object attributes for improved scene comprehension.

CAR – Cityscapes Attributes Recognition A Multi-category Attributes Dataset for Autonomous Vehicles

<p><strong>Abstract:</strong> High-quality labeled datasets play a crucial role in fueling the development of machine learning (ML), and in particular the development of deep learning (DL). However, since the emergence of the ImageNet dataset and the AlexNet model in 2012, the size of new open-source labeled vision datasets has remained roughly constant. Consequently, only a minority of publications in the computer vision community tackle supervised learning on datasets that are orders of magnitude larger than Imagenet.</p>
<p>In this paper, we survey computer vision research domains that study the effects of such large datasets on model performance across different vision tasks. We summarize the community&rsquo;s current understanding of those effects, and highlight some open questions related to training with massive datasets. In particular, we tackle: (a) The largest datasets currently used in computer vision research and the interesting takeaways from training on such datasets; (b) The effectiveness of pretraining on large datasets; (c) Recent advancements and hurdles facing synthetic datasets; (d) An overview of double descent and sample nonmonotonicity phenomena; and finally, (e) A brief discussion of lifelong/continual learning and how it fares compared to learning from huge labeled datasets in an offline setting. Overall, our findings are that research on optimization for deep learning focuses on perfecting the training routine and thus making DL models less data hungry, while research on synthetic datasets aims to offset the cost of data labeling. However, for the time being, acquiring non-synthetic labeled data remains indispensable to boost performance.</p>
<p><sup>1</sup>Scale AI, CA, USA, <sup>2</sup>Department of Mathematics, University of Maryland, MD, USA.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{emam2021statedatacomputervision,
      title={On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models}, 
      author={Zeyad Emam and Andrew Kondrich and Sasha Harrison and Felix Lau and Yushi Wang and Aerin Kim and Elliot Branson},
      year={2021},
      eprint={2108.00114},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2108.00114}, 
}</code></pre>

On the State.png

High-quality labeled datasets are vital for ML and DL, yet the size of open-source vision datasets has stagnated since ImageNet, limiting supervised learning progress.

On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models

<p><strong>Abstract: </strong>Documents are a core part of many businesses in many fields such as law, finance,&nbsp;and technology among others. Automatic understanding of documents such as&nbsp;invoices, contracts, and resumes is lucrative, opening up many new avenues of&nbsp;business. The fields of natural language processing and computer vision have seen&nbsp;tremendous progress through the development of deep learning such that these&nbsp;methods have started to become infused in contemporary document understanding&nbsp;systems. In this survey paper, we review different techniques for document understanding for documents written in English and consolidate methodologies present&nbsp;in literature to act as a jumping-off point for researchers exploring this area.</p>
<p>This paper has been accepted to the 34th NeurIPS 2020 Workshop: ML Retrospectives, Surveys &amp; Meta-analyses (ML-RSA).</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{subramani2021surveydeeplearningapproaches,
      title={A Survey of Deep Learning Approaches for OCR and Document Understanding}, 
      author={Nishant Subramani and Alexandre Matton and Malcolm Greaves and Adrian Lam},
      year={2021},
      eprint={2011.13534},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2011.13534}, 
}</code></pre>

A Survey of Deep - 02.png

Reviewing deep learning techniques for automatic document understanding, covering invoices, contracts, and resumes, to guide future research in NLP and computer vision.

A Survey of Deep Learning Approaches for OCR and Document Understanding

<p>As large language models (LLMs) are applied to more use cases, creating high-quality, task-specific datasets for fine-tuning becomes a bottleneck for model improvement. Using high-quality human data has been the most common approach to unlock model performance, but it is prohibitively expensive in many scenarios. Several alternative methods have also emerged, such as generating synthetic or hybrid data, but the effectiveness of these approaches remains unclear, especially in resource-constrained scenarios and tasks that are not easily verified. To investigate this, we group various synthetic data generation strategies into three representative categories &ndash; Answer Augmentation, Question Rephrase, and New Question &ndash; and study the performance of student LLMs trained under various constraints, namely seed instruction set size and query budget. We demonstrate that these strategies are not equally effective across settings. Notably, the optimal data generation strategy depends strongly on the ratio between the available teacher query budget and the size of the seed instruction set. When this ratio is low, generating new answers to existing questions proves most effective, but generating new questions becomes optimal as this ratio increases. Across all tasks, we find that the choice of augmentation method and other design choices matter substantially more in low to mid-data regimes than in high-data regimes. We provide a practical framework for selecting the appropriate augmentation method across settings, taking into account additional factors such as the scalability of each method, the importance of verifying synthetic data, and using different LLMs for synthetic data generation.</p>
<h3>Method</h3>
<p>In this work, we examine techniques in synthetic data that were initially introduced in the math reasoning domains and extend the analysis of these strategies to a more diverse collection of tasks and constraints (Setlur et al., 2024; Yu et al., 2023; Li et al., 2024). We choose supervised finetuning (SFT) as the learning objective for the student model, which requires a dataset consisting of instruction-response pairs. We identify three main types of synthetic data generation strategies pertinent to an SFT instruction set &ndash; Answer Augmentation, Question Rephrasing and New Question &ndash; that are in essence, either the transformations of the seed instructions by an augmenter LLM, generation of corresponding responses by a teacher LLM or both.</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=39260c0163d4676d6d4267b23ffa4f61.png&amp;f=webp&amp;q=75&amp;w=3840" alt="" width="1503" height="935"></p>
<p>In the above figure, we present an overview of our data generation methods. Given a seed instruction set, we have 3 different methods to create instruction-response pairs for fine-tuning our student model. We use an example seed instruction from the ARC-C training set with synthetic instructions and responses generated with Llama 3.1 70b Instruct.</p>
<p>For all experiments, we assess the effectiveness of each synthetic data generation strategy by comparing the accuracy of fine-tuned student model on the evaluation split of the dataset used in the experiment. First, we understand the effectiveness of each data generation strategy through scalability and data constraints. Next, we investigate the cost-effectiveness of creating new instructions and responses. Finally, we cover ablation studies focused on understanding the impact of different teacher models, verification of responses, and a different student model.</p>
<p><strong>This paper&nbsp;<em>Balancing Cost and Effectiveness of Synthetic Data&nbsp;Generation Strategies for LLMs</em> (<a href="https://arxiv.org/pdf/2409.19759">arxiv</a>), has been accepted to the <a href="https://sites.google.com/view/neurips2024-ftw">Fine-Tuning in Machine Learning (FITML) Workshop at NeurIPS 2024</a>.</strong></p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-markup"><code>@article{chan2024balancing,
  title={Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs},
  author={Chan, Yung-Chieh and Pu, George and Shanker, Apaar and Suresh, Parth and Jenks, Penn and Heyer, John and Denton, Sam},
  journal={arXiv preprint arXiv:2409.19759},
  year={2024}
}</code></pre>

Legal-Balance-v2 1.png

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for Fine-Tuning LLMs 

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs

Open Graph.png

Synthetic Data Generation Strategies for LLM Fine-Tuning

<p>The Superficial Alignment Hypothesis suggests that most of a language model's knowledge and skills come from its initial training, while post-training is about giving a model the right style and format. This study revisits this hypothesis by empirically studying the scaling behavior of post-training with&nbsp;increasing finetuning examples and evaluating them using objective task-specific standardized benchmarks for skills like math, coding, and reasoning.</p>
<p>The findings show that, similar to pre-training scaling laws, post-training task performance scales as a power law against the number of finetuning examples. For complex tasks like math and reasoning,a handful of post-training examples merely align the model stylistically but do not saturate performance on the benchmarks; more extensive examples lead to meaningful improvements in domains like multi-hop reasoning. This indicates that models aren&rsquo;t strictly limited by their initial training knowledge. With the right post-training, models can learn and apply new information to better tackle complex questions.&nbsp;</p>
<div class="c-pillow_file__description"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=2b88f5e0c1122df86e8052689d8e5522.png&amp;f=webp&amp;q=75" alt="" width="2293" height="717"></div>
<p><em>Performance improvements as finetuning data is scaled up, for models in the sub-10 Billion parameter range. The points are fitted with a power law curve of the form P &prop; D<sup>1/b</sup> . Model performance consistently scales in a power law fashion, across model families.</em></p>
<p>These findings suggest that evaluating models using a single preference vote can be misleading for&nbsp; reasoning tasks, especially those with an objective component. A holistic evaluation program covering both human preference alignment and rigorous reasoning benchmarks is required to&nbsp; valuate general purpose frontier models. This suggests the Superficial Alignment Hypothesis may oversimplify how language models learn and develop skills.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-markup"><code>@misc{raghavendra2024revisitingsuperficialalignmenthypothesis,
      title={Revisiting the Superficial Alignment Hypothesis}, 
      author={Mohit Raghavendra and Vaskar Nath and Sean Hendryx},
      year={2024},
      eprint={2410.03717},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.03717}, 
}</code></pre>
<p dir="ltr">&nbsp;</p>
<p>&nbsp;</p>

image (5).png

Shedding new light on the Superficial Alignment Hypothesis, and understanding of model's abilities and knowledge.

Revisiting the Superficial Alignment Hypothesis

Superficial Alignment 

<p>Techniques that learn improved representations via offline data or self-supervised objectives have shown impressive results in traditional reinforcement learning (RL). Nevertheless, it is unclear how improved representation learning can benefit reinforcement learning from human feedback (RLHF) on language models (LMs). In this work, we propose training reward models (RMs) in a contrastive, <em>goal-conditioned</em> fashion by increasing the representation similarity of future states along sampled preferred trajectories and decreasing the similarity along randomly sampled dispreferred trajectories. This objective significantly improves reward model performance by up to 0.09 AUROC across challenging benchmarks, such as MATH and GSM8k. These findings extend to general alignment as well &ndash; on the Helpful-Harmless dataset, we observe 2.3% increase in accuracy. Beyond improving reward model performance, we show this way of training RM representations enables improved <em>steerability</em> because it allows us to evaluate the likelihood of an action achieving a particular goal-state (<em>e.g.,</em> whether a solution is correct or helpful). Leveraging this insight, we find that we can filter up to 55% of generated tokens during majority voting by discarding trajectories likely to end up in an &ldquo;incorrect&rdquo; state, which leads to significant cost savings. We additionally find that these representations can perform fine-grained control by conditioning on desired future goal-states. For example, we show that steering a Llama 3 model towards helpful generations with our approach improves helpfulness by 9.6% over a supervised-fine-tuning trained baseline. Similarly, steering the model towards complex generations improves complexity by 21.6% over the baseline. Overall, we find that training RMs in this contrastive, goal-conditioned fashion significantly improves performance and enables model steerability.&nbsp;<sup>1</sup></p>
<h3>Introduction</h3>
<p>Aligning Language Models (LMs) with human preferences has proven to be an essential step for the adoption and safe use of these models, with the dominant paradigm being Reinforcement Learning from Human Feedback (RLHF) [50]. To accomplish this, a standard setup is to collect labels from humans for generated responses (<em>e.g.,</em> preferences, quality ratings) [50]. These labels can then be used to train a reward model to produce a ranking/scoring of a given sequence or set of sequences.</p>
<p><sup><br><img src="https://img.plasmic.app/img-optimizer/v1/img?src=dbb9caf9f36378276a549c36335c998c.png&amp;f=webp&amp;q=75"><br></sup></p>
<p><em>Figure 1: <strong>Overview of contrastive goal-conditioned learning for text</strong>. Pictured is a prompt with a preferred and dispreferred response. Both source state tokens (ten) for the positive and negative trajectory are sampled from the preferred response. For illustrative purposes, the positve and negative source states are sampled as the same token, but in practice they can be different. The positive goal state is sampled as some future token (subtract) from the preferred response, and the negative goal state is sampled from any token (add) from the dispreferred response. The corresponding representations are retrieved from the last hidden state of the reward model. The training objective is then to maximize and minimize the similarity of the positive and negative representation pairs, respectively.</em></p>
<p>The policy LM is then trained to maximize the expected returns from this reward model using a Reinforcement Learning (RL) algorithm.&nbsp;</p>
<p>High-quality representations have been shown to be an important piece for the success of RL algorithms [6, 42]. Although such representations can be learned during end-to-end training, many efforts have found it important to integrate more explicit representation learning components into RL algorithms, such as via data augmentation [39] or auxiliary losses [21]. Some work even casts certain RL algorithms as representation learning methods where using the similarity between state representations serves as a value function, demonstrating success on manipulation and navigation tasks [20]. Despite these successes in different areas, representation learning for aligning LMs has been less explored, while more emphasis has been placed on, e.g., pre-training reward models [7, 37] or learning from different types of rewards [15, 38].</p>
<p>In this paper, we present a simple yet effective approach to improve the representations learned by reward models for aligning LMs. We train LM-based reward models to learn representations that capture the expected reward or likelihood of achieving a goal state (e.g., correct solution to a problem, helpful response) at intermediate steps of the input sequence, inspired by goal-conditioned RL [5, 13, 20]. To do so, we use a contrastive objective and apply it to the reward model&rsquo;s hidden representations from desirable and undesirable sequences. Enforcing this loss on representations from intermediate steps of the sequence helps encode a dense signal as to which trajectories are more promising at different points in the sequence, which we show offers several useful properties, such as helping to localize errors or evaluating partial completed sequences. This method is flexible enough to support different kinds of alignment data and does not require further annotations beyond common sequence-level annotations.</p>
<p>We find that this approach improves the reward model&rsquo;s ability to identify correct/incorrect solutions in mathematical reasoning, boosting the AUROC on the task by up to 0.09 over standard preference ranking training. Towards natural language alignment, we find this method is able to increase the reward model&rsquo;s accuracy of identifying helpful versus harmless responses by 2.3% on the HelpfulHarmless dataset [8].</p>
<p>We also show the utility of the learned representations themselves, e.g., for filtering solutions to improve accuracy and steering the outputs towards responses with certain attributes in guided decoding (e.g., helpfulness, coherence, and complexity). For mathematical reasoning, we show that 2 we are able to filter up to 55% of generated tokens by discarding trajectories that are likely to lead to incorrect solutions as deemed by the learned representations while achieving similar or better performance. Similarly, using these representations to steer a Llama 3 model by conditioning on desired future goal-states, we improve helpfulness by 9.6%, correctness by 12.2%, coherence by 16.5%, and complexity by 21.6% over the supervised-fine-tuning trained baseline.</p>
<p>In summary, our contributions are as follows: 1) We explore improving the learned representations of reward models and its effect on LM alignment. Towards this, we present a simple and effective representation learning method based on a goal-conditioned contrastive objective. 2) We demonstrate that training reward models with this method can improve reward model performance on mathematical reasoning and helpfulness/harmlessness benchmarks. 3) We show that simply utilizing a reward model trained with this method can improve policy LM alignment on math reasoning benchmarks. 4) We investigate using these representations as a mechanism to evaluate the likelihood of achieving a desired goal state by filtering generations in a majority-vote scheme and guided decoding, showing that they can be used to increase accuracy and control policy LM outputs.<br><br></p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{nath2024learninggoalconditionedrepresentationslanguage,
      title={Learning Goal-Conditioned Representations for Language Reward Models}, 
      author={Vaskar Nath and Dylan Slack and Jeff Da and Yuntao Ma and Hugh Zhang and Spencer Whitehead and Sean Hendryx},
      year={2024},
      eprint={2407.13887},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.13887}, 
}</code></pre>

image (22).png

Exploring reinforcement learning from human feedback (RLHF) to align language models with human preferences for safer adoption and use.

Learning Goal-Conditioned Representations for Language Reward Models

Og Image.jpg

Learning Goal-Conditioned Representations 

<p dir="ltr">This study identifies a key limitation in scaling inference compute for LLMs: the lack of diversity in generated outputs, which leads to inefficient problem-solving. To address this, the authors propose PLANSEARCH, a novel search algorithm that generates diverse observations in natural language and constructs plans for solving problems based on these observations. PLANSEARCH significantly improves search diversity compared to baseline methods, achieving state-of-the-art results across coding benchmarks, including a 77% pass@200 on LiveCodeBench. The study also shows that performance gains from search algorithms can be predicted by measuring diversity in generated ideas.</p>
<p dir="ltr"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=51a27c0fcfc7865ab25ae6f338540566.png&amp;f=webp&amp;q=75" alt="" width="715" height="541"></p>
<p dir="ltr">The results show that both PLANSEARCH and IDEASEARCH significantly outperform basic sampling across all models and benchmarks, with PLANSEARCH achieving the highest overall scores. Interestingly, IDEASEARCH performs slightly better due to splitting the solution sketch into two model responses. Pass@k results vary across models, with the diversity of generated ideas playing a key role in these differences. The diversity score correlates strongly with performance improvements from scaling inference compute, making it a useful predictor for pass@k gains. However, PLANSEARCH can slightly reduce pass@1 by increasing idea diversity, but this trade-off results in better pass@k performance by improving the chance of generating at least one correct solution.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{wang2024planningnaturallanguageimproves,
      title={Planning In Natural Language Improves LLM Search For Code Generation}, 
      author={Evan Wang and Federico Cassano and Catherine Wu and Yunfeng Bai and Will Song and Vaskar Nath and Ziwen Han and Sean Hendryx and Summer Yue and Hugh Zhang},
      year={2024},
      eprint={2409.03733},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2409.03733}, 
}
</code></pre>

Network.png

PLANSEARCH improves search diversity in LLMs, enhancing problem-solving efficiency and achieving state-of-the-art results across coding benchmarks.

Planning In Natural Language Improves LLM Search For Code Generation

Open Graph - Netwokr.png

Planning Natural Language

<p dir="ltr">This study addresses concerns about dataset contamination in LLMs by introducing the Grade School Math 1000 (GSM1k) benchmark, designed to match the style and complexity of the widely-used GSM8k benchmark for mathematical reasoning. The research shows that many models, including Phi and Mistral, experience accuracy drops of up to 13% on GSM1k, indicating systematic overfitting, while frontier models like GPT and Claude show minimal overfitting. A positive correlation (Spearman&rsquo;s r&sup2; = 0.32) was found between a model's likelihood of generating examples from GSM8k and the performance gap between GSM8k and GSM1k, suggesting partial memorization of GSM8k by some models.</p>
<p dir="ltr"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=ed227464f2357d33d4b5b7c62d487b6c.png&amp;f=webp&amp;q=75" width="1233" height="994"></p>
<p style="text-align: left;">The study evaluated LLMs on both the GSM8k and GSM1k benchmarks using a standardized evaluation setup for fair comparison. Open-source models were tested using EleutherAI's LM Evaluation Harness, and closed-source models were queried through the LiteLLM library. All models were tested with identical prompts using five examples from the GSM8k train set. Results indicate that some lesser-known models, particularly those near the top of the OpenLLMLeaderboard, performed significantly worse on GSM1k, suggesting they may have over-optimized for GSM8k, in line with Goodhart&rsquo;s law.&nbsp;</p>
<h3 dir="ltr" style="text-align: center;"><strong>Analysis</strong></h3>
<ol>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Certain model families, notably Phi and Mistral, consistently overfit to GSM8k, showing better performance on it compared to GSM1k across almost all model versions and scales. Other model families, including Yi, Xwin, Gemma, and CodeLlama, also exhibit this pattern but to a lesser degree.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Many models do not display signs of overfitting and perform similarly on both GSM8k and GSM1k. Two possible reasons for this are: 1) these models possess advanced reasoning abilities, allowing them to generalize to new problems even if they have seen GSM8k data, and 2) frontier model builders may be more cautious about data contamination.&nbsp;</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Overfit Models Are Still Capable of Reasoning. The fact that a model is overfit does not mean that it is poor at reasoning, merely that it is not as good as the benchmarks might indicate it to be. In fact, we find that many of the most overfit models are still capable of reasoning and solving novel problems.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Data Contamination Is Likely Not The Full Explanation for Overfitting. This result suggests that some of the reason for overfitting is due to partial memorization of the test set. Overfitting on GSM8k may be through other indirect means, such as model builders collecting data similar in nature to benchmarks as training data or selecting final model checkpoints based on performance on benchmarks, even if the model itself may have not seen the GSM8k dataset at any point via training. Conversely, the reverse is also true: small amounts of data contamination do not necessarily lead to overfitting.</p>
</li>
</ol>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-markup"><code>@inproceedings{zhang2024carefulexaminationlargelanguage, title={A Careful Examination of Large Language Model Performance on Grade School Arithmetic}, author={Hugh Zhang and Jeff Da and Dean Lee and Vaughn Robinson and Catherine Wu and Will Song and Tiffany Zhao and Pranav Raja and Charlotte Zhuang and Dylan Slack and Qin Lyu and Sean Hendryx and Russell Kaplan and Michele Lunati and Summer Yue}, booktitle={Advances in Neural Information Processing Systems}, year={2024}</code></pre>

image 6.png

LLM performance on mathematical reasoning benchmarks may involve memorization rather than true reasoning ability.



A Careful Examination of Large Language Model Performance on Grade School Arithmetic

LLM Performance School

<p>Recent large language model (LLM) defenses have greatly improved models' ability to refuse harmful queries, even when adversarially attacked. However, LLM defenses are primarily evaluated against automated adversarial attacks in a single turn of conversation, an insufficient threat model for real-world malicious use. We demonstrate that multi-turn human jailbreaks uncover significant vulnerabilities, exceeding 70% attack success rate (ASR) on HarmBench against defenses that report single-digit ASRs with automated single-turn attacks. Human jailbreaks also reveal vulnerabilities in machine unlearning defenses, successfully recovering dual-use biosecurity knowledge from unlearned models. We compile these results into Multi-Turn Human Jailbreaks (MHJ), a dataset of 2,912 prompts across 537 multi-turn jailbreaks. We publicly release MHJ alongside a compendium of jailbreak tactics developed across dozens of commercial red teaming engagements, supporting research towards stronger LLM defenses.</p>
<p>Scale publishes the results of expert-driven private evaluations on <a title="SEAL Leaderboards" href="https://scale.com/leaderboard">SEAL Leaderboards</a>.</p>
<h3 style="text-align: center;">Main Results</h3>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=e11ecad7941e445e6c585502a7a7dde3.png&amp;f=webp&amp;q=75"></p>
<p>We compare our Multi-turn Human Jailbreaks (MHJ) results on HarmBench (consisting of 240 harmful behaviors) for its attack success rate (ASR) with the state-of-the-art automated attacks, targeting jailbreaking the state-of-the-art defense methods for improving LLM robustness. In the following plot, Ensemble Automated Attack is constructed such that it succeeds if any of the 6 automated attacks succeeds (Pass@6). Best Automated Attack shows the highest ASR for each defense from a single automated attack. For the detailed results and discussions, please see the full paper.Try examples <a href="https://scale.com/research/mhj/playground">here</a>.&nbsp;</p>
<h3 style="text-align: center;">Ethics and Disclosure</h3>
<p>In releasing the Multi-Turn Human Jailbreaks (MHJ) dataset, we carefully weighed the benefits of empowering research in defense robustness with the risks of enabling further malicious use. Following Zou et al. (2024), we believe the publication of MHJ poses low marginal risk, as datasets of many other manual jailbreaks are widely disseminated. Towards reducing risk, we removed a complete set of model completions and any jailbreaks that may contain sensitive information in the main paper and in the site. With the support of legal counsel, we verified MHJ&rsquo;s compliance with applicable U.S. export control requirements, including with respect to the International Traffic in Arms Regulations (22 CFR Parts 120-130) and Export Administration Regulations (15 CFR Parts 730-774).</p>
<p>We received explicit permission for red teaming any proprietary models. Prior to release, we also disclosed our findings and datasets to the authors of the defenses we examined.</p>
<h3 style="text-align: center;">Accessing the full set of MHJ</h3>
<p>After filtering sensitive &amp; export-controlled information, we publicly release the adversarial prompts in MHJ, supporting research towards stronger LLM defenses and robustness evaluations at Huggingface for research purpose only (under Creative Commons Attribution Non Commercial 4.0). Please apply for the access on <a href="https://huggingface.co/datasets/ScaleAI/mhj">Huggingface</a>.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-markup"><code>@misc{li2024llmdefensesrobustmultiturn,
      title={LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet}, 
      author={Nathaniel Li and Ziwen Han and Ian Steneker and Willow Primack and Riley Goodside and Hugh Zhang and Zifan Wang and Cristina Menghini and Summer Yue},
      year={2024},
      eprint={2408.15221},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2408.15221}, 
}</code></pre>

Jailbreak.png

Multi-turn human jailbreaks expose vulnerabilities in LLM defenses, achieving over 70% attack success.

LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet

Open Graph Jailbreak.png

<p>McConnell led opposition to stricter campaign finance regulations, with the Lexington Herald Leader describing it as McConnell's "pet issue."[8] He has argued that campaign finance regulations reduce participation in political campaigns and protect incumbents from competition.[9] In 1990, he led a filibuster against a campaign finance reform bill.[10] He spearheaded the movement against the Bipartisan Campaign Reform Act. His opposition to the bill culminated in the 2003 Supreme Court case McConnell v. Federal Election Commission and the 2010 Citizens United v. Federal Election Commission, the latter of which ruled part of the act unconstitutional. McConnell has been an advocate for free speech at least as far back as the early 1970s when he was teaching night courses at the University of Louisville. "No issue has shaped his career more than the intersection of campaign financing and free speech," political reporter Robert Costa wrote in 2012.[11] In a recording of a 2014 fundraiser McConnell expressed his disapproval of the McCain-Feingold law, saying,"The worst day of my political life was when President George W. Bush signed McCain-Feingold into law in the early part of his first Administration."[12] On January 2, 2013, the Public Campaign Action Fund, a liberal nonprofit group that backs stronger campaign finance regulation, released a report highlighting eight instances from McConnell's political career in which a vote or a blocked vote (filibuster), coincided with an influx of campaign contributions to McConnell's campaign.[13][14] In December 2018, McConnell opposed a Democratic resolution that overturned IRS guidance to reduce the amount of donor information certain tax-exempt groups would have to give the IRS, saying,"In a climate that is increasingly hostile to certain kinds of political expression and open debate, the last thing Washington needs to do is to chill the exercise of free speech and add to the sense of intimidation."[15] Prior to his Senate run, McConnell favored campaign finance reform, describing money in politics as a "cancer" in 1973.[10]</p>

<p>Cruz received more than US$1 million in campaign donations from the oil and gas industry since 2011.[104] Cruz is an opponent of the EPA, calling it "unbelievably abusive" and "populated by zealots,"[105] and has accused the Obama administration of waging a "war on coal."[105][106] Cruz opposes EPA's Clean Power Plan, which aims to cut carbon emissions from power plants; Cruz accused President Obama of engaging in a "lawless and radical attempt to destabilize the nation's energy system" and called the plan "flatly unconstitutional."[107] Cruz, who says that he is an advocate for "volunteer conservation," and also disagreed with efforts by the EPA to expand regulatory oversight on water, which Cruz characterized an attempt "to turn irrigation ditches into lakes and rivers and oceans."[108] Cruz is a supporter of TransCanada's controversial Keystone XL Pipeline,[106] and along with every other Republican senator was a cosponsor of legislation in support of the pipeline.[109] Like most Republican senators, Cruz supports opening areas for offshore drilling.[106] Cruz favors "expanding energy development on federal land" and has "pushed for preventing federal restrictions on fracking."[106] In July 2015, Cruz's super PAC,"Keep the Promise," disclosed that the billionaire brothers Dan and Farris Wilks, who earned a fortune in the West Texas fracking boom, contributed a record-setting $15 million to the super PAC.[110]</p>

<p>For safety reasons, large language models (LLMs) are trained to refuse harmful user instructions, such as assisting dangerous activities. We study an open question in this work. Can the desired safety refusal, typically enforced in chat contexts, be generalized to non-chat and agentic use cases? Unlike chatbots, LLM agents equipped with general-purpose tools, such as web browsers and mobile devices, can directly influence the real world, making it even more crucial to refuse harmful instructions. In this work, we primarily focus on red-teaming browser agents &ndash; LLMs that manipulate information via web browsers.&nbsp;<br><br>To this end, we introduce Browser Agent Red-teaming Toolkit (BrowserART), a comprehensive test suite designed specifically for red-teaming browser agents. BrowserART consists of 100 diverse browser-related harmful behaviors (including original behaviors and ones sourced from HarmBench [Mazeika et al., 2024] and AirBench 2024 [Zeng et al., 2024b]) across both synthetic and real websites.&nbsp;<br><br>Our empirical study on state-of-the-art browser agents reveals that, while the backbone LLM refuses harmful instructions as a chatbot, the corresponding agent does not. Moreover, attack methods designed to jailbreak refusal-trained LLMs in the chat settings transfer effectively to browser agents. With human rewrites, GPT-4o and o1-preview-based browser agents attempted 98 and 63 harmful behaviors (out of 100), respectively. We publicly release BrowserART and call on LLM developers, policymakers, and agent developers to collaborate on improving agent safety.</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=1b13997d80e8646969c95cee06965802.png&amp;f=webp&amp;q=75"></p>
<p>BrowserART consists of 100 harmful browser-related behaviors (including original behaviors and ones sourced from HarmBench [Mazeika et al., 2024] and AirBench 2024 [Zeng et al., 2024b]) that an agent is not supposed to assist. We divided all behaviors into two main categories: harmful content and harmful interaction. Under each main category, we created sub-categories for the harm semantics. We created 40 synthetic websites under 19 domains for red teaming browser behaviors that target specific websites (e.g., Twitter/X). These synthetic pages are hosted locally for running red teaming experiments in a sandbox without polluting the real world, especially social media and government sites.</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=6b9db5c096e4a32216e92f1b221d3af1.png&amp;f=webp&amp;q=75"></p>
<p>One of the major findings in our work is a clear gap between Attack Success Rates (ASRs) of the backbone LLM (blue) and its agent (lavender). That is, while the LLM refuses to follow the harmful behavior in the user instruction, the agent will. Gaps of GPT-4o and GPT-4-turbo models are most outstanding, compared to other backbone LLMs. In particular, we find Opus-3 and Llama-3.1-405B show the least drop in their refusal capabilities.</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=a10a6df8f013bd702541020a090824f4.png&amp;f=webp&amp;q=75"></p>
<h3 style="text-align: center;">Ethics and Disclosure</h3>
<p>This research &mdash; including the methodology detailed in the paper, the code, and the content of this webpage &mdash; contains material that may enable users to generate harmful content using certain publicly available LLM agents. While we recognize the associated risks, we believe it is essential to disclose this research in its entirety. The agent frameworks, beyond those used in this study, are publicly accessible and relatively easy to use. Comparable results will inevitably be achievable by any determined team seeking to utilize language models to produce harmful content and interactions.<br><br>In releasing BrowserART and our main results, we carefully weighed the benefits of empowering research in defense robustness with the risks of enabling further malicious use. Following Zou et al. (2024), we believe the publication of this work helps the agent safety community to release this frontier challenge.&nbsp;<br><br>Prior to release, we have also disclosed our findings and datasets to the companies providing the API access to the models, together with the creators of browser agent frameworks. Our findings highlight the crucial alignment gap between chatbots and browser agents and call upon the research community to explore safeguarding techniques for LLM agents.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<p>If you are using the behavior set of /BrowserART, in addition to this work, please consider to cite HarmBench and AirBench 2024 using the following citations:</p>
<pre class="language-python"><code>@InProceedings{mazeika2024harmbench,
  title = 	 {{H}arm{B}ench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal},
  author =       {Mazeika, Mantas and Phan, Long and Yin, Xuwang and Zou, Andy and Wang, Zifan and Mu, Norman and Sakhaee, Elham and Li, Nathaniel and Basart, Steven and Li, Bo and Forsyth, David and Hendrycks, Dan},
  booktitle = 	 {Proceedings of the 41st International Conference on Machine Learning},
  year = 	 {2024},
  series = 	 {Proceedings of Machine Learning Research},
  publisher =    {PMLR},
}

@article{zeng2024air,
  title={AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies},
  author={Zeng, Yi and Yang, Yu and Zhou, Andy and Tan, Jeffrey Ziwei and Tu, Yuheng and Mai, Yifan and Klyman, Kevin and Pan, Minzhou and Jia, Ruoxi and Song, Dawn and others},
  journal={arXiv preprint arXiv:2407.17436},
  year={2024}
}

@misc{kumar2024refusaltrainedllmseasilyjailbroken,
      title={Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents}, 
      author={Priyanshu Kumar and Elaine Lau and Saranya Vijayakumar and Tu Trinh and Scale Red Team and Elaine Chang and Vaughn Robinson and Sean Hendryx and Shuyan Zhou and Matt Fredrikson and Summer Yue and Zifan Wang},
      year={2024},
      eprint={2410.13886},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2410.13886}, 
}</code></pre>

Browser Art.png

For safety reasons, large language models (LLMs) are trained to refuse harmful user instructions, such as assisting dangerous activities

Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents 

Open Graph Browser Art.png

Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents

Browser Art

customers

slugs must be all lowercase. ie: ‘my-slug’

[Depecrated] If creating a new slide, use the `file` and `type` to specify the content. 

Products

Enterprise

Government

Resources

Customers

Leaderboards →

Scale AI Research

Scale AI’s mission is to accelerate the development of AI applications. By advancing research, we aim to create AI systems capable of solving complex, human-level problems.

ENIGMAEVAL: A Benchmark of Long Multimodal Reasoning Challenges

J2: Jailbreaking to Jailbreak

MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs

Humanity's Last Exam

ToolComp: A Multi-Tool Reasoning & Process Supervision Benchmark

Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs

Revisiting the Superficial Alignment Hypothesis

Planning In Natural Language Improves LLM Search For Code Generation

LLM Leaderboards

Expert-Led Private Evaluations for precise and reliable LLM rankings

Join the Scale AI Research Team

News

The future of your industry starts here.