experiment

HeroTitleAsset

Resources

Header

Research

Scale AI’s mission is to accelerate the development of AI applications.

Scale AI Research

Footer

PreFooter

SectionTitle

BoxAsset

NextImage

AnnouncementContextProvider

Screen

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Scale Meta.png

Partner

Meta and Scale Partner to Drive Enterprise Adoption of Llama

Scale Anthropic.png

Anthropic Partners with Scale to Bring Generative AI to Enterprises

Scale Open AI.png

OpenAI Names Scale as Preferred Fine-Tuning Partner

Scale Aws.png

Blog

AWS + Scale Partner to Bring Generative AI to Enterprises

Meta and Scale Partner to Drive Enterprise Adoption of Llama 3.1 405B

Test and Evaluation New.png

Introducing WMDP: Measuring and Mitigating Catastrophic Risk Potential from LLMs

Seal Leadeaboard.png

Scale's SEAL Research Lab Launches Expert-Evaluated LLM Leaderboards

1200x800.jpg

Bloomberg

Scale Valued at Nearly $14 Billion With Funding From Amazon, AMD

alexandrwang-e1716239360815.jpg

Fortune

Exclusive: Scale AI secures $1B funding at $14B valuation as its CEO predicts big revenue growth and profitability by year-end

43cce6ab-f909-4044-994a-d1b102560e68.jpg

Financial Times

AI data labelling start-up Scale valuation doubles to $14bn 

Fine-Tuning OpenAI’s GPT 3.5 to Unlock Enterprise Use Cases.png

Webinar

Fine-Tuning OpenAI’s GPT-3.5 to Unlock Enterprise Use Cases

We Fine-Tuned GPT-4 to Beat the Industry Standard for Text2SQL.png

We Fine-Tuned GPT-4 to Beat the Industry Standard for Text2SQL

Enhancing LLMs with Retrieval Augmented Generation.png

Enhancing LLMs with Retrieval Augmented Generation

Accelerate Generative AI.png

Feature Release Announcement Blog

Scale has been selected by the U.S. Army and the Defense Innovation Unit to deploy Data Engine in support of the Army’s Robotic Combat Vehicle program

covier_cdao.png

Scale AI chosen by the U.S. Army for Robotic Combat Vehicle Program

mmm.png

Blueprint to AI

Guides Finance.png

Guides

Guide to AI in Finance

Finance.png

Generative AI for the Enterprise

Scale Collaborates with NVIDIA to Power the Next Generation of LLMs with NeMo SteerLM

covier_nvidia.png

Case Studies

NVIDIA

1628598606707.jpeg

Forbes Cloud 100

A Funding Frenzy Helped Catapult 34 New Companies Onto the List and Boost Their Combined

MNMGG23H3ZAIJPEEHVQIP5HHU4.jpeg

Data-labeling company Scale AI valued at $7.3 billion with new funding

download.jpeg

Wall Street Journal

Scale AI’s Rapid Growth Reflects Widening Demand for Smart Software

In this blog, we walk through how you can create your own production-ready LLM App in less than ten minutes using Scale Spellbook.

assets_e0438815ba51486bbb6a202747122d4b_2d7fa2c7e11643769cc174ff90dc520f.png

Deploying LLM Apps

The 2nd edition of our AI Readiness Report helps you get past the hype and understand what it really takes to adopt Generative AI.

Zeitgeist New.png

Zeitgeist

AI Readiness Report

This one-pager covers how Scale can help you whether you are building your own models or applying foundation models to your business.

assets_e0438815ba51486bbb6a202747122d4b_e265f19f1fda4d368d01992c1a774f1f.png

One-Pager

Scale Overview

ML models take data as inputs and deliver a classification, a prediction, or some other indicator as an output. Read our guide to learn more about how to train models.

Test and Evaluation B.png

Guide

Guide: Training & Building Models

Computer Vision focuses on developing systems that can process, analyze, and make sense of visual data. Read our guide to learn about how it works and top use cases by industry.

Guide_ Computer Vision.png

Guide: Computer Vision

The researchers at Harvard Medical School's Datta Lab leverage Scale's Data Engine to speed up their annotation process and advance their research.

cover_hardvard.png

Harvard Medical School

Cohere enhances its Command model with Scale Data Engine.

cover_cohere.png

Customer Case Study: Cohere

Scale worked with OpenAI to build InstructGPT using the Scale Data Engine.

Chat Gpt.png

OpenAI's InstructGPT

Scale built PRM800K to improve mathematical reasoning with process supervision.

OpenAI Process Supervision

Scale helps the DoD bring their autonomy efforts onto the battlefield with a high-quality data layer.

Department of Defense B.png

Autonomy Table Stakes: DoD’s Data Layer

Scale Unlocks Open-Source LLMs With New Platform and Partnership with Meta

Llama2.png

Meta Llama 2 Launch Partner

The Right Approach for Safe, Secure, and Trustworthy AI

Test and Evaluation - Training.png

Test and Evaluation Blog

Building Trust in AI: Our vision for the testing and evaluation of AI

test.png

Test and Evaluation Vision

Scale AI Awarded DoD Contract to Deliver Data Curation and Annotation Capabilities to Power AI Across the Joint Force

DoD Contract Across the Joint Force

Scale AI Framework for Successful Government AI Adoption

Capitol.png

Framework for Successful Government AI Adoption

Scale is proud to be OpenAI’s preferred partner for GPT-3.5 Fine-Tuning

cover_openai.png

OpenAI GPT-3.5 Fine-Tuning Partnership

Our Commitments to Safe, Secure, and Trustworthy AI

White House Ops.png

White House Voluntary Commitments 

Government Leaders discuss key opportunities and challenges for applying AI within the U.S. Government.

youthumb_3HaWHZxN-QE_maxres.png

Video

Scale Gov Summit

Nuro began its collaboration with Scale for 2D and 3D data labeling, progressing to HD map labeling, and today extending to dataset management and curation.

cover_nuro.png

Customer Case Study

Nuro

The Authoritative Guide

Webinar Upgrading Your Fleet inoto an AV Data Engine.png

Data Engine

Upgrading Your Fleet Into an AV Data Engine

OpenAI applied reinforcement learning with human feedback (RLHF) to enhance ChatGPT. Understand the role RLHF plays in enhancing large language models and how to implement it.

Why Is ChatGPT So Good?

The success of ML models is dependent on data and label quality. Read our authoritative guide to ensure you get the highest quality labels.

Guide Data Anotation.png

Guide to Data Annotation

Debagreement_ Reddit 50K.png

Dataset

Open Dataset: Agreement/Disagreement Dataset of Reddit Interactions

Video Trends.png

Discovering Emerging Short Form Video Trends with Clustering and HiTL

How Much Better Is OpenAI's Newest GPT-3 Model?

How OpenAI worked with Scale to build InstructGPT

The Toyota Research Institute relies on Scale's Data Engine to annotate large volumes of training data without sacrificing quality to build a new approach to mobility.

cover_toyota.png

Toyota Research Institute

cover_goodcall.png

Goodcall

Guide to AI for eCommerce.png

Guide to AI for eCommerce

Urkanie Detection.png

Open Dataset: Ukraine Damage Identification

Meet Claude_ Anthropic’s Rival to ChatGPT.png

Meet Claude: Anthropic’s rival to ChatGPT

Guide Diffusion Model.png

Diffusion Models: A Practical Guide

<p>The Superficial Alignment Hypothesis suggests that most of a language model's knowledge and skills come from its initial training, while post-training is about giving a model the right style and format. This study revisits this hypothesis by empirically studying the scaling behavior of post-training with&nbsp;increasing finetuning examples and evaluating them using objective task-specific standardized benchmarks for skills like math, coding, and reasoning.</p>
<p>The findings show that, similar to pre-training scaling laws, post-training task performance scales as a power law against the number of finetuning examples. For complex tasks like math and reasoning,a handful of post-training examples merely align the model stylistically but do not saturate performance on the benchmarks; more extensive examples lead to meaningful improvements in domains like multi-hop reasoning. This indicates that models aren&rsquo;t strictly limited by their initial training knowledge. With the right post-training, models can learn and apply new information to better tackle complex questions.&nbsp;</p>
<div class="c-pillow_file__description"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=2b88f5e0c1122df86e8052689d8e5522.png&amp;f=webp&amp;q=75" alt="" width="2293" height="717"></div>
<p><em>Performance improvements as finetuning data is scaled up, for models in the sub-10 Billion parameter range. The points are fitted with a power law curve of the form P &prop; D<sup>1/b</sup> . Model performance consistently scales in a power law fashion, across model families.</em></p>
<p>These findings suggest that evaluating models using a single preference vote can be misleading for&nbsp; reasoning tasks, especially those with an objective component. A holistic evaluation program covering both human preference alignment and rigorous reasoning benchmarks is required to&nbsp; valuate general purpose frontier models. This suggests the Superficial Alignment Hypothesis may oversimplify how language models learn and develop skills.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-markup"><code>@misc{raghavendra2024revisitingsuperficialalignmenthypothesis,
      title={Revisiting the Superficial Alignment Hypothesis}, 
      author={Mohit Raghavendra and Vaskar Nath and Sean Hendryx},
      year={2024},
      eprint={2410.03717},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.03717}, 
}</code></pre>
<p dir="ltr">&nbsp;</p>
<p>&nbsp;</p>

image (5).png

Shedding new light on the Superficial Alignment Hypothesis, and understanding of model's abilities and knowledge.

Revisiting the Superficial Alignment Hypothesis

Superficial Alignment 

<p>Techniques that learn improved representations via offline data or self-supervised objectives have shown impressive results in traditional reinforcement learning (RL). Nevertheless, it is unclear how improved representation learning can benefit reinforcement learning from human feedback (RLHF) on language models (LMs). In this work, we propose training reward models (RMs) in a contrastive, <em>goal-conditioned</em> fashion by increasing the representation similarity of future states along sampled preferred trajectories and decreasing the similarity along randomly sampled dispreferred trajectories. This objective significantly improves reward model performance by up to 0.09 AUROC across challenging benchmarks, such as MATH and GSM8k. These findings extend to general alignment as well &ndash; on the Helpful-Harmless dataset, we observe 2.3% increase in accuracy. Beyond improving reward model performance, we show this way of training RM representations enables improved <em>steerability</em> because it allows us to evaluate the likelihood of an action achieving a particular goal-state (<em>e.g.,</em> whether a solution is correct or helpful). Leveraging this insight, we find that we can filter up to 55% of generated tokens during majority voting by discarding trajectories likely to end up in an &ldquo;incorrect&rdquo; state, which leads to significant cost savings. We additionally find that these representations can perform fine-grained control by conditioning on desired future goal-states. For example, we show that steering a Llama 3 model towards helpful generations with our approach improves helpfulness by 9.6% over a supervised-fine-tuning trained baseline. Similarly, steering the model towards complex generations improves complexity by 21.6% over the baseline. Overall, we find that training RMs in this contrastive, goal-conditioned fashion significantly improves performance and enables model steerability.&nbsp;<sup>1</sup></p>
<h3>Introduction</h3>
<p>Aligning Language Models (LMs) with human preferences has proven to be an essential step for the adoption and safe use of these models, with the dominant paradigm being Reinforcement Learning from Human Feedback (RLHF) [50]. To accomplish this, a standard setup is to collect labels from humans for generated responses (<em>e.g.,</em> preferences, quality ratings) [50]. These labels can then be used to train a reward model to produce a ranking/scoring of a given sequence or set of sequences.</p>
<p><sup><br><img src="https://img.plasmic.app/img-optimizer/v1/img?src=dbb9caf9f36378276a549c36335c998c.png&amp;f=webp&amp;q=75"><br></sup></p>
<p><em>Figure 1: <strong>Overview of contrastive goal-conditioned learning for text</strong>. Pictured is a prompt with a preferred and dispreferred response. Both source state tokens (ten) for the positive and negative trajectory are sampled from the preferred response. For illustrative purposes, the positve and negative source states are sampled as the same token, but in practice they can be different. The positive goal state is sampled as some future token (subtract) from the preferred response, and the negative goal state is sampled from any token (add) from the dispreferred response. The corresponding representations are retrieved from the last hidden state of the reward model. The training objective is then to maximize and minimize the similarity of the positive and negative representation pairs, respectively.</em></p>
<p>The policy LM is then trained to maximize the expected returns from this reward model using a Reinforcement Learning (RL) algorithm.&nbsp;</p>
<p>High-quality representations have been shown to be an important piece for the success of RL algorithms [6, 42]. Although such representations can be learned during end-to-end training, many efforts have found it important to integrate more explicit representation learning components into RL algorithms, such as via data augmentation [39] or auxiliary losses [21]. Some work even casts certain RL algorithms as representation learning methods where using the similarity between state representations serves as a value function, demonstrating success on manipulation and navigation tasks [20]. Despite these successes in different areas, representation learning for aligning LMs has been less explored, while more emphasis has been placed on, e.g., pre-training reward models [7, 37] or learning from different types of rewards [15, 38].</p>
<p>In this paper, we present a simple yet effective approach to improve the representations learned by reward models for aligning LMs. We train LM-based reward models to learn representations that capture the expected reward or likelihood of achieving a goal state (e.g., correct solution to a problem, helpful response) at intermediate steps of the input sequence, inspired by goal-conditioned RL [5, 13, 20]. To do so, we use a contrastive objective and apply it to the reward model&rsquo;s hidden representations from desirable and undesirable sequences. Enforcing this loss on representations from intermediate steps of the sequence helps encode a dense signal as to which trajectories are more promising at different points in the sequence, which we show offers several useful properties, such as helping to localize errors or evaluating partial completed sequences. This method is flexible enough to support different kinds of alignment data and does not require further annotations beyond common sequence-level annotations.</p>
<p>We find that this approach improves the reward model&rsquo;s ability to identify correct/incorrect solutions in mathematical reasoning, boosting the AUROC on the task by up to 0.09 over standard preference ranking training. Towards natural language alignment, we find this method is able to increase the reward model&rsquo;s accuracy of identifying helpful versus harmless responses by 2.3% on the HelpfulHarmless dataset [8].</p>
<p>We also show the utility of the learned representations themselves, e.g., for filtering solutions to improve accuracy and steering the outputs towards responses with certain attributes in guided decoding (e.g., helpfulness, coherence, and complexity). For mathematical reasoning, we show that 2 we are able to filter up to 55% of generated tokens by discarding trajectories that are likely to lead to incorrect solutions as deemed by the learned representations while achieving similar or better performance. Similarly, using these representations to steer a Llama 3 model by conditioning on desired future goal-states, we improve helpfulness by 9.6%, correctness by 12.2%, coherence by 16.5%, and complexity by 21.6% over the supervised-fine-tuning trained baseline.</p>
<p>In summary, our contributions are as follows: 1) We explore improving the learned representations of reward models and its effect on LM alignment. Towards this, we present a simple and effective representation learning method based on a goal-conditioned contrastive objective. 2) We demonstrate that training reward models with this method can improve reward model performance on mathematical reasoning and helpfulness/harmlessness benchmarks. 3) We show that simply utilizing a reward model trained with this method can improve policy LM alignment on math reasoning benchmarks. 4) We investigate using these representations as a mechanism to evaluate the likelihood of achieving a desired goal state by filtering generations in a majority-vote scheme and guided decoding, showing that they can be used to increase accuracy and control policy LM outputs.<br><br></p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{nath2024learninggoalconditionedrepresentationslanguage,
      title={Learning Goal-Conditioned Representations for Language Reward Models}, 
      author={Vaskar Nath and Dylan Slack and Jeff Da and Yuntao Ma and Hugh Zhang and Spencer Whitehead and Sean Hendryx},
      year={2024},
      eprint={2407.13887},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.13887}, 
}</code></pre>

image (22).png

Learning Goal-Conditioned Representations for Language Reward Models

Og Image.jpg

Learning Goal 

<p dir="ltr">This study identifies a key limitation in scaling inference compute for LLMs: the lack of diversity in generated outputs, which leads to inefficient problem-solving. To address this, the authors propose PLANSEARCH, a novel search algorithm that generates diverse observations in natural language and constructs plans for solving problems based on these observations. PLANSEARCH significantly improves search diversity compared to baseline methods, achieving state-of-the-art results across coding benchmarks, including a 77% pass@200 on LiveCodeBench. The study also shows that performance gains from search algorithms can be predicted by measuring diversity in generated ideas.</p>
<p dir="ltr"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=51a27c0fcfc7865ab25ae6f338540566.png&amp;f=webp&amp;q=75" alt="" width="715" height="541"></p>
<p dir="ltr">The results show that both PLANSEARCH and IDEASEARCH significantly outperform basic sampling across all models and benchmarks, with PLANSEARCH achieving the highest overall scores. Interestingly, IDEASEARCH performs slightly better due to splitting the solution sketch into two model responses. Pass@k results vary across models, with the diversity of generated ideas playing a key role in these differences. The diversity score correlates strongly with performance improvements from scaling inference compute, making it a useful predictor for pass@k gains. However, PLANSEARCH can slightly reduce pass@1 by increasing idea diversity, but this trade-off results in better pass@k performance by improving the chance of generating at least one correct solution.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{wang2024planningnaturallanguageimproves,
      title={Planning In Natural Language Improves LLM Search For Code Generation}, 
      author={Evan Wang and Federico Cassano and Catherine Wu and Yunfeng Bai and Will Song and Vaskar Nath and Ziwen Han and Sean Hendryx and Summer Yue and Hugh Zhang},
      year={2024},
      eprint={2409.03733},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2409.03733}, 
}
</code></pre>

Network.png

Planning In Natural Language Improves LLM Search For Code Generation

Open Graph - Netwokr.png

Planning Natural Language

<p dir="ltr">This study addresses concerns about dataset contamination in LLMs by introducing the Grade School Math 1000 (GSM1k) benchmark, designed to match the style and complexity of the widely-used GSM8k benchmark for mathematical reasoning. The research shows that many models, including Phi and Mistral, experience accuracy drops of up to 13% on GSM1k, indicating systematic overfitting, while frontier models like GPT and Claude show minimal overfitting. A positive correlation (Spearman&rsquo;s r&sup2; = 0.32) was found between a model's likelihood of generating examples from GSM8k and the performance gap between GSM8k and GSM1k, suggesting partial memorization of GSM8k by some models.</p>
<p dir="ltr"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=ed227464f2357d33d4b5b7c62d487b6c.png&amp;f=webp&amp;q=75" width="1233" height="994"></p>
<p style="text-align: left;">The study evaluated LLMs on both the GSM8k and GSM1k benchmarks using a standardized evaluation setup for fair comparison. Open-source models were tested using EleutherAI's LM Evaluation Harness, and closed-source models were queried through the LiteLLM library. All models were tested with identical prompts using five examples from the GSM8k train set. Results indicate that some lesser-known models, particularly those near the top of the OpenLLMLeaderboard, performed significantly worse on GSM1k, suggesting they may have over-optimized for GSM8k, in line with Goodhart&rsquo;s law. The full evaluation code will be released for reproducibility.</p>
<h3 dir="ltr" style="text-align: center;"><strong>Analysis</strong></h3>
<ol>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Certain model families, notably Phi and Mistral, consistently overfit to GSM8k, showing better performance on it compared to GSM1k across almost all model versions and scales. Other model families, including Yi, Xwin, Gemma, and CodeLlama, also exhibit this pattern but to a lesser degree.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Many models do not display signs of overfitting and perform similarly on both GSM8k and GSM1k. Two possible reasons for this are: 1) these models possess advanced reasoning abilities, allowing them to generalize to new problems even if they have seen GSM8k data, and 2) frontier model builders may be more cautious about data contamination.&nbsp;</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Overfit Models Are Still Capable of Reasoning. The fact that a model is overfit does not mean that it is poor at reasoning, merely that it is not as good as the benchmarks might indicate it to be. In fact, we find that many of the most overfit models are still capable of reasoning and solving novel problems.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Data Contamination Is Likely Not The Full Explanation for Overfitting. This result suggests that some of the reason for overfitting is due to partial memorization of the test set. Overfitting on GSM8k may be through other indirect means, such as model builders collecting data similar in nature to benchmarks as training data or selecting final model checkpoints based on performance on benchmarks, even if the model itself may have not seen the GSM8k dataset at any point via training. Conversely, the reverse is also true: small amounts of data contamination do not necessarily lead to overfitting.</p>
</li>
</ol>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-markup"><code>@inproceedings{zhang2024carefulexaminationlargelanguage, title={A Careful Examination of Large Language Model Performance on Grade School Arithmetic}, author={Hugh Zhang and Jeff Da and Dean Lee and Vaughn Robinson and Catherine Wu and Will Song and Tiffany Zhao and Pranav Raja and Charlotte Zhuang and Dylan Slack and Qin Lyu and Sean Hendryx and Russell Kaplan and Michele Lunati and Summer Yue}, booktitle={Advances in Neural Information Processing Systems}, year={2024}</code></pre>

image 6.png

LLM performance on mathematical reasoning benchmarks may involve memorization rather than true reasoning ability.



A Careful Examination of Large Language Model Performance on Grade School Arithmetic

Open Graph.png

LL Performance School

<p>Recent large language model (LLM) defenses have greatly improved models' ability to refuse harmful queries, even when adversarially attacked. However, LLM defenses are primarily evaluated against automated adversarial attacks in a single turn of conversation, an insufficient threat model for real-world malicious use. We demonstrate that multi-turn human jailbreaks uncover significant vulnerabilities, exceeding 70% attack success rate (ASR) on HarmBench against defenses that report single-digit ASRs with automated single-turn attacks. Human jailbreaks also reveal vulnerabilities in machine unlearning defenses, successfully recovering dual-use biosecurity knowledge from unlearned models. We compile these results into Multi-Turn Human Jailbreaks (MHJ), a dataset of 2,912 prompts across 537 multi-turn jailbreaks. We publicly release MHJ alongside a compendium of jailbreak tactics developed across dozens of commercial red teaming engagements, supporting research towards stronger LLM defenses.</p>
<p>Scale publishes the results of expert-driven private evaluations on <a title="SEAL Leaderboards" href="https://scale.com/leaderboard">SEAL Leaderboards</a>.</p>
<h3 style="text-align: center;">Main Results</h3>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=e11ecad7941e445e6c585502a7a7dde3.png&amp;f=webp&amp;q=75"></p>
<p>We compare our Multi-turn Human Jailbreaks (MHJ) results on HarmBench (consisting of 240 harmful behaviors) for its attack success rate (ASR) with the state-of-the-art automated attacks, targeting jailbreaking the state-of-the-art defense methods for improving LLM robustness. In the following plot, Ensemble Automated Attack is constructed such that it succeeds if any of the 6 automated attacks succeeds (Pass@6). Best Automated Attack shows the highest ASR for each defense from a single automated attack. For the detailed results and discussions, please see the full paper.</p>
<h3 style="text-align: center;">Ethics and Disclosure</h3>
<p>In releasing the Multi-Turn Human Jailbreaks (MHJ) dataset, we carefully weighed the benefits of empowering research in defense robustness with the risks of enabling further malicious use. Following Zou et al. (2024), we believe the publication of MHJ poses low marginal risk, as datasets of many other manual jailbreaks are widely disseminated. Towards reducing risk, we removed a complete set of model completions and any jailbreaks that may contain sensitive information in the main paper and in the site. With the support of legal counsel, we verified MHJ&rsquo;s compliance with applicable U.S. export control requirements, including with respect to the International Traffic in Arms Regulations (22 CFR Parts 120-130) and Export Administration Regulations (15 CFR Parts 730-774).</p>
<p>We received explicit permission for red teaming any proprietary models. Prior to release, we also disclosed our findings and datasets to the authors of the defenses we examined.</p>
<h3 style="text-align: center;">Accessing the full set of MHJ</h3>
<p>After filtering sensitive &amp; export-controlled information, we publicly release the adversarial prompts in MHJ, supporting research towards stronger LLM defenses and robustness evaluations at Huggingface for research purpose only (under Creative Commons Attribution Non Commercial 4.0). Please apply for the access on <a href="https://huggingface.co/datasets/ScaleAI/mhj">Huggingface</a>.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-markup"><code>@misc{li2024llmdefensesrobustmultiturn,
      title={LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet}, 
      author={Nathaniel Li and Ziwen Han and Ian Steneker and Willow Primack and Riley Goodside and Hugh Zhang and Zifan Wang and Cristina Menghini and Summer Yue},
      year={2024},
      eprint={2408.15221},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2408.15221}, 
}</code></pre>

Jailbreak.png

Multi-turn human jailbreaks expose vulnerabilities in LLM defenses, achieving over 70% attack success.

LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet

Open Graph Jailbreak.png

<p>McConnell led opposition to stricter campaign finance regulations, with the Lexington Herald Leader describing it as McConnell's "pet issue."[8] He has argued that campaign finance regulations reduce participation in political campaigns and protect incumbents from competition.[9] In 1990, he led a filibuster against a campaign finance reform bill.[10] He spearheaded the movement against the Bipartisan Campaign Reform Act. His opposition to the bill culminated in the 2003 Supreme Court case McConnell v. Federal Election Commission and the 2010 Citizens United v. Federal Election Commission, the latter of which ruled part of the act unconstitutional. McConnell has been an advocate for free speech at least as far back as the early 1970s when he was teaching night courses at the University of Louisville. "No issue has shaped his career more than the intersection of campaign financing and free speech," political reporter Robert Costa wrote in 2012.[11] In a recording of a 2014 fundraiser McConnell expressed his disapproval of the McCain-Feingold law, saying,"The worst day of my political life was when President George W. Bush signed McCain-Feingold into law in the early part of his first Administration."[12] On January 2, 2013, the Public Campaign Action Fund, a liberal nonprofit group that backs stronger campaign finance regulation, released a report highlighting eight instances from McConnell's political career in which a vote or a blocked vote (filibuster), coincided with an influx of campaign contributions to McConnell's campaign.[13][14] In December 2018, McConnell opposed a Democratic resolution that overturned IRS guidance to reduce the amount of donor information certain tax-exempt groups would have to give the IRS, saying,"In a climate that is increasingly hostile to certain kinds of political expression and open debate, the last thing Washington needs to do is to chill the exercise of free speech and add to the sense of intimidation."[15] Prior to his Senate run, McConnell favored campaign finance reform, describing money in politics as a "cancer" in 1973.[10]</p>

<p>Cruz received more than US$1 million in campaign donations from the oil and gas industry since 2011.[104] Cruz is an opponent of the EPA, calling it "unbelievably abusive" and "populated by zealots,"[105] and has accused the Obama administration of waging a "war on coal."[105][106] Cruz opposes EPA's Clean Power Plan, which aims to cut carbon emissions from power plants; Cruz accused President Obama of engaging in a "lawless and radical attempt to destabilize the nation's energy system" and called the plan "flatly unconstitutional."[107] Cruz, who says that he is an advocate for "volunteer conservation," and also disagreed with efforts by the EPA to expand regulatory oversight on water, which Cruz characterized an attempt "to turn irrigation ditches into lakes and rivers and oceans."[108] Cruz is a supporter of TransCanada's controversial Keystone XL Pipeline,[106] and along with every other Republican senator was a cosponsor of legislation in support of the pipeline.[109] Like most Republican senators, Cruz supports opening areas for offshore drilling.[106] Cruz favors "expanding energy development on federal land" and has "pushed for preventing federal restrictions on fracking."[106] In July 2015, Cruz's super PAC,"Keep the Promise," disclosed that the billionaire brothers Dan and Farris Wilks, who earned a fortune in the West Texas fracking boom, contributed a record-setting $15 million to the super PAC.[110]</p>

<p>For safety reasons, large language models (LLMs) are trained to refuse harmful user instructions, such as assisting dangerous activities. We study an open question in this work. Can the desired safety refusal, typically enforced in chat contexts, be generalized to non-chat and agentic use cases? Unlike chatbots, LLM agents equipped with general-purpose tools, such as web browsers and mobile devices, can directly influence the real world, making it even more crucial to refuse harmful instructions. In this work, we primarily focus on red-teaming browser agents &ndash; LLMs that manipulate information via web browsers.&nbsp;<br><br>To this end, we introduce Browser Agent Red-teaming Toolkit (BrowserART), a comprehensive test suite designed specifically for red-teaming browser agents. BrowserART consists of 100 diverse browser-related harmful behaviors (including original behaviors and ones sourced from HarmBench [Mazeika et al., 2024] and AirBench 2024 [Zeng et al., 2024b]) across both synthetic and real websites.&nbsp;<br><br>Our empirical study on state-of-the-art browser agents reveals that, while the backbone LLM refuses harmful instructions as a chatbot, the corresponding agent does not. Moreover, attack methods designed to jailbreak refusal-trained LLMs in the chat settings transfer effectively to browser agents. With human rewrites, GPT-4o and o1-preview-based browser agents attempted 98 and 63 harmful behaviors (out of 100), respectively. We publicly release BrowserART and call on LLM developers, policymakers, and agent developers to collaborate on improving agent safety.</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=1b13997d80e8646969c95cee06965802.png&amp;f=webp&amp;q=75"></p>
<p>BrowserART consists of 100 harmful browser-related behaviors (including original behaviors and ones sourced from HarmBench [Mazeika et al., 2024] and AirBench 2024 [Zeng et al., 2024b]) that an agent is not supposed to assist. We divided all behaviors into two main categories: harmful content and harmful interaction. Under each main category, we created sub-categories for the harm semantics. We created 40 synthetic websites under 19 domains for red teaming browser behaviors that target specific websites (e.g., Twitter/X). These synthetic pages are hosted locally for running red teaming experiments in a sandbox without polluting the real world, especially social media and government sites.</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=6b9db5c096e4a32216e92f1b221d3af1.png&amp;f=webp&amp;q=75"></p>
<p>One of the major findings in our work is a clear gap between Attack Success Rates (ASRs) of the backbone LLM (blue) and its agent (lavender). That is, while the LLM refuses to follow the harmful behavior in the user instruction, the agent will. Gaps of GPT-4o and GPT-4-turbo models are most outstanding, compared to other backbone LLMs. In particular, we find Opus-3 and Llama-3.1-405B show the least drop in their refusal capabilities.</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=a10a6df8f013bd702541020a090824f4.png&amp;f=webp&amp;q=75"></p>
<h3 style="text-align: center;">Ethics and Disclosure</h3>
<p>This research &mdash; including the methodology detailed in the paper, the code, and the content of this webpage &mdash; contains material that may enable users to generate harmful content using certain publicly available LLM agents. While we recognize the associated risks, we believe it is essential to disclose this research in its entirety. The agent frameworks, beyond those used in this study, are publicly accessible and relatively easy to use. Comparable results will inevitably be achievable by any determined team seeking to utilize language models to produce harmful content and interactions.<br><br>In releasing BrowserART and our main results, we carefully weighed the benefits of empowering research in defense robustness with the risks of enabling further malicious use. Following Zou et al. (2024), we believe the publication of this work helps the agent safety community to release this frontier challenge.&nbsp;<br><br>Prior to release, we have also disclosed our findings and datasets to the companies providing the API access to the models, together with the creators of browser agent frameworks. Our findings highlight the crucial alignment gap between chatbots and browser agents and call upon the research community to explore safeguarding techniques for LLM agents.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<p>If you are using the behavior set of /BrowserART, in addition to this work, please consider to cite HarmBench and AirBench 2024 using the following citations:</p>
<pre class="language-python"><code>@InProceedings{mazeika2024harmbench,
  title = 	 {{H}arm{B}ench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal},
  author =       {Mazeika, Mantas and Phan, Long and Yin, Xuwang and Zou, Andy and Wang, Zifan and Mu, Norman and Sakhaee, Elham and Li, Nathaniel and Basart, Steven and Li, Bo and Forsyth, David and Hendrycks, Dan},
  booktitle = 	 {Proceedings of the 41st International Conference on Machine Learning},
  year = 	 {2024},
  series = 	 {Proceedings of Machine Learning Research},
  publisher =    {PMLR},
}

@article{zeng2024air,
  title={AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies},
  author={Zeng, Yi and Yang, Yu and Zhou, Andy and Tan, Jeffrey Ziwei and Tu, Yuheng and Mai, Yifan and Klyman, Kevin and Pan, Minzhou and Jia, Ruoxi and Song, Dawn and others},
  journal={arXiv preprint arXiv:2407.17436},
  year={2024}
}

@misc{kumar2024refusaltrainedllmseasilyjailbroken,
      title={Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents}, 
      author={Priyanshu Kumar and Elaine Lau and Saranya Vijayakumar and Tu Trinh and Scale Red Team and Elaine Chang and Vaughn Robinson and Sean Hendryx and Shuyan Zhou and Matt Fredrikson and Summer Yue and Zifan Wang},
      year={2024},
      eprint={2410.13886},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2410.13886}, 
}</code></pre>

For safety reasons, large language models (LLMs) are trained to refuse harmful user instructions, such as assisting dangerous activities

Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents 

Browser Art

Products

Enterprise

Government