experiment

HeroTitleAsset

AnnouncementsSlider

Header

Research

Scale AI’s mission is to accelerate the development of AI applications.

Scale AI Research

CTAs

Footer

PreFooter

SectionTitle

BoxAsset

LoopingVideo

Positions

NextImage

AnnouncementContextProvider

Screen

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

The New ChatGPT-4o Update Promises Better Writing, How Does It Compare To The New Claude 3.5 Sonnet?

Scale's SEAL Research Lab Launches Expert-Evaluated LLM Leaderboards

Submit Your Toughest Questions for Humanity's Last Exam

First Impressions of OpenAI’s o1

Scale & NVIDIA: NeMo SteerLM for Next-Gen LLMs

Scale announces collaboration with Microsoft Azure

Fortune has named Scale as one of their 50 AI Innovators

leaderboard.png

center.svg

aicpa-soc-logo-freelogovectors.net_ 3.svg

iso-2-1 1.svg

fedramp-logo-vert 2.svg

ntt_logo.svg

cengage_logo.svg.svg

DLA_Piper_Logo 1.svg

Cisco_logo 1.svg

Logos - Valeo.svg

50-AI-logo.svg

bcg.svg

Black.png

Logos - Global Atlantic.svg

Howard_Hughes_logo.png

4DV Analytics - png.png

Logos - Uncountable.svg

Logos - Grata.svg

stability ai.svg

Logos Antrhopic.svg

image.jpg

Logos - Brex.svg

Logos - Nuro.svg

kaleido_ai_black.svg

Luminar Logo White Horizontal Vector 4.svg

<p>As large language models (LLMs) are applied to more use cases, creating high-quality, task-specific datasets for fine-tuning becomes a bottleneck for model improvement. Using high-quality human data has been the most common approach to unlock model performance, but it is prohibitively expensive in many scenarios. Several alternative methods have also emerged, such as generating synthetic or hybrid data, but the effectiveness of these approaches remains unclear, especially in resource-constrained scenarios and tasks that are not easily verified. To investigate this, we group various synthetic data generation strategies into three representative categories &ndash; Answer Augmentation, Question Rephrase, and New Question &ndash; and study the performance of student LLMs trained under various constraints, namely seed instruction set size and query budget. We demonstrate that these strategies are not equally effective across settings. Notably, the optimal data generation strategy depends strongly on the ratio between the available teacher query budget and the size of the seed instruction set. When this ratio is low, generating new answers to existing questions proves most effective, but generating new questions becomes optimal as this ratio increases. Across all tasks, we find that the choice of augmentation method and other design choices matter substantially more in low to mid-data regimes than in high-data regimes. We provide a practical framework for selecting the appropriate augmentation method across settings, taking into account additional factors such as the scalability of each method, the importance of verifying synthetic data, and using different LLMs for synthetic data generation.</p>
<h3>Method</h3>
<p>In this work, we examine techniques in synthetic data that were initially introduced in the math reasoning domains and extend the analysis of these strategies to a more diverse collection of tasks and constraints (Setlur et al., 2024; Yu et al., 2023; Li et al., 2024). We choose supervised finetuning (SFT) as the learning objective for the student model, which requires a dataset consisting of instruction-response pairs. We identify three main types of synthetic data generation strategies pertinent to an SFT instruction set &ndash; Answer Augmentation, Question Rephrasing and New Question &ndash; that are in essence, either the transformations of the seed instructions by an augmenter LLM, generation of corresponding responses by a teacher LLM or both.</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=39260c0163d4676d6d4267b23ffa4f61.png&amp;f=webp&amp;q=75&amp;w=3840" alt="" width="1503" height="935"></p>
<p>In the above figure, we present an overview of our data generation methods. Given a seed instruction set, we have 3 different methods to create instruction-response pairs for fine-tuning our student model. We use an example seed instruction from the ARC-C training set with synthetic instructions and responses generated with Llama 3.1 70b Instruct.</p>
<p>For all experiments, we assess the effectiveness of each synthetic data generation strategy by comparing the accuracy of fine-tuned student model on the evaluation split of the dataset used in the experiment. First, we understand the effectiveness of each data generation strategy through scalability and data constraints. Next, we investigate the cost-effectiveness of creating new instructions and responses. Finally, we cover ablation studies focused on understanding the impact of different teacher models, verification of responses, and a different student model.</p>
<p><strong>This paper&nbsp;<em>Balancing Cost and Effectiveness of Synthetic Data&nbsp;Generation Strategies for LLMs</em> (<a href="https://arxiv.org/pdf/2409.19759">arxiv</a>), has been accepted to the <a href="https://sites.google.com/view/neurips2024-ftw">Fine-Tuning in Machine Learning (FITML) Workshop at NeurIPS 2024</a>.</strong></p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-markup"><code>@article{chan2024balancing,
  title={Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs},
  author={Chan, Yung-Chieh and Pu, George and Shanker, Apaar and Suresh, Parth and Jenks, Penn and Heyer, John and Denton, Sam},
  journal={arXiv preprint arXiv:2409.19759},
  year={2024}
}</code></pre>

Legal-Balance-v2 1.png

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for Fine-Tuning LLMs 

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs

Open Graph.png

Synthetic Data Generation Strategies for LLM Fine-Tuning

<p>The Superficial Alignment Hypothesis suggests that most of a language model's knowledge and skills come from its initial training, while post-training is about giving a model the right style and format. This study revisits this hypothesis by empirically studying the scaling behavior of post-training with&nbsp;increasing finetuning examples and evaluating them using objective task-specific standardized benchmarks for skills like math, coding, and reasoning.</p>
<p>The findings show that, similar to pre-training scaling laws, post-training task performance scales as a power law against the number of finetuning examples. For complex tasks like math and reasoning,a handful of post-training examples merely align the model stylistically but do not saturate performance on the benchmarks; more extensive examples lead to meaningful improvements in domains like multi-hop reasoning. This indicates that models aren&rsquo;t strictly limited by their initial training knowledge. With the right post-training, models can learn and apply new information to better tackle complex questions.&nbsp;</p>
<div class="c-pillow_file__description"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=2b88f5e0c1122df86e8052689d8e5522.png&amp;f=webp&amp;q=75" alt="" width="2293" height="717"></div>
<p><em>Performance improvements as finetuning data is scaled up, for models in the sub-10 Billion parameter range. The points are fitted with a power law curve of the form P &prop; D<sup>1/b</sup> . Model performance consistently scales in a power law fashion, across model families.</em></p>
<p>These findings suggest that evaluating models using a single preference vote can be misleading for&nbsp; reasoning tasks, especially those with an objective component. A holistic evaluation program covering both human preference alignment and rigorous reasoning benchmarks is required to&nbsp; valuate general purpose frontier models. This suggests the Superficial Alignment Hypothesis may oversimplify how language models learn and develop skills.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-markup"><code>@misc{raghavendra2024revisitingsuperficialalignmenthypothesis,
      title={Revisiting the Superficial Alignment Hypothesis}, 
      author={Mohit Raghavendra and Vaskar Nath and Sean Hendryx},
      year={2024},
      eprint={2410.03717},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.03717}, 
}</code></pre>
<p dir="ltr">&nbsp;</p>
<p>&nbsp;</p>

image (5).png

Shedding new light on the Superficial Alignment Hypothesis, and understanding of model's abilities and knowledge.

Revisiting the Superficial Alignment Hypothesis

Superficial Alignment 

<p>Techniques that learn improved representations via offline data or self-supervised objectives have shown impressive results in traditional reinforcement learning (RL). Nevertheless, it is unclear how improved representation learning can benefit reinforcement learning from human feedback (RLHF) on language models (LMs). In this work, we propose training reward models (RMs) in a contrastive, <em>goal-conditioned</em> fashion by increasing the representation similarity of future states along sampled preferred trajectories and decreasing the similarity along randomly sampled dispreferred trajectories. This objective significantly improves reward model performance by up to 0.09 AUROC across challenging benchmarks, such as MATH and GSM8k. These findings extend to general alignment as well &ndash; on the Helpful-Harmless dataset, we observe 2.3% increase in accuracy. Beyond improving reward model performance, we show this way of training RM representations enables improved <em>steerability</em> because it allows us to evaluate the likelihood of an action achieving a particular goal-state (<em>e.g.,</em> whether a solution is correct or helpful). Leveraging this insight, we find that we can filter up to 55% of generated tokens during majority voting by discarding trajectories likely to end up in an &ldquo;incorrect&rdquo; state, which leads to significant cost savings. We additionally find that these representations can perform fine-grained control by conditioning on desired future goal-states. For example, we show that steering a Llama 3 model towards helpful generations with our approach improves helpfulness by 9.6% over a supervised-fine-tuning trained baseline. Similarly, steering the model towards complex generations improves complexity by 21.6% over the baseline. Overall, we find that training RMs in this contrastive, goal-conditioned fashion significantly improves performance and enables model steerability.&nbsp;<sup>1</sup></p>
<h3>Introduction</h3>
<p>Aligning Language Models (LMs) with human preferences has proven to be an essential step for the adoption and safe use of these models, with the dominant paradigm being Reinforcement Learning from Human Feedback (RLHF) [50]. To accomplish this, a standard setup is to collect labels from humans for generated responses (<em>e.g.,</em> preferences, quality ratings) [50]. These labels can then be used to train a reward model to produce a ranking/scoring of a given sequence or set of sequences.</p>
<p><sup><br><img src="https://img.plasmic.app/img-optimizer/v1/img?src=dbb9caf9f36378276a549c36335c998c.png&amp;f=webp&amp;q=75"><br></sup></p>
<p><em>Figure 1: <strong>Overview of contrastive goal-conditioned learning for text</strong>. Pictured is a prompt with a preferred and dispreferred response. Both source state tokens (ten) for the positive and negative trajectory are sampled from the preferred response. For illustrative purposes, the positve and negative source states are sampled as the same token, but in practice they can be different. The positive goal state is sampled as some future token (subtract) from the preferred response, and the negative goal state is sampled from any token (add) from the dispreferred response. The corresponding representations are retrieved from the last hidden state of the reward model. The training objective is then to maximize and minimize the similarity of the positive and negative representation pairs, respectively.</em></p>
<p>The policy LM is then trained to maximize the expected returns from this reward model using a Reinforcement Learning (RL) algorithm.&nbsp;</p>
<p>High-quality representations have been shown to be an important piece for the success of RL algorithms [6, 42]. Although such representations can be learned during end-to-end training, many efforts have found it important to integrate more explicit representation learning components into RL algorithms, such as via data augmentation [39] or auxiliary losses [21]. Some work even casts certain RL algorithms as representation learning methods where using the similarity between state representations serves as a value function, demonstrating success on manipulation and navigation tasks [20]. Despite these successes in different areas, representation learning for aligning LMs has been less explored, while more emphasis has been placed on, e.g., pre-training reward models [7, 37] or learning from different types of rewards [15, 38].</p>
<p>In this paper, we present a simple yet effective approach to improve the representations learned by reward models for aligning LMs. We train LM-based reward models to learn representations that capture the expected reward or likelihood of achieving a goal state (e.g., correct solution to a problem, helpful response) at intermediate steps of the input sequence, inspired by goal-conditioned RL [5, 13, 20]. To do so, we use a contrastive objective and apply it to the reward model&rsquo;s hidden representations from desirable and undesirable sequences. Enforcing this loss on representations from intermediate steps of the sequence helps encode a dense signal as to which trajectories are more promising at different points in the sequence, which we show offers several useful properties, such as helping to localize errors or evaluating partial completed sequences. This method is flexible enough to support different kinds of alignment data and does not require further annotations beyond common sequence-level annotations.</p>
<p>We find that this approach improves the reward model&rsquo;s ability to identify correct/incorrect solutions in mathematical reasoning, boosting the AUROC on the task by up to 0.09 over standard preference ranking training. Towards natural language alignment, we find this method is able to increase the reward model&rsquo;s accuracy of identifying helpful versus harmless responses by 2.3% on the HelpfulHarmless dataset [8].</p>
<p>We also show the utility of the learned representations themselves, e.g., for filtering solutions to improve accuracy and steering the outputs towards responses with certain attributes in guided decoding (e.g., helpfulness, coherence, and complexity). For mathematical reasoning, we show that 2 we are able to filter up to 55% of generated tokens by discarding trajectories that are likely to lead to incorrect solutions as deemed by the learned representations while achieving similar or better performance. Similarly, using these representations to steer a Llama 3 model by conditioning on desired future goal-states, we improve helpfulness by 9.6%, correctness by 12.2%, coherence by 16.5%, and complexity by 21.6% over the supervised-fine-tuning trained baseline.</p>
<p>In summary, our contributions are as follows: 1) We explore improving the learned representations of reward models and its effect on LM alignment. Towards this, we present a simple and effective representation learning method based on a goal-conditioned contrastive objective. 2) We demonstrate that training reward models with this method can improve reward model performance on mathematical reasoning and helpfulness/harmlessness benchmarks. 3) We show that simply utilizing a reward model trained with this method can improve policy LM alignment on math reasoning benchmarks. 4) We investigate using these representations as a mechanism to evaluate the likelihood of achieving a desired goal state by filtering generations in a majority-vote scheme and guided decoding, showing that they can be used to increase accuracy and control policy LM outputs.<br><br></p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{nath2024learninggoalconditionedrepresentationslanguage,
      title={Learning Goal-Conditioned Representations for Language Reward Models}, 
      author={Vaskar Nath and Dylan Slack and Jeff Da and Yuntao Ma and Hugh Zhang and Spencer Whitehead and Sean Hendryx},
      year={2024},
      eprint={2407.13887},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.13887}, 
}</code></pre>

image (22).png

Learning Goal-Conditioned Representations for Language Reward Models

Og Image.jpg

Learning Goal 

<p dir="ltr">This study identifies a key limitation in scaling inference compute for LLMs: the lack of diversity in generated outputs, which leads to inefficient problem-solving. To address this, the authors propose PLANSEARCH, a novel search algorithm that generates diverse observations in natural language and constructs plans for solving problems based on these observations. PLANSEARCH significantly improves search diversity compared to baseline methods, achieving state-of-the-art results across coding benchmarks, including a 77% pass@200 on LiveCodeBench. The study also shows that performance gains from search algorithms can be predicted by measuring diversity in generated ideas.</p>
<p dir="ltr"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=51a27c0fcfc7865ab25ae6f338540566.png&amp;f=webp&amp;q=75" alt="" width="715" height="541"></p>
<p dir="ltr">The results show that both PLANSEARCH and IDEASEARCH significantly outperform basic sampling across all models and benchmarks, with PLANSEARCH achieving the highest overall scores. Interestingly, IDEASEARCH performs slightly better due to splitting the solution sketch into two model responses. Pass@k results vary across models, with the diversity of generated ideas playing a key role in these differences. The diversity score correlates strongly with performance improvements from scaling inference compute, making it a useful predictor for pass@k gains. However, PLANSEARCH can slightly reduce pass@1 by increasing idea diversity, but this trade-off results in better pass@k performance by improving the chance of generating at least one correct solution.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-javascript"><code>@misc{wang2024planningnaturallanguageimproves,
      title={Planning In Natural Language Improves LLM Search For Code Generation}, 
      author={Evan Wang and Federico Cassano and Catherine Wu and Yunfeng Bai and Will Song and Vaskar Nath and Ziwen Han and Sean Hendryx and Summer Yue and Hugh Zhang},
      year={2024},
      eprint={2409.03733},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2409.03733}, 
}
</code></pre>

Network.png

Planning In Natural Language Improves LLM Search For Code Generation

Open Graph - Netwokr.png

Planning Natural Language

<p dir="ltr">This study addresses concerns about dataset contamination in LLMs by introducing the Grade School Math 1000 (GSM1k) benchmark, designed to match the style and complexity of the widely-used GSM8k benchmark for mathematical reasoning. The research shows that many models, including Phi and Mistral, experience accuracy drops of up to 13% on GSM1k, indicating systematic overfitting, while frontier models like GPT and Claude show minimal overfitting. A positive correlation (Spearman&rsquo;s r&sup2; = 0.32) was found between a model's likelihood of generating examples from GSM8k and the performance gap between GSM8k and GSM1k, suggesting partial memorization of GSM8k by some models.</p>
<p dir="ltr"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=ed227464f2357d33d4b5b7c62d487b6c.png&amp;f=webp&amp;q=75" width="1233" height="994"></p>
<p style="text-align: left;">The study evaluated LLMs on both the GSM8k and GSM1k benchmarks using a standardized evaluation setup for fair comparison. Open-source models were tested using EleutherAI's LM Evaluation Harness, and closed-source models were queried through the LiteLLM library. All models were tested with identical prompts using five examples from the GSM8k train set. Results indicate that some lesser-known models, particularly those near the top of the OpenLLMLeaderboard, performed significantly worse on GSM1k, suggesting they may have over-optimized for GSM8k, in line with Goodhart&rsquo;s law.&nbsp;</p>
<h3 dir="ltr" style="text-align: center;"><strong>Analysis</strong></h3>
<ol>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Certain model families, notably Phi and Mistral, consistently overfit to GSM8k, showing better performance on it compared to GSM1k across almost all model versions and scales. Other model families, including Yi, Xwin, Gemma, and CodeLlama, also exhibit this pattern but to a lesser degree.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Many models do not display signs of overfitting and perform similarly on both GSM8k and GSM1k. Two possible reasons for this are: 1) these models possess advanced reasoning abilities, allowing them to generalize to new problems even if they have seen GSM8k data, and 2) frontier model builders may be more cautious about data contamination.&nbsp;</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Overfit Models Are Still Capable of Reasoning. The fact that a model is overfit does not mean that it is poor at reasoning, merely that it is not as good as the benchmarks might indicate it to be. In fact, we find that many of the most overfit models are still capable of reasoning and solving novel problems.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Data Contamination Is Likely Not The Full Explanation for Overfitting. This result suggests that some of the reason for overfitting is due to partial memorization of the test set. Overfitting on GSM8k may be through other indirect means, such as model builders collecting data similar in nature to benchmarks as training data or selecting final model checkpoints based on performance on benchmarks, even if the model itself may have not seen the GSM8k dataset at any point via training. Conversely, the reverse is also true: small amounts of data contamination do not necessarily lead to overfitting.</p>
</li>
</ol>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-markup"><code>@inproceedings{zhang2024carefulexaminationlargelanguage, title={A Careful Examination of Large Language Model Performance on Grade School Arithmetic}, author={Hugh Zhang and Jeff Da and Dean Lee and Vaughn Robinson and Catherine Wu and Will Song and Tiffany Zhao and Pranav Raja and Charlotte Zhuang and Dylan Slack and Qin Lyu and Sean Hendryx and Russell Kaplan and Michele Lunati and Summer Yue}, booktitle={Advances in Neural Information Processing Systems}, year={2024}</code></pre>

image 6.png

LLM performance on mathematical reasoning benchmarks may involve memorization rather than true reasoning ability.



A Careful Examination of Large Language Model Performance on Grade School Arithmetic

LLM Performance School

<p>Recent large language model (LLM) defenses have greatly improved models' ability to refuse harmful queries, even when adversarially attacked. However, LLM defenses are primarily evaluated against automated adversarial attacks in a single turn of conversation, an insufficient threat model for real-world malicious use. We demonstrate that multi-turn human jailbreaks uncover significant vulnerabilities, exceeding 70% attack success rate (ASR) on HarmBench against defenses that report single-digit ASRs with automated single-turn attacks. Human jailbreaks also reveal vulnerabilities in machine unlearning defenses, successfully recovering dual-use biosecurity knowledge from unlearned models. We compile these results into Multi-Turn Human Jailbreaks (MHJ), a dataset of 2,912 prompts across 537 multi-turn jailbreaks. We publicly release MHJ alongside a compendium of jailbreak tactics developed across dozens of commercial red teaming engagements, supporting research towards stronger LLM defenses.</p>
<p>Scale publishes the results of expert-driven private evaluations on <a title="SEAL Leaderboards" href="https://scale.com/leaderboard">SEAL Leaderboards</a>.</p>
<h3 style="text-align: center;">Main Results</h3>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=e11ecad7941e445e6c585502a7a7dde3.png&amp;f=webp&amp;q=75"></p>
<p>We compare our Multi-turn Human Jailbreaks (MHJ) results on HarmBench (consisting of 240 harmful behaviors) for its attack success rate (ASR) with the state-of-the-art automated attacks, targeting jailbreaking the state-of-the-art defense methods for improving LLM robustness. In the following plot, Ensemble Automated Attack is constructed such that it succeeds if any of the 6 automated attacks succeeds (Pass@6). Best Automated Attack shows the highest ASR for each defense from a single automated attack. For the detailed results and discussions, please see the full paper.</p>
<h3 style="text-align: center;">Ethics and Disclosure</h3>
<p>In releasing the Multi-Turn Human Jailbreaks (MHJ) dataset, we carefully weighed the benefits of empowering research in defense robustness with the risks of enabling further malicious use. Following Zou et al. (2024), we believe the publication of MHJ poses low marginal risk, as datasets of many other manual jailbreaks are widely disseminated. Towards reducing risk, we removed a complete set of model completions and any jailbreaks that may contain sensitive information in the main paper and in the site. With the support of legal counsel, we verified MHJ&rsquo;s compliance with applicable U.S. export control requirements, including with respect to the International Traffic in Arms Regulations (22 CFR Parts 120-130) and Export Administration Regulations (15 CFR Parts 730-774).</p>
<p>We received explicit permission for red teaming any proprietary models. Prior to release, we also disclosed our findings and datasets to the authors of the defenses we examined.</p>
<h3 style="text-align: center;">Accessing the full set of MHJ</h3>
<p>After filtering sensitive &amp; export-controlled information, we publicly release the adversarial prompts in MHJ, supporting research towards stronger LLM defenses and robustness evaluations at Huggingface for research purpose only (under Creative Commons Attribution Non Commercial 4.0). Please apply for the access on <a href="https://huggingface.co/datasets/ScaleAI/mhj">Huggingface</a>.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<pre class="language-markup"><code>@misc{li2024llmdefensesrobustmultiturn,
      title={LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet}, 
      author={Nathaniel Li and Ziwen Han and Ian Steneker and Willow Primack and Riley Goodside and Hugh Zhang and Zifan Wang and Cristina Menghini and Summer Yue},
      year={2024},
      eprint={2408.15221},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2408.15221}, 
}</code></pre>

Jailbreak.png

Multi-turn human jailbreaks expose vulnerabilities in LLM defenses, achieving over 70% attack success.

LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet

Open Graph Jailbreak.png

<p>McConnell led opposition to stricter campaign finance regulations, with the Lexington Herald Leader describing it as McConnell's "pet issue."[8] He has argued that campaign finance regulations reduce participation in political campaigns and protect incumbents from competition.[9] In 1990, he led a filibuster against a campaign finance reform bill.[10] He spearheaded the movement against the Bipartisan Campaign Reform Act. His opposition to the bill culminated in the 2003 Supreme Court case McConnell v. Federal Election Commission and the 2010 Citizens United v. Federal Election Commission, the latter of which ruled part of the act unconstitutional. McConnell has been an advocate for free speech at least as far back as the early 1970s when he was teaching night courses at the University of Louisville. "No issue has shaped his career more than the intersection of campaign financing and free speech," political reporter Robert Costa wrote in 2012.[11] In a recording of a 2014 fundraiser McConnell expressed his disapproval of the McCain-Feingold law, saying,"The worst day of my political life was when President George W. Bush signed McCain-Feingold into law in the early part of his first Administration."[12] On January 2, 2013, the Public Campaign Action Fund, a liberal nonprofit group that backs stronger campaign finance regulation, released a report highlighting eight instances from McConnell's political career in which a vote or a blocked vote (filibuster), coincided with an influx of campaign contributions to McConnell's campaign.[13][14] In December 2018, McConnell opposed a Democratic resolution that overturned IRS guidance to reduce the amount of donor information certain tax-exempt groups would have to give the IRS, saying,"In a climate that is increasingly hostile to certain kinds of political expression and open debate, the last thing Washington needs to do is to chill the exercise of free speech and add to the sense of intimidation."[15] Prior to his Senate run, McConnell favored campaign finance reform, describing money in politics as a "cancer" in 1973.[10]</p>

<p>Cruz received more than US$1 million in campaign donations from the oil and gas industry since 2011.[104] Cruz is an opponent of the EPA, calling it "unbelievably abusive" and "populated by zealots,"[105] and has accused the Obama administration of waging a "war on coal."[105][106] Cruz opposes EPA's Clean Power Plan, which aims to cut carbon emissions from power plants; Cruz accused President Obama of engaging in a "lawless and radical attempt to destabilize the nation's energy system" and called the plan "flatly unconstitutional."[107] Cruz, who says that he is an advocate for "volunteer conservation," and also disagreed with efforts by the EPA to expand regulatory oversight on water, which Cruz characterized an attempt "to turn irrigation ditches into lakes and rivers and oceans."[108] Cruz is a supporter of TransCanada's controversial Keystone XL Pipeline,[106] and along with every other Republican senator was a cosponsor of legislation in support of the pipeline.[109] Like most Republican senators, Cruz supports opening areas for offshore drilling.[106] Cruz favors "expanding energy development on federal land" and has "pushed for preventing federal restrictions on fracking."[106] In July 2015, Cruz's super PAC,"Keep the Promise," disclosed that the billionaire brothers Dan and Farris Wilks, who earned a fortune in the West Texas fracking boom, contributed a record-setting $15 million to the super PAC.[110]</p>

<p>For safety reasons, large language models (LLMs) are trained to refuse harmful user instructions, such as assisting dangerous activities. We study an open question in this work. Can the desired safety refusal, typically enforced in chat contexts, be generalized to non-chat and agentic use cases? Unlike chatbots, LLM agents equipped with general-purpose tools, such as web browsers and mobile devices, can directly influence the real world, making it even more crucial to refuse harmful instructions. In this work, we primarily focus on red-teaming browser agents &ndash; LLMs that manipulate information via web browsers.&nbsp;<br><br>To this end, we introduce Browser Agent Red-teaming Toolkit (BrowserART), a comprehensive test suite designed specifically for red-teaming browser agents. BrowserART consists of 100 diverse browser-related harmful behaviors (including original behaviors and ones sourced from HarmBench [Mazeika et al., 2024] and AirBench 2024 [Zeng et al., 2024b]) across both synthetic and real websites.&nbsp;<br><br>Our empirical study on state-of-the-art browser agents reveals that, while the backbone LLM refuses harmful instructions as a chatbot, the corresponding agent does not. Moreover, attack methods designed to jailbreak refusal-trained LLMs in the chat settings transfer effectively to browser agents. With human rewrites, GPT-4o and o1-preview-based browser agents attempted 98 and 63 harmful behaviors (out of 100), respectively. We publicly release BrowserART and call on LLM developers, policymakers, and agent developers to collaborate on improving agent safety.</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=1b13997d80e8646969c95cee06965802.png&amp;f=webp&amp;q=75"></p>
<p>BrowserART consists of 100 harmful browser-related behaviors (including original behaviors and ones sourced from HarmBench [Mazeika et al., 2024] and AirBench 2024 [Zeng et al., 2024b]) that an agent is not supposed to assist. We divided all behaviors into two main categories: harmful content and harmful interaction. Under each main category, we created sub-categories for the harm semantics. We created 40 synthetic websites under 19 domains for red teaming browser behaviors that target specific websites (e.g., Twitter/X). These synthetic pages are hosted locally for running red teaming experiments in a sandbox without polluting the real world, especially social media and government sites.</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=6b9db5c096e4a32216e92f1b221d3af1.png&amp;f=webp&amp;q=75"></p>
<p>One of the major findings in our work is a clear gap between Attack Success Rates (ASRs) of the backbone LLM (blue) and its agent (lavender). That is, while the LLM refuses to follow the harmful behavior in the user instruction, the agent will. Gaps of GPT-4o and GPT-4-turbo models are most outstanding, compared to other backbone LLMs. In particular, we find Opus-3 and Llama-3.1-405B show the least drop in their refusal capabilities.</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=a10a6df8f013bd702541020a090824f4.png&amp;f=webp&amp;q=75"></p>
<h3 style="text-align: center;">Ethics and Disclosure</h3>
<p>This research &mdash; including the methodology detailed in the paper, the code, and the content of this webpage &mdash; contains material that may enable users to generate harmful content using certain publicly available LLM agents. While we recognize the associated risks, we believe it is essential to disclose this research in its entirety. The agent frameworks, beyond those used in this study, are publicly accessible and relatively easy to use. Comparable results will inevitably be achievable by any determined team seeking to utilize language models to produce harmful content and interactions.<br><br>In releasing BrowserART and our main results, we carefully weighed the benefits of empowering research in defense robustness with the risks of enabling further malicious use. Following Zou et al. (2024), we believe the publication of this work helps the agent safety community to release this frontier challenge.&nbsp;<br><br>Prior to release, we have also disclosed our findings and datasets to the companies providing the API access to the models, together with the creators of browser agent frameworks. Our findings highlight the crucial alignment gap between chatbots and browser agents and call upon the research community to explore safeguarding techniques for LLM agents.</p>
<h3 style="text-align: center;">Bibtex Citation</h3>
<p>If you are using the behavior set of /BrowserART, in addition to this work, please consider to cite HarmBench and AirBench 2024 using the following citations:</p>
<pre class="language-python"><code>@InProceedings{mazeika2024harmbench,
  title = 	 {{H}arm{B}ench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal},
  author =       {Mazeika, Mantas and Phan, Long and Yin, Xuwang and Zou, Andy and Wang, Zifan and Mu, Norman and Sakhaee, Elham and Li, Nathaniel and Basart, Steven and Li, Bo and Forsyth, David and Hendrycks, Dan},
  booktitle = 	 {Proceedings of the 41st International Conference on Machine Learning},
  year = 	 {2024},
  series = 	 {Proceedings of Machine Learning Research},
  publisher =    {PMLR},
}

@article{zeng2024air,
  title={AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies},
  author={Zeng, Yi and Yang, Yu and Zhou, Andy and Tan, Jeffrey Ziwei and Tu, Yuheng and Mai, Yifan and Klyman, Kevin and Pan, Minzhou and Jia, Ruoxi and Song, Dawn and others},
  journal={arXiv preprint arXiv:2407.17436},
  year={2024}
}

@misc{kumar2024refusaltrainedllmseasilyjailbroken,
      title={Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents}, 
      author={Priyanshu Kumar and Elaine Lau and Saranya Vijayakumar and Tu Trinh and Scale Red Team and Elaine Chang and Vaughn Robinson and Sean Hendryx and Shuyan Zhou and Matt Fredrikson and Summer Yue and Zifan Wang},
      year={2024},
      eprint={2410.13886},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2410.13886}, 
}</code></pre>

Browser Art.png

For safety reasons, large language models (LLMs) are trained to refuse harmful user instructions, such as assisting dangerous activities

Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents 

Open Graph Browser Art.png

Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents

Browser Art

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

homepage-event

custom-fonts

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

customers

Display order on the page /customers. Higher values show first

summit-2023-speakers

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

basecamp-agenda

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

summit-2023-agenda

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

research

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

leaderboards

Products

Enterprise

Government

Resources

Customers

Leaderboards →

Scale AI Research

Scale AI’s mission is to accelerate the development of AI applications. By advancing research, we aim to create AI systems capable of solving complex, human-level problems.

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs

Revisiting the Superficial Alignment Hypothesis

Learning Goal-Conditioned Representations for Language Reward Models

Planning In Natural Language Improves LLM Search For Code Generation

A Careful Examination of Large Language Model Performance on Grade School Arithmetic

LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet

Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents

LLM Leaderboards

Expert-Led Private Evaluations for precise and reliable LLM rankings

Join the Scale AI Research Team

News

The future of your industry starts here.