Scale AI is one of the world’s leading suppliers of high-quality foundation model training data. In recent months we’ve seen a marked increase in demand for PhD-level, multimodal reasoning data across a multitude of domains, including math, coding, science, and humanities. Because of the ever-increasing difficulty of such expert-level data, ScaleAI invests in frontier research on quality control (QC) with LLM agents, also known as “autoraters.”

The release of the U.S. AI Action Plan is a major step toward solidifying America’s leadership in artificial intelligence. At Scale, we’re proud to support its priorities—from promoting U.S. AI globally, to expanding its use in government, to investing in innovation and infrastructure. Here’s our perspective on what this means for the future of AI.

Our Enterprise team breaks down recent AI developments and explores what (if anything) it means for the enterprise world.

Building truly intelligent and equitable multilingual AI requires a new way to measure cultural reasoning. Scale's new Multilingual Native Reasoning Challenge (MultiNRC) is designed to do just that. Created from scratch by native speakers, this benchmark tests for deep linguistic and cultural understanding beyond simple translation, providing a clear path for the AI community to accelerate progress.

As AI agents become more powerful, ensuring their safety is the most critical challenge for deployment. This post explores WebGuard, a new benchmark from researchers at Scale, UC Berkeley, and The Ohio State University that reveals a significant safety gap in current models. Learn how high-quality, human-in-the-loop data provides a path forward, dramatically improving a model's ability to avoid risky behavior.

A recent viral study from METR challenged a core assumption of the AI era: that AI tools make developers more productive. It found that expert developers were actually 19% slower when using them, even though they felt 20% faster. In this post, we break down the likely reasons for this surprising slowdown and argue the study's focus on expert speed leads us to consider a more profound story: AI's true value may be in empowering a new generation of vibe coders and managers to build things that otherwise would never have existed. Ultimately, the question isn't just whether AI makes us faster. It's about how we measure value in an era where enjoyment, context, and empowerment are becoming just as important as the clock.

Our Enterprise team reacts to controversial statements about AI and votes on whether they agree or disagree.

One of AI's biggest challenges is "reward hacking," where models learn to game the system for a correct answer instead of actually reasoning. This hidden deception makes AI untrustworthy. Scale research has found a powerful solution: instead of stopping the hacking, get the model to admit to it in its Chain-of-Thought reasoning. This new paper details how Verbalization Fine-Tuning (VFT) trains models to announce their shortcuts, dramatically increasing transparency from 11% to 94% and making AI systems fundamentally safer.

A new research collaboration led by a MATS scholar and advised by a team of researchers from Anthropic, Scale, and other research institutes introduces SHADE-Arena, a benchmark for detecting and evaluating subtle sabotage by AI agents. Within 17 complex scenarios, advanced models were tasked with completing a primary goal while secretly pursuing a harmful objective, all under the watch of an AI monitor. The results show that even top models like Claude 3.7 Sonnet and Gemini 2.5 Pro rarely succeed at this deception, often making simple errors. However, the study also reveals that monitors are not yet reliable enough for safety-critical systems and that an agent's private "scratchpad" is a key vulnerability. This work establishes a vital baseline for tracking and defending against agentic risks as AI capabilities evolve.

In response to Anthropic's system card and safety testing for Claude 4 Opus and Sonnet, this post explores the complex behaviors of today's frontier AI models. In a comparative testing of reasoning models, we observed emergent behaviors that included instances of blackmail, user impersonation, and deception, with different models reacting to the scenario in unique ways. These findings contribute to the ongoing industry-wide conversation about AI safety, highlighting the nuances of model alignment and the critical importance of carefully defining system access and agency as these powerful tools evolve.