Scale AI, in collaboration with the Center for Strategic and International Studies (CSIS), is proud to introduce the Critical Foreign Policy Decision (CFPD) Benchmark—a pioneering effort to evaluate large language models (LLMs) on national security and foreign policy decision-making tendencies.

The Ministry of Communications and Information Technology (MCIT) and Scale AI, the leader in frontier AI solutions, are announcing a strategic, long-term partnership to drive Qatar’s digital transformation.

Contested basing environments require scalable solutions for perimeter security. Scale AI, the Defense Innovation Unit, and the U.S. Air Force are demonstrating the value of computer vision to force protection challenges globally.

Scale researchers have discovered a groundbreaking method for AI safety testing called J2 (Jailbreaking to Jailbreak), where language models are taught to systematically test their own and other models' safety measures. This hybrid approach combines human-like strategic reasoning with automated scalability, achieving success rates of over 90% in vulnerability testing, nearly matching professional human red-teaming effectiveness. While highlighting significant advances in automated security testing, these findings also reveal important challenges for the future of AI safety.

Scale AI leads groundbreaking research to build safer, more capable AI systems through innovative approaches in post-training optimization, agent development, and evaluation frameworks. Their comprehensive work spans from improving model performance and reliability to developing robust safety measures, all while maintaining a commitment to open collaboration and industry-wide advancement. Through the Safety, Evaluations, and Alignment Lab (SEAL) and various research initiatives, Scale AI is shaping the future of responsible AI development.

Scale’s AISI-approved AI model evaluations are setting a new standard for pre-deployment testing. By offering voluntary, efficient, and third-party validated assessments, we are empowering AI developers to create more reliable models—without the complexities that typically slow down the process.

Text2SQL systems promise to democratize access to enterprise data but often fail to handle the complexity of real-world database queries, even if they perform well on test datasets. We found that Reinforcement Learning from Human Feedback (RLHF) is a viable approach for active learning from incorrect production queries to improve Text2SQL accuracy.

Scale AI and the Center for AI Safety (CAIS) are proud to publish the results of Humanity’s Last Exam, a groundbreaking new AI benchmark that was designed to test the limits of AI knowledge at the frontiers of human expertise.

On January 21st, 2025, Scale AI Founder and CEO, Alexandr Wang sent a letter to President Donald J. Trump on what America needs to do to win the AI war. Read the blog to see the full letter.

As we return to work after the holiday break, the Scale AI Public Sector team wanted to reflect on our work heading into 2025. As strategic rivalries continue to intensify and adversaries form new alliances globally to challenge U.S. leadership in AI, the mission of Scale’s Public Sector team has never been more vital. We are dedicated to ensuring that the U.S. and its allies have the best technology to lead in this increasingly complex global landscape. The snapshot below captures a few key highlights from last year: