experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Product

Company

General

Government

People

Engineering

Scale AI is fueling the public sector&rsquo;s use of AI, serving as the engine for building, deploying, and evaluating AI across the U.S. government. America&rsquo;s national security depends on equipping our government with the best AI tools and data infrastructure. This requires investing in both cutting-edge technology as well as a talented national workforce. Our AI Center in St. Louis provides a model for how advancing the United States&rsquo; AI capabilities brings new career opportunities for Americans eager to break into a career in tech.
Scale&rsquo;s St. Louis AI Center demonstrates how investment in AI creates high-tech jobs for Americans from diverse backgrounds to support U.S. national security. This center was originally created three years ago to label geospatial AI training data. This data is used to improve AI models used by our government to analyze vast amounts of information collected by satellites, drones, and other sensors. Today, Scale&rsquo;s AI Center provides work to hundreds of data labeling specialists who have produced tens of millions of high-quality geospatial data annotations, which fuel the systems used by government leaders making mission-critical decisions. The AI Center also recently expanded into generative AI. This involves training AI models to communicate clearly using military terminology. Together, this work enhances the performance of AI systems and equips our deployed forces around the world with a technological edge to help them stay safe and effective.
Scale is committed to ensuring the U.S. government has the world&rsquo;s most advanced AI technology. One example of the center&rsquo;s work is&nbsp;<a href="https://scale.com/blog/ukraine-detection">a free dataset</a> Scale created in 2022 for our national security partners to generate more accurate building damage assessments during Russia&rsquo;s invasion of Ukraine and the dataset received TIME&rsquo;s <a href="https://time.com/collection/best-inventions-2022/6226995/scale-ai-automated-damage-identification-as-a-service/">&ldquo;Best Inventions&rdquo; award</a> in 2022. At a time when the United States&rsquo; competitors are investing heavily in advancing their own AI capabilities, our nation&rsquo;s AI systems remain bottle-necked by our ability to produce high-quality data. The demand is growing for AI workforces that can refine the data we need to train and update the U.S. government&rsquo;s AI models. Our St. Louis center is one example of how that demand leads to quality work for Americans across the country.&nbsp;
As Scale has learned from our center in St. Louis, this demand is also creating a new category of jobs that requires unique skill sets. The majority of our work at the AI Center focuses on &ldquo;cracking the code&rdquo; of complicated geospatial data, and it rewards complex pattern-matching abilities and a high attention to detail. Most of our St. Louis workforce comes from non-traditional backgrounds, like baristas, artists, and nurses. A portion of our workforce also includes military veterans who sought out Scale to continue to serve our nation after their time in uniform. Regardless of their backgrounds, our AI center workers develop the ability to analyze multiple data types through advanced training and instruction from geospatial experts with decades of experience.&nbsp;
Scale&rsquo;s work also offers people a chance to build careers in a high-tech field. As our workforce at the AI Center has grown, so has the need for managerial and specialized roles. Two-thirds of our current workforce have advanced beyond their initial entry-level positions. Alex, an operations manager, joined in an entry-level role two years ago and has now earned four promotions to her current role. Now that Scale also supports classified work for the U.S. government, we have also enabled more than one hundred team members to receive security clearances. This credential allows them to work on some of the most pressing national security challenges even beyond their roles at Scale.
This work is strengthening communities by bringing new investment into local businesses, as our AI Center is fully integrated into the St. Louis business and geospatial communities. For example, we joined the Rung for Women Employer Network, an organization that helps local women build careers in high-opportunity industries. Scale&rsquo;s AI Center is also partnering with the following St. Louis-based small businesses including:
<ul>
<li dir="ltr" aria-level="1">
Chameleon Integrated Services, a minority-owned small business
</li>
<li dir="ltr" aria-level="1">
Mark One Solutions, a service disabled veteran-owned small business
</li>
<li dir="ltr" aria-level="1">
Hopewell Enterprises, a woman-owned, minority-owned small business&nbsp;
</li>
</ul>
As St. Louis&rsquo; Mayor Tishaura Jones said, &ldquo;We&rsquo;re fortunate that Scale has those entry-level opportunities for people to get into tech&hellip;If a mayor from another city called me and asked whether Scale should expand to their community, I&rsquo;d say absolutely. We&rsquo;re trying to provide the environment for companies like Scale to thrive and grow.&rdquo;
As we continue to partner with public sector leaders in advancing our nation&rsquo;s AI capabilities, we are proud to support the United States&rsquo; emerging AI workforce.

image (35).png

Scale's AI Center in St. Louis provides a model for advancing the United States’ AI capabilities.

Scale's St. Louis AI Center

Scale's St. Louis AI Center: A Blueprint for America's AI Workforce

Scale recently had the privilege of hosting the second Joint Cyber Defense Collaborative (JCDC) Artificial Intelligence Cyber Tabletop Exercise (TTX) at our new Global Headquarters in San Francisco. This event, organized by the <a href="https://www.cisa.gov/">Cybersecurity and Infrastructure Security Agency (CISA)</a>, brought together more than 90 experts from government and industry to explore how we can collectively address the unique challenges of AI security. It was an inspiring opportunity to collaborate with some of the brightest minds in cybersecurity and help shape the future of AI incident response.
Public-private partnerships are essential to tackle emerging threats quickly in ways that minimize impact and risk. The diversity of participants and perspectives, including representatives from the U.S. government, financial sector, tech, and leading AI companies, is what made this exercise so impactful because no single organization can solve these complex challenges alone.&nbsp;
AI security is not just about building safe models&mdash;it involves securing every layer of AI integration, from model development to deployment into existing technology systems. At Scale, we recognize that effective AI security begins with a comprehensive development life cycle. This life cycle includes safety measures during AI model pre-training as well as post-training activities like Test and Evaluation. Both of these ultimately focus on the underlying security of the model itself. The outermost layers&mdash;where we integrate model inference and retrieval-augmented generation (RAG) into traditional technology and data systems&mdash;must be protected to ensure comprehensive security controls for AI. This is where traditional cybersecurity has the most to contribute towards securing the AI future.&nbsp;
Using what we&rsquo;ve learned over the last several decades, <a href="https://csrc.nist.gov/glossary/term/defense_in_depth">defense-in-depth</a> principles can be applied to AI-enabled technologies in ways that allow rapid adoption and transformation. A key part of those traditional security controls focuses on monitoring, detection, and incident response to emerging and previously unknown vulnerabilities. The JCDC.AI TTX allowed us to engage in meaningful conversations about these challenges, brainstorm best practices, and raise our collective understanding of how to ensure societal trust in AI-enabled systems. <a href="https://www.cisa.gov/topics/partnerships-and-collaboration/joint-cyber-defense-collaborative/Joint-Cyber-Defense-Collaborative-Artificial-Intelligence-Cyber-Tabletop-Exercise-Series">Following the first TTX, </a>CISA Director Jen Easterly said, "This exercise marks another step in our collective commitment to reducing the risks posed by AI. It also highlights the importance of developing and delivering AI products that are designed with security as the top priority."
 At Scale, we are proud to support CISA&rsquo;s mission and contribute our expertise to initiatives like JCDC.AI. We see our role not just as a leader in AI technology, but also as an advocate for the safe and responsible use of AI across industries. By working closely with these partners, we can help ensure that the benefits of AI are realized while mitigating the risks that come with its adoption. We invite other organizations to join us in this effort. <a href="https://scale.com/evaluation/public-sector">Click here to learn more about our work to ensure safe and secure AI.</a>

image (6).png

Scale AI recently hosted the second JCDC AI-Cyber TTX at our new HQ in SF. 

Scale AI hosts the second JCDC.AI Cyber TTX

Building a Secure AI Future: Scale Hosts the Joint Cyber Defense Collaborative AI Cyber Tabletop Exercise

We are thrilled to announce that Scale AI has achieved FedRAMP&reg; HIGH authorization, a monumental achievement that solidifies our commitment to delivering secure, cutting-edge AI solutions to federal agencies. With this authorization, we&rsquo;re proud to offer the full power of&nbsp;Scale&nbsp;<a href="https://scale.com/public-sector-data-engine">Data Engine</a>, <a href="https://scale.com/genai-platform">GenAI Platform (SGP)</a>, and <a href="https://scale.com/donovan">Donovan</a>, along with our rigorous <a href="https://scale.com/evaluation/public-sector">Test and Evaluation (T&amp;E) process</a>, to help federal agencies unlock the potential of AI in a secure and compliant environment. This milestone reflects our unwavering dedication to supporting the federal government with transformative AI technologies that drive innovation and&nbsp;mission success. Federal agencies can now access industry-leading tools through Scale AI&nbsp;to optimize AI model performance and meet the rigorous security and compliance requirements for public sector missions.
 Scale Data Engine is recognized as the premier solution for curating and labeling massive datasets that power the latest advancements in generative AI, computer vision, and machine learning models. Our platform accelerates the development of trustworthy AI systems by ensuring that the underlying data is accurate, high-quality, and comprehensive. As a result, government agencies can deploy AI solutions across a wide variety of critical missions across several modalities - geospatial, multimodal, and text annotation.
 With the addition of Scale GenAI Platform (SGP), federal agencies have powerful tools at their disposal to develop, test, evaluate, and monitor AI applications. Our Test and Evaluation (T&amp;E) process is a foundational element that ensures these systems are safe, performant, and reliable, especially in mission-critical environments. Scale&rsquo;s solutions incorporate T&amp;E at every layer, from data labeling to model development, ensuring that AI technologies meet the stringent requirements of public sector customers. Beyond data and AI development capabilities, Scale Donovan is a large language model (LLM) enterprise chat application that can operationalize agentic AI Assistants built with SGP. Donovan, which is LLM agnostic, enhances operational efficiency by providing real-time insights and AI-driven collaboration, enabling agencies to streamline workflows, leverage leading AI models, and improve mission execution. As new workflows and use cases are identified, Donovan and SGP provide the necessary foundation to test, integrate and leverage Generative AI for responsible mission impact.
Scale&rsquo;s Commitment to Security in Federal Data Environments
Scale&rsquo;s platform is designed to address the complex security challenges that are unique to the public sector. Our Security Team is dedicated to meeting the needs of agencies handling sensitive, highly regulated data. Whether it&rsquo;s safeguarding classified information or ensuring compliance with FedRAMP&rsquo;s strict standards, Scale&rsquo;s team works tirelessly to support the diverse security demands of our public sector customers.
 Achieving FedRAMP HIGH authorization is a testament to our ongoing commitment to protecting the federal government&rsquo;s most sensitive data. This authorization allows Scale to provide secure cloud environments that support a broad spectrum of sensitive data, ensuring compliance with Controlled Unclassified Information (CUI) and other federal security requirements. Additionally, by supporting classified environments (SIPR/JWICS) in DoD and Intelligence Community (IC) settings, along with unclassified environments for civilian agencies, Scale delivers mission-critical success across the entire federal landscape. Scale AI has also earned Impact Level 4 (IL-4) Provisional Authorization from the Defense Information Systems Agency (DISA).
 As we continue to drive innovation, Scale remains committed to providing secure, AI-driven solutions that empower the federal government. Through the strength of Scale Data Engine, GenAI Platform, Donovan, and our robust Test and Evaluation, we are helping federal agencies achieve their missions with safe, secure, and transformative AI capabilities. Learn how we can help your organization at <a href="https://scale.com/federal">https://scale.com/federal</a>

image (4).png

We are thrilled to announce that Scale AI has achieved FedRAMP  ® HIGH authorization/

Scale Achieves FedRAMP HIGH Authorization

long_context.png

A Guide to Improving Long Context Instruction Following on Open Source Models

A Guide to Improving Long Context Instruction Following

For months, anticipation has mounted over rumors of a next-generation model from OpenAI, reportedly codenamed &ldquo;Strawberry,&rdquo; that would bring groundbreaking improvements in reasoning and complex problem-solving. That wait ended on September 12, with the preview release of two new reasoning models: o1-preview and o1-mini.
According to OpenAI, the models represent a "new paradigm" for scaling inference alongside training compute, and show state-of-the-art performance on challenging reasoning benchmarks.
The o1 models quickly gained attention for their remarkable scores, particularly in domains like advanced coding, mathematics, and science, and independent evaluations from <a href="https://scale.com/leaderboard">Scale&rsquo;s SEAL Leaderboards</a> broadly confirm these results.
However, the models also introduce new changes to prompt engineering, with updated practices compared to previous models. And despite demonstrations of o1's ability to solve complex puzzles and advanced reasoning tasks, it can still at times be fooled by "gotcha" questions viral on social media &mdash; like counting the number of "r"s in the word strawberry. The models also do not yet support some key features of ChatGPT and GPT-4o, like image input.
Sifting through the claims, LLM users may be left wondering which model is right for their needs.
In this post, I overview benchmark results for these models, and give qualitative impressions and dialog examples from my time testing them, with some thoughts and discussion on how prompting is changing in this new &ldquo;paradigm&rdquo; of scaling.
<h1 dir="ltr">Introducing the o1 models</h1>
OpenAI's o1-preview and o1-mini are a major step forward in reasoning, designed for tasks that require many complex steps to solve, as well as an improved familiarity with advanced math and science. o1-mini offers a lower-cost, faster option optimized for coding and STEM knowledge while being around 80% cheaper than o1-preview. Although smaller, o1-mini performs competitively in coding tasks compared to&nbsp;o1-preview.
Much of o1&rsquo;s improved reasoning is attributed to its use of chain-of-thought (CoT) to emit reasoning steps. But what sets o1 apart from previous models is its use of reinforcement learning (RL) in training the model to produce these steps. This RL-enhanced CoT enables o1 to learn to navigate deeper, more complex challenges without fixating on failed reasoning paths &mdash; a common result for long chain-of-thought output in current models.
Another advantage of this approach is it allows OpenAI to monitor the model&rsquo;s reasoning for signs of deceptive behavior, a major concern for the company as models become more capable. OpenAI also notes o1&rsquo;s use of chain-of-thought rationales improves resistance to jailbreaks and adversarial prompts.
OpenAI explains that o1 uses reinforcement learning to teach the model to reason productively using its chain of thought in a new, data-efficient training process. Crucially, they found that more compute consistently improves performance both at training and inference time. It&rsquo;s this change to the scaling relationship of performance and compute that leads OpenAI to describe o1 as the start of a &ldquo;new paradigm.&rdquo; (However, they note constraints on this new form of scaling differ significantly from LLM pretraining and are still being explored.)
The current o1-preview and o1-mini do have some practical limitations compared to GPT-4o. The o1 models do not support many existing ChatGPT features like web browsing, image or file uploads, image generation, or the code interpreter tool. OpenAI&rsquo;s documentation notes that even with the o1 models available, GPT-4o will still be a better choice for many existing tasks using LLMs, especially when reliably low latency or multimodal input/output is needed.
<h1>Scale SEAL Leaderboard rankings</h1>
All SEAL Leaderboards updated to include the new o1 models show leading performance from both.
Currently, o1-preview leads in Precise Instruction Following and Spanish-language fluency, and for Coding o1-mini outperforms o1-preview to take the top spot, with o1-preview in second place.
<a title="Scale SEAL LLM Leaderboards" href="https://scale.com/leaderboard"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=7a5566cf04edf1999da3d0288f98a580.png&amp;f=webp&amp;q=75" alt="" width="1415" height="929"></a>
Scale&rsquo;s <a href="https://scale.com/leaderboard">SEAL Leaderboards</a> provide trusted, third-party evaluations of large language models (LLMs) using curated, private datasets. Each Leaderboard focuses on a specific domain such as coding, instruction-following, and multilingual abilities. Building on insights from research like Scale&rsquo;s GSM1K dataset (<a href="https://arxiv.org/abs/2405.00332">H. Zhang et al. 2024</a>), SEAL evaluations are maintained privately, avoiding leakage issues that increasingly plague open-source benchmarks.
<h4 dir="ltr"><a href="https://scale.com/leaderboard/coding">Scale SEAL Leaderboard for Coding</a></h4>
On the SEAL Leaderboard for coding, o1-mini leads with a score of 1271, followed by o1-preview at 1198. The Coding Leaderboard dataset uses 1,000 prompts to test various coding tasks, from code generation to optimization and documentation creation. Each response is evaluated for correctness, performance, and readability, combining human reviewers and code execution tests.
<h4 dir="ltr"><a href="https://scale.com/leaderboard/instruction_following">Scale SEAL Leaderboard for Instruction Following</a></h4>
In the SEAL Leaderboard&rsquo;s evaluation of precise instruction following, o1-preview leads with a score of 87.27, surpassing Claude 3.5 Sonnet and Llama 3.1 405B Instruct. The dataset contains 1,054 prompts across domains including text generation, brainstorming, and educational support.
<h4 dir="ltr"><a href="https://scale.com/leaderboard/spanish">Scale SEAL Leaderboard for Spanish</a></h4>
In the SEAL Leaderboard's evaluation of Spanish-language fluency, o1-preview emerges as the top model, achieving a score of 1119, surpassing models like GPT-4o (May 2024) and Gemini 1.5 Pro (August 2024). The evaluation includes a range of single-turn prompts, assessing general conversations as well as culturally nuanced topics across Spanish-speaking regions. The evaluation's categories included educational support, technical assistance, daily life advice, and more.
<h1 dir="ltr">Other evaluations and benchmarks</h1>
In their announcement, OpenAI highlights remarkable achievements from o1 models on complex reasoning benchmarks. Most notably, o1 (the larger, still unreleased model, not&nbsp;o1-preview or o1-mini) ranks in the 89th percentile on Codeforces, underscoring its coding proficiency, and it placed among the top 500 students in the USA on the American Invitational Mathematics Examination (AIME). Furthermore, the larger o1 showed human PhD-level accuracy on the GPQA benchmark, which evaluates reasoning in physics, biology, and chemistry​.
Other third-party reviews largely affirm these results, but with some nuance. As detailed above, o1 models lead on all current SEAL Leaderboard benchmarks. When o1 models were added to the LMSYS Chatbot Arena Leaderboard, they quickly rose to the top &mdash; at that time, o1-preview held the top spot with an Elo score of 1355, with o1-mini tied for second and GPT-4o-latest in third at 1324. However, performance on the ARC-AGI benchmark was more modest. o1-preview scored 21%, similar to Claude 3.5 Sonnet, with o1-mini scoring just 13%.
<h1 dir="ltr">Changes to prompt engineering</h1>
Prompting and steerability with o1 models is notably different from familiar models like GPT, Gemini, or Claude, requiring some new best practices for prompts. While powerful, the new o1 models are not always as intuitive to use, especially for precise, procedural tasks.
According to <a href="https://platform.openai.com/docs/guides/reasoning/advice-on-prompting">OpenAI&rsquo;s recommendations</a>, simple and direct instruction help for getting the most out of o1. Unlike with previous models, users should avoid asking for chain-of-thought reasoning. They also note irrelevant context in the prompt can be more distracting to o1 models than the previous GPT series, making it important to include fewer examples in retrieval-augmented generation (RAG) prompts.
Cognition Labs offers similar advice from their experience with o1, reporting that asking the model to &ldquo;think out loud&rdquo; actually harms performance, and asking for only a final answer improves it, as o1 models will produce an internal chain-of-thought regardless. They also note verbose or repetitive instructions hurt performance, and prescriptive detail seems to impair the model&rsquo;s reasoning.
My own experience working with o1-preview and o1-mini the past few weeks aligns with these points. Reasoning from o1 models is extremely impressive when it works, but they tend to ignore clear (or even emphatic) instructions for how to solve a problem. An example of this behavior is shown in the last of the three examples below.
While o1 delivers outstanding results on benchmarks, getting your own tasks to work can seem to take more effort. I suspect this hints at a gap between real-world prompts and one seen in benchmarks: the latter aims to have only unambiguous, self-contained, minimally presented problems with no advice or opinions on how to solve them. I also expect this issue will improve greatly once o1 models are deployed without their current usage caps, which can impede discovering high-quality prompts.
It&rsquo;s also worth noting that the latency of o1-preview&rsquo;s responses, especially its &ldquo;time to first token&rdquo;, is significantly higher than GPT-4o. This would limit the use of these models for some applications. But for interactive use in ChatGPT, I didn&rsquo;t find this a major annoyance. Only o1-preview is noticeably slower; o1-mini more than makes up for its &ldquo;thinking&rdquo; time with faster token inference.
<h1 dir="ltr">Dialog examples</h1>
<h2>Example: Vocabulary constraint problem</h2>
OpenAI&rsquo;s o1 model announcements mentions improved performance on viral &ldquo;gotcha&rdquo; tasks known to trip up LLMs, like the notorious &ldquo;How many R&rsquo;s are in the word &lsquo;strawberry&rsquo;?&rdquo; &mdash; a question that fools many leading models. One of their examples nods to this question by elaborately presenting it as the result of an encryption puzzle.
To illustrate the real, but still incomplete, improvements in this area, I present o1-preview with a newly written riddle:
Name an English adjective of Latin origin that begins and ends with the same letter, has 11 letters in total, and for which all vowels in the word are ordered alphabetically.
On the first attempt, the model successfully solves this riddle with the answer: sententious.&nbsp;
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=b7299a65bfc23cda566b8afffda6854c.png&amp;f=webp&amp;q=75" alt="" width="1536" height="753">
I repeated this prompt a total of five times, receiving the following responses:
<ul>
<li>sententious ✅</li>
<li>facetiously ❌</li>
<li>transparent ✅</li>
<li>abstentious ❌</li>
<li>facetiously ❌</li>
</ul>
For comparison, ChatGPT 4o and Llama 3.1 70B Instruct each returned 0 correct solutions for this prompt in 10 attempts each.
<h2>Dialog example: Decoding ciphers</h2>
The <a href="https://openai.com/index/learning-to-reason-with-llms/">OpenAI technical blog</a> post announcing the o1 models includes an impressive example of o1-preview decoding a complex cipher. In the example, the model is presented with a text using an ad hoc cipher based on successive pairs of characters.
To explore how well this ability generalizes to similar examples, I tried variations of this same prompt for various compositions of the ROT13 cipher, Atbash cipher, Base64 encoding, string reversal. Most of these tests were not successful &mdash; in only 2 of 7 informal tests was o1-preview able to decode the encoded message given (the&nbsp;Litany Against Fear from Frank Herbert&rsquo;s Dune) in one attempt &mdash; however, these tests are quite difficult.
In each prompt the model is asked to infer an encoding from an example of the transformation applied to the plaintext &ldquo;Think step by step&rdquo; as OpenAI&rsquo;s example.
Each of the following tests, the model failed to decode the target message in one attempt:
<ul>
<li>ROT13 cipher &rarr; reverse string &rarr; Base64 encode &rarr; reverse string</li>
<li>ROT13 cipher &rarr; Base64 encode &rarr; ROT13 cipher &rarr; reverse string</li>
<li>ROT13 cipher &rarr; Base64 encode &rarr; ROT13 cipher</li>
<li>ROT13 cipher &rarr; Base64 encode &rarr; Atbash cipher</li>
<li>ROT13 cipher &rarr; Base58 encode</li>
</ul>
The two tests decoded successfully on the first attempt were:
<ul>
<li>Atbash cipher &rarr; Base64 encode</li>
<li>ROT13 cipher &rarr; Base64 encode</li>
</ul>
The first of these successes is shown here &mdash; other tests are identical except for the encoding used:
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=d1a468066ff4b60960343bc4a9569464.png&amp;f=webp&amp;q=75" alt="" width="1536" height="1638">
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=1806bb7ede68cdc4edccaf7cdd95bf7c.png&amp;f=webp&amp;q=75" alt="" width="1536" height="1622">
<h2>Dialog example: Potential steerability issues</h2>
In this example o1-mini notably gives a false, detailed report of seemingly hallucinated verifications of an incorrect answer.
My prompt requests from the model a &ldquo;poem about strawberries&rdquo; containing exactly 13 R&rsquo;s, and requesting that at least 5 successive verification checks be performed before returning an answer.
In its response, the model returns a poem containing 21 R&rsquo;s. This is not surprising in itself &mdash; most LLMs are known to struggle to accurately identify or count letters within words when asked naively.
What&rsquo;s less typical is that the response below includes hallucinated verification steps as though it were a human completing the task using a computer: &ldquo;I applied a regex pattern,&rdquo; &ldquo;I read the poem aloud,&rdquo; and so on.
Reviewing summaries of chain-of-thought given in ChatGPT often include &ldquo;anthropomorphic&rdquo; descriptions of imagined events in this style. This example is somewhat consistent with the o1 <a href="https://openai.com/index/openai-o1-system-card/">system card</a> which notes some testers reported increased hallucination.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=863c64431b431e7949d002b10ecb7e20.png&amp;f=webp&amp;q=75" alt="" width="1536" height="1552">
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=0399c1529d939823383708b36dfca5ba.png&amp;f=webp&amp;q=75" alt="" width="1535" height="1924">
<h1 dir="ltr">Conclusion</h1>
OpenAI&rsquo;s o1 models are, by any measure, a major leap forward in reasoning capabilities, with standout performances on key benchmarks like AIME, Codeforces, Scale&rsquo;s SEAL Leaderboards, and others. These results show o1-preview and o1-mini are powerful tools for hard reasoning problems. Nonetheless, getting the most out of these models may take more experimentation than users are used to from other model releases.
As an independent, private evaluation, Scale&rsquo;s <a href="https://scale.com/leaderboard">SEAL Leaderboards</a> provide a clearer picture of where new model releases shine, and provide trusted validation of results needed for developers to choose the best models for their LLM-powered applications.
For industry professionals training frontier LLMs, Scale offers custom evaluations to benchmark and optimize models on secure proprietary datasets. For more information, <a href="https://scale.com/demo?utm_source=link&amp;utm_medium=blog&amp;utm_campaign=o1impression&amp;utm_id=o1impression">schedule a demo with Scale</a>.
<hr>
About the author: One of the first LLM prompt engineers, since 2022 Riley Goodside has developed prompts for Scale and is also known for the prompting examples and insights he shares on <a href="https://x.com/goodside">X</a>. Riley is now Staff AI Evangelist at Scale.
&nbsp;

image.png

First impressions on OpenAI o1 performance with prompt examples

First Impressions of OpenAI’s o1

When people ask why I joined Scale, I tell them it&rsquo;s about the opportunity to be a part of the most fundamental change happening in technology in my lifetime. I&rsquo;ve been lucky enough to start companies and live through the phases of the internet back to the days of dial up and this is the most significant shift yet.
Previously at Uber Eats and AXON we built technology that improves everyday life. Both impactful in very different ways. At Scale the opportunity is to be impactful across every interaction people have with AI which will touch every aspect of our personal and professional lives. That depth and breadth of potential impact is extremely compelling.
Specifically, Scale tackles one of the hardest challenges in AI: improving models through data. Doing this requires incredible talent, sophisticated operations *and* strong vision for where AI is going. It&rsquo;s remarkable what the team has accomplished so far and, yet, we&rsquo;re still at the very beginning! I learned at Uber that the magic is in the details and that&rsquo;s something Scale takes very seriously.
<h3 dir="ltr">The Allure of Scale</h3>
The culture at Scale is not just about aligning with the company&rsquo;s core credos but about thriving in an environment where innovation is at its core. Scale&rsquo;s commitment to solving complex problems and pushing the boundaries of what&rsquo;s possible in AI was evident from my very first conversation with the team.
What really resonated with me was the deep dedication to finding the right solutions, regardless of the challenge&rsquo;s complexity. Scale is not just another tech startup, it's a place that constantly thinks from first principles about what is right and necessary.
The team&rsquo;s enthusiasm and dedication to advancing the field of AI, especially through generative AI, were incredibly inspiring. This is what makes Scale stand out as a company: it doesn&rsquo;t just adapt to change but actively drives it.
<h3 dir="ltr">The Impact of AI&nbsp;</h3>
Scale&rsquo;s mission to accelerate AI development by solving the key bottleneck of data scarcity and quality struck a chord with me, given that the company has been at the forefront of AI data for 8 years, fueling major advancements in areas like autonomous vehicles, defense, and generative AI. Their mission to enhance AI data quality has the power to influence every industry.
<h3 dir="ltr">Growth in a Rapidly Evolving Landscape</h3>
At Scale, the stakes are high but that&rsquo;s how we like it.
If you like big problems and high standards, you should probably be here.&nbsp;
The opportunity to contribute to a company that stands at the forefront of AI and data technology is exhilarating. At Scale, I see the chance to not only drive innovation but also to be part of a mission that is fundamentally transforming how businesses operate and how AI interacts with the world. This mission is not just about incremental improvements but about making groundbreaking strides that can redefine entire industries.
If you&rsquo;re inspired by what you&rsquo;ve read and believe Scale could be the right fit for you, I invite you to apply to join us on this exciting journey.&nbsp;

image (47).png

Jason Droege is joining Scale as Chief Strategy Officer and explains why he joined. 

Why I Joined Scale: Jason Droege

Scale AI and CAIS are excited to announce the <a href="https://agi.safe.ai/submit">launch of Humanity's Last Exam</a>, a project aimed at measuring how close we are to achieving expert-level AI systems. The exam is aimed at building the world's most difficult public AI benchmark gathering experts across all fields. People who submit successful questions will be invited as coauthors on the paper for the dataset and have a chance to win money from a $500,000 prize pool.&nbsp;
<h2 dir="ltr">Scale &amp; CAIS</h2>
Scale&rsquo;s Safety, Evaluations, and Alignment Lab (SEAL) is dedicated to researching robust evaluation methods for frontier models, and enhancing the transparency of LLM progress. As part of this mission, SEAL periodically publishes <a href="https://scale.com/leaderboard">leaderboards</a> based on expert-driven private evaluations, helping the AI community gain deep insights into leading models.
The Center for AI Safety (CAIS) is a San Francisco-based research and field-building nonprofit. CAIS&rsquo;s mission is to reduce societal-scale risks associated with AI by conducting safety research, building the field of AI safety researchers, and advocating for safety standards.
<h2 dir="ltr">Why Participate?</h2>
AI is developing at a rapid pace. Just a few years ago, AI systems performed no better than random chance on MMLU, the AI community&rsquo;s <a href="https://huggingface.co/datasets?sort=downloads">most-downloaded benchmark</a> (developed by CAIS). But just last week, <a href="https://openai.com/index/learning-to-reason-with-llms/">OpenAI&rsquo;s newest model</a> performed around the ceiling on all of the most popular benchmarks, including MMLU, and received top scores on a variety of highly competitive STEM olympiads. As existing tests become too easy, we lose the ability to distinguish between AI systems which can ace undergrad exams, and those which can genuinely contribute to frontier research and problem solving.&nbsp;
Because AI systems of the future will eventually surpass all static benchmarks which can be created, pushing the boundary of benchmarks and evaluation is paramount. To keep track of how far AI systems are from expert-level capabilities, Scale and CAIS are developing Humanity&rsquo;s Last Exam, which aims to be the world&rsquo;s most difficult AI test.
<h2 dir="ltr">Your Role</h2>
We're assembling the largest, broadest coalition of experts in history to design questions that test how far AIs are from the human intelligence frontier. If there is a question that would genuinely impress you if an AI could solve it, we&rsquo;d like to hear it from you!
<ul>
<li dir="ltr" aria-level="1">
If one or more of your questions is accepted, you will be offered optional co-authorship of the resulting paper. We have already received questions from researchers from MIT, UC Berkeley, Stanford, and more. The more questions accepted, the higher your name will appear.
</li>
<li dir="ltr" aria-level="1">
The top 50 questions will earn $5000 each.
</li>
<li dir="ltr" aria-level="1">
The next top 500 questions will earn $500 each.
</li>
</ul>
Prizes may be awarded on question quality or question novelty compared to other questions. People who have already submitted questions prior to this announcement are also eligible for these prizes. A small set of questions will be kept private to catch if an AI is memorizing answers to public questions, but prizes can and co-authorship can be awarded to people who have their questions kept part of the private set.
<h2 dir="ltr">Submission Guidelines</h2>
<ul>
<li dir="ltr" aria-level="1">
Challenge Level: Questions should be difficult for non-experts and not easily answerable via a quick online search. Avoid trick questions. Frontier AI systems are very good at answering even masters-level questions. It&rsquo;s strongly encouraged that question-writers have 5+ years of experience in a technical industry job (e.g., SpaceX, Boston Dynamics, Siemens, etc) or are a PhD student or above in academic training. In preparation for Humanity&rsquo;s Last Exam, we found questions written by undergraduates tend to be too easy for the models. As a rule of thumb, if a randomly selected undergraduate can understand what is being asked, it is likely too easy for the frontier LLMs of today and tomorrow.
</li>
<li dir="ltr" aria-level="1">
Objectivity: Answers should be accepted by other experts in the field and free from personal taste, ambiguity, or subjectivity. Provide all necessary context and definitions within the question. Use standard, unambiguous jargon and notation.
</li>
<li dir="ltr" aria-level="1">
Originality: Questions must be your own work and not copied from others.
</li>
<li dir="ltr" aria-level="1">
Confidentiality: Questions and answers should not be publicly available. You may use questions from past exams you've given if they're not accessible to the public.
</li>
<li dir="ltr" aria-level="1">
Weaponization Restrictions: Do not submit questions related to chemical, biological, radiological, nuclear, cyberweapons, or virology.
</li>
</ul>
Terms and conditions <a href="https://www.safe.ai/competition-terms">here</a>.
Deadline: November 1, 2024
For a detailed list of instructions and example questions, please visit <a href="https://agi.safe.ai/submit">agi.safe.ai/submit</a>.
&nbsp;
<hr>
&nbsp;
Be part of a project that could redefine the future of artificial intelligence. Your expertise could help create the ultimate challenge for AI systems worldwide!
This prize pool is sponsored by the Scale AI. We cannot give awards to teams on US terrorist lists or those subject to sanctions. Sponsor may confirm the legality of sending prize money to winners who are residents of countries outside of the United States. Winners will be emailed prior to the paper publication, and Scale AI and Center for AI Safety organizers will judge the question submissions. Only authors on awarded questions are winners. All decisions of judges are final. The legality of accepting the prize in his or her country is the responsibility of the winners. All taxes are the responsibility of the winners. Employees or current contractors of Scale AI and contest organizers are not eligible to win prizes. Entrants must be over the age of 18. By entering the contest, entrants agree to the Terms &amp; Conditions. Entrants agree that Scale AI and Center for AI Safety shall not be liable to entrants for any type of damages that arise out of or are related to the contest and/or the prizes. By submitting an entry, entrant represents and warrants that, consistent with the terms of the Terms and Conditions: (a) the entry is entrant&rsquo;s original work; (b) entrant owns any copyright applicable to the entry; (c) the entry does not violate, in whole or in part, any existing copyright, trademark, patent or any other intellectual property right of any other person, organization or entity; (d) entrant has confirmed and is unaware of any contractual obligations entrant has which may be inconsistent with these Terms and Conditions and the rights entrant is required to have in the entry, including but not limited to any prohibitions, obligations or limitations arising from any current or former employment arrangement entrant may have; (e) entrant is not disclosing the confidential, trade secret or proprietary information of any other person or entity, including any obligation entrant may have in connection arising from any current or former employment, without authorization or a license; and (f) entrant has full power and all legal rights to submit an entry in full compliance with these Terms and Conditions.

Scale_CAIS.png

Scale AI and CAIS are excited to announce the launch of Humanity's Last Exam: the toughest open-source LLM benchmark.

Submit Your Toughest Questions for Humanity's Last Exam

Scale is proud to be a <a href="https://ai.meta.com/blog/meta-llama-3-1/" target="_blank" rel="noopener">Llama 3.1 Launch Partner</a>! Llama 3.1 405B is the largest openly available foundation model with capabilities that rival the best closed-source models. <a href="https://scale.com/generative-ai-data-engine">Scale Data Engine</a> provided a large amount of data for supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) to help make Llama 3.1 the leading open-source LLM. Now, Meta and Scale also partnered to help businesses customize, evaluate, and deploy Llama 3.1 405B for enterprise use cases using <a href="https://scale.com/genai-platform">Scale GenAI Platform</a>.
Only 10% of enterprises have GenAI in production, caused by a lack of trust in the performance and reliability of models. For enterprises to bridge this trust gap, they need to systematically <a href="https://scale.com/evaluation/enterprise">evaluate, improve, and monitor GenAI systems</a> for performance, safety, and reliability. Unbiased evaluations of LLMs for enterprise tasks are difficult, as there are no widely accepted benchmarks. So, we have built a private set of evaluations using Scale GenAI Platform to analyze the performance of Llama 3.1 on these enterprise use cases. These evaluations include enterprise-style chatbot prompts for financial services, legal, and edTech.
In addition to unbiased benchmarks, enterprises also need a platform to manage evaluations of LLMs. Scale GenAI Platform is the perfect enterprise testing ground for closed- and open-source models as it has built-in evaluation capabilities that enable users to easily compare results across different models, along with application development and deployment functionality.
<h3>The Setup</h3>
We evaluated the performance of Llama 3.1 405B across thousands of prompts and compared its performance to the best closed and open-source models across several enterprise domains:
<ol>
<li dir="ltr" aria-level="1">
Enterprise-style chatbot prompts covering legal, edTech, and financial services use cases. We grouped prompts into two different categories: short (1 - 2 sentences) and complex (3 - 5 sentences).
</li>
<li dir="ltr" aria-level="1">
Synthesis from content&nbsp;including legal, financial services, biology, and history domains
</li>
</ol>
<h3>Datasets</h3>
To start, we developed custom knowledge bases comprised of finance and legal textbooks, as well as other source materials that were relevant to each domain, such as legal codes. These datasets consisted of what would be considered general knowledge for each domain.
These knowledge bases were then imported into Scale GenAI Platform to generate evaluation datasets for each of the domains. For each domain, we generated chatbot-style datasets that simulated lawyer queries to a legal chatbot, financial services employee queries to a finance chatbot, and student queries to an edTech chatbot. Each model was also provided relevant context (chunks of text) from a data source on which it should ground its answer.&nbsp;
For real-world enterprise engagements, we use proprietary source documents and relevant industry data to generate custom evaluation datasets. For enterprise engagements, we would often include a gold standard &ldquo;expected output&rdquo; for each query which would then be used to evaluate model accuracy. For the purposes of our evaluations of Llama 3.1, we focused on head-to-head model comparisons only.
<h3>Performing the Evaluations</h3>
We then relied on human experts to evaluate models head-to-head in Scale GenAI Platform and created a weighted average score across multiple factors, such as which model provided the most complete, accurate, or relevant answer. The table below summarizes this rubric and the criteria used to determine a superior response.&nbsp;
<div dir="ltr" align="left">
<table><colgroup><col width="155"><col width="469"></colgroup>
<tbody>
<tr>
<td>
Completeness
</td>
<td>
Does the response fully answer all explicit aspects of the prompt? Does the response omit any essential information?
</td>
</tr>
<tr>
<td>
Accuracy
</td>
<td>
Which response contains the most accurate and reliable information, and which aligns with established facts or evidence? Established facts can be from common knowledge or verified using the provided context.
</td>
</tr>
<tr>
<td>
Depth
</td>
<td>
Which response provides a higher level of detail, insight, and nuance?
</td>
</tr>
<tr>
<td>
Relevance
</td>
<td>
Which response provides the most useful supporting information and claims in answering the main question or prompt? The supporting information logically defends or clearly illustrates the key points and the central claims made in the response.
</td>
</tr>
<tr>
<td>
Clarity
</td>
<td>
Which response is more clearly worded and understandable?
</td>
</tr>
<tr>
<td>
Formatting
</td>
<td>
Which response organizes the information in a way that supports the key points with clarity and brevity?
</td>
</tr>
</tbody>
</table>
</div>
We leveraged Scale GenAI Platform to perform these evaluations with human experts across thousands of examples to measure model performance using our enterpise rubrics. Expert evaluators were presented with the prompts and the response from two models and were asked to identify which model performed better in the different categories of the evaluation rubric.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=4eb9aea8b1fe1b38af4e9f9d4c67d7cd.png&amp;f=webp&amp;q=75" alt="" width="1600" height="799">Evaluation interface in Scale GenAI Platform
In addition to the prompt and individual model responses, human raters had access to the context used to generate the query. This context is useful for determining the accuracy of model responses and maps closely to how these models will perform at summarizing source material as part of a retrieval augmented generation (RAG) workflow.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=34a528e41bfb9877687518dab4705455.png&amp;f=webp&amp;q=75" alt="Example of the context provided for each evaluation within Scale GenAI Platform" width="1771" height="1411">
Example of the context provided for each evaluation within Scale GenAI Platform
<h3>The Results</h3>
Our evaluation datasets included thousands of queries for domains including financial services, legal, and edTech. After completing the evaluations, we aggregated the responses and generated a weighted average score across the rubric criteria and ranked models for performance in each query and, more broadly, for each evaluation dataset. We implemented controls for <a href="https://scale.com/blog/irr">inter-rater reliability</a> along with audits to ensure that we had reliable scores for each task and evaluation set.
From our evaluations, Llama 3.1 rivals the performance of many closed-source models and outperforms all other open-source models. Llama 3.1 performs particularly well in the categories of completeness, accuracy, and depth. Demonstrated common knowledge from the model is not as extensive as leading closed-source models but is quite close. That being said, Llama 3.1 is very good at summarizing content from the provided context, which is especially useful for enterprise workflows that rely on retrieval augmented generation (RAG).&nbsp;
Let&rsquo;s now examine a couple of detailed evaluation examples to demonstrate how Llama 3.1 405B stacks up.
<h4 dir="ltr">Llama 3.1 405B vs. Leading Closed-Source Models</h4>
In the example below, there is a legal domain question (&ldquo;User Prompt&rdquo;) followed by the model responses from Llama 3.1 405B on the left and a leading closed-source model on the right. Llama 3.1 performs favorably in accuracy and completeness compared to the closed-source model, as noted by the human preference rankings of &ldquo;neutral&rdquo; (meaning there is an equal preference for the responses from each model) for the categories of accuracy, completeness, and depth.
Qualitatively, Llama 3.1 405B is able to provide a clear, well-structured, and thorough response to the query. The model accesses the context (&ldquo;Reference Material&rdquo;) and synthesizes its answers in reference to the most relevant portions. For example, the model response cites the specific rule 4.10.3: &ldquo;This regime is based on Listing Rule 4.10.3, which makes it a requirement for listed entities to comply with the recommendations or provide an explanation for non-compliance.&rdquo; Other instances of these references are highlighted in the image. Because the closed source model also references this context well, the evaluation earned a neutral rating from human raters.&nbsp;
The ability to reference context and provide clear and succinct summaries is critical for not just legal use cases, but many enterprise use cases in any industry. Retrieval Augmented Generation (RAG) workflows rely on completion models like Llama 3.1 405B to be able to reliably summarize the relevant context with accuracy.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=e64027bca5d0ec5d34c52432b4c40b3d.png&amp;f=webp&amp;q=75" alt="" width="1600" height="1518">
&nbsp;
<h4 dir="ltr">Llama 3.1 405B vs. Leading Open-Source Models</h4>
In the finance example below, we evaluate responses from Llama 3.1 405B on the left and&nbsp; Llama 3 70B on the right. Llama 3.1 405B outperforms Llama 3 70B (as well as all other open-source models) in accuracy, completeness, and depth per our human evaluators.
Llama 3 70B is already quite good at many tasks, especially for a smaller model (see rankings on the <a href="https://scale.com/leaderboard">SEAL Leaderboard</a>, where this model outperforms many larger closed-source models). However, Llama 3.1 405B&rsquo;s responses stand out in a few ways. The response is more complete and robust, often including additional steps that Llama 3 70B does not. It also is more useful, for example by breaking down the cost-of-capital calculation and separating it into bullets, while Llama 3 70B only includes some of these details tucked away in a paragraph. Better formatting and more complete explanations make it much easier for users to quickly understand core concepts.&nbsp;
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=3c101283c062cb16d749ae6003e985cb.png&amp;f=webp&amp;q=75" alt="" width="1491" height="1600">
<h3>Conclusion</h3>
These are just a couple of examples out of the thousands of responses that we evaluated where Llama 3.1 stood out as the clear leader of open-source models for enterprise applications and&nbsp; having comparable performance to leading closed-source models.
While we focused on legal, financial services, and edTech in our evaluations, similar performance can be expected across a wide range of other enterprise domains including telecommunications, insurance, customer service, and more. As Llama 3.1 is open-source, it can be customized and leveraged for enterprise tasks for a lower cost of entry compared to most closed-source models.&nbsp;
With the evaluation capabilities in Scale GenAI Platform, your enterprise can get an unbiased sense of how leading LLMs such as Llama 3.1 perform for your specific use case. You can further customize the model via retrieval augmented generation (RAG) and fine-tuning, and deploy these models to production, including in your own virtual private cloud (VPC).To see how well Llama 3.1 performs for your unique enterprise use case using <a href="https://scale.com/genai-platform">Scale GenAI Platform</a>, book a demo below today.

scale_meta_blog_cover.png

Get started with Llama 3.1

Meta looked to Scale to evaluate the enterprise capabilities of Llama 3.1 405B with Scale GenAI Platform.

Meta and Scale Partner on Enterprise Adoption of Llama 3.1

Meta and Scale Partner to Drive Enterprise Adoption of Llama 3.1 405B Using Scale GenAI Platform

Today, we are excited to announce a multi-year strategic partnership with Amazon Web Services (AWS) to accelerate enterprise and public sector generative AI adoption. With an exponentially growing number of enterprise and public sector organizations looking to incorporate generative AI (GenAI) to augment their workforce, challenges remain in successfully deploying this technology in production.&nbsp;
In our experience, customer GenAI initiatives often stall after proof of concept because they do not trust and are unable to customize GenAI models with their proprietary data. That is why we are excited to partner with AWS to be their first model customization and evaluation partner to accelerate the GenAI journey. Through this collaboration, AWS customers will be able to leverage the&nbsp;<a href="https://aws.amazon.com/marketplace/pp/prodview-3odiiiv22ydjc?sr=0-3&amp;ref_=beagle&amp;applicationId=AWSMPContessa">Scale GenAI Platform</a> with the AWS Generative AI Innovation Center or on the AWS Marketplace to systematically evaluate their GenAI apps and leverage those insights to further customize, configure, and fine-tune the underlying models &ndash; all on top of Amazon Bedrock.&nbsp;
In addition to evaluating and fine-tuning models, Public Sector customers can access <a href="https://aws.amazon.com/marketplace/pp/prodview-e7onk7j7ju6cw?sr=0-1&amp;ref_=beagle&amp;applicationId=AWSMPContessa">Scale Donovan</a> through the AWS Marketplace. Donovan is an AI platform built for government organizations with federal security requirements in mind. Donovan uncovers insights and expedites traditional workflows to drive mission success. Public sector teams can leverage existing AWS infrastructure services to host Donovan and access leading GenAI models on secure networks. Our commercial customers have realized increased worker productivity by implementing AI in production systems, and with Donovan, the public sector can also achieve better mission outcomes.
To learn more about the partnership, visit our Scale&nbsp;<a href="https://aws.amazon.com/marketplace/seller-profile?id=seller-j2pcb7tbmx6du">AWS partner page</a> or engage with the <a href="https://aws.amazon.com/ai/generative-ai/innovation-center/">AWS Generative AI Innovation Center</a> which pairs you with AWS strategy and science experts.

scale-aws.png

Amazon Web Services and Scale partner to accelerate enterprise and public sector Generative AI adoption. 

AWS + Scale Partner to Accelerate Generative AI Adoption

AWS + Scale Partner to Bring Generative AI to Enterprises and Public Sector Customers

All of our external success&mdash;powering breakthroughs in L4 autonomy, partnering with OpenAI on RLHF going back to GPT-2, supporting the DoD and every major AI lab, and the recent $1bn financing transaction&mdash;all of it is downstream from us hiring the best people for the job. Talent is our #1 input metric.
Because of this, I spend a lot of my time on recruiting. I either personally interview every hire or sign off on every candidate packet. It&rsquo;s the thing I spend the plurality of my time on, easily. But everyone can and should contribute to this effort. There are almost a thousand of us now, and it takes a lot to hire quickly while maintaining, and continuing to raise, our bar for quality.
That&rsquo;s why this is the time to codify a hiring principle that I consider crucial to our success:&nbsp;
Scale is a meritocracy, and we must always remain one.
Hiring on merit will be a permanent policy at Scale.&nbsp;
It&rsquo;s a big deal whenever we invite someone to join our mission, and those decisions have never been swayed by orthodoxy or virtue signaling or whatever the current thing is. I think of our guiding principle as MEI: merit, excellence, and intelligence.
That means we hire only the best person for the job, we seek out and demand excellence, and we unapologetically prefer people who are very smart.
We treat everyone as an individual. We do not unfairly stereotype, tokenize, or otherwise treat anyone as a member of a demographic group rather than as an individual.&nbsp;
We believe that people should be judged by the content of their character &mdash; and, as colleagues, be additionally judged by their talent, skills, and work ethic.
There is a mistaken belief that meritocracy somehow conflicts with diversity. I strongly disagree. No group has a monopoly on excellence. A hiring process based on merit will naturally yield a variety of backgrounds, perspectives, and ideas. Achieving this requires casting a wide net for talent and then objectively selecting the best, without bias in any direction. We will not pick winners and losers based on someone being the &ldquo;right&rdquo; or &ldquo;wrong&rdquo; race, gender, and so on. It should be needless to say, and yet it needs saying: doing so would be racist and sexist, not to mention illegal.
Upholding meritocracy is good for business and is the right thing to do. This approach not only results in the strongest possible team, but also ensures we&rsquo;re treating our colleagues with fairness and respect.
As a result, everyone who joins Scale can be confident that they were chosen for their outstanding talent, not any other reasons.
MEI has gotten us to where we are today. And it&rsquo;s the same thing that&rsquo;ll get us where we&rsquo;re going, as we embark on our next chapter focusing on data abundance, frontier data, and reliable measurement to accelerate the development and adoption of AI models.
&nbsp;
&nbsp;

Cover.png

Scale is a meritocracy, and we must always remain one.

Meritocracy at Scale

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

custom-fonts

homepage-event

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

summit-2023-speakers

customers

Display order on the page /customers. Higher values show first

summit-2023-agenda

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

collapsible-sections

research

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

leaderboards

Text for a short description like: Deprecated (as of January 2025)

Keyword for search. Add multiple keyword with comma. Example: hle, hleto, vista, enigma

Blog