Methodology

Challenging LLMs at the frontier of human knowledge

Humanity's Last Exam

Humanity's Last Exam (Preview)

Humanity's Last Exam (Text Only)

Models evaluated on text-only HLE questions

Humanity's Last Exam Text Only (Preview)

Evaluate model honesty when pressured to lie

MASK

Evaluating model performance on complex, multi-step reasoning tasks

EnigmaEval

Assessing models across diverse, interdisciplinary challenges

MultiChallenge

Vision-Language Understanding benchmark for multimodal models

VISTA

Evaluating AI agents' ability to use enterprise tools effectively

Agentic Tool Use (Enterprise)

Assessing chatbots' proficiency in leveraging external tools

Agentic Tool Use (Chat)

Assessing models' ability to understand and generate programming code

Coding

Assessing performance on Arabic language understanding and generation

Arabic

Measuring capabilities in Korean language processing and comprehension

Korean

Testing models' proficiency in Japanese language tasks and cultural nuances

Japanese

Evaluating Spanish language skills across various linguistic challenges

Spanish

Evaluating language models' proficiency in Chinese language tasks

Chinese

Previously used for evaluating mathematical problem-solving capabilities

Math

Former benchmark for assessing models' ability to follow complex instructions

Instruction Following

Retired test for measuring models' resilience against adversarial inputs

Adversarial Robustness

<h2>Introduction</h2>
<p>The Scale Coding Evaluation consists of 1,000 prompts covering various programming languages, fields, and tasks. This comprehensive dataset spans a wide range of software engineering challenges, including debugging, code optimization, documentation creation, and analysis of complex codebases.</p>
<p>While the general use and understanding of LLMs&rsquo; usage for coding applications has grown, there are limited tools or benchmarks available to compare different models. The most well-known benchmarks include:</p>
<ol>
<li>HumanEval Dataset: 164 handcrafted programming problems testing language comprehension, algorithms, and simple mathematics.</li>
<li>Pass@k Metric: Assesses the likelihood that at least one of the top k generated code samples for a problem passes unit tests, evaluating functional correctness.</li>
<li>MBPP: The Mostly Basic Programming Problems (MBPP) dataset contains 974 programming tasks, designed to be solvable by entry-level programmers, and is designed to measure the ability of LLMs to synthesize short Python programs from natural language descriptions.</li>
<li>SWE-Bench (Software Engineering Benchmark):Tests LLMs' capability to address real-world issues sourced from GitHub issues and pull requests.</li>
<li>LiveCodeBench: A collection of programming puzzles from recent competitive coding competitions designed to benchmark advanced reasoning skills while mitigating effects of data contamination.</li>
</ol>
<p>While these evaluation benchmarks were useful when they first came out, as models started to overfit them, their application has become less valuable. Multiple coding models fine tuned from GPT-4 pre-trained models are scoring above 85 on the Pass@1 metric. Additionally, the methodology via which these models are evaluated against these benchmarks are often non-standardized, lacking a core requirement for comparing scores across tests or over time.</p>
<p>The goal of the Scale AI coding evaluation is to establish a uniform framework for evaluating LLMs&rsquo; coding capabilities.</p>
<h2>Prompts Dataset Description</h2>
<p>The Scale Coding Evaluation provides a standardized assessment framework for LLMs, enabling comparisons across models and identifying their strengths and weaknesses. It currently encompasses a set of use cases across the most requested coding languages.</p>
<table class="w-full caption-bottom text-sm">
<thead class="[&amp;_th:first-child]:rounded-[8px_0_0_8px] [&amp;_th:first-child]:pl-3 [&amp;_th:last-child]:rounded-[0_8px_8px_0] [&amp;_th:last-child]:pr-3 [&amp;_th]:bg-[rgba(18,18,18,0.6)] [&amp;_tr]:border-b-0">
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors">
<th class="h-12 px-1.5 text-left align-middle font-medium md:px-2">
<div class="">Use Case</div>
</th>
<th class="h-12 px-1.5 text-left align-middle font-medium md:px-2">
<div class="">Definition</div>
</th>
<th class="h-12 px-1.5 text-left align-middle font-medium md:px-2">
<div class=""># of prompts</div>
</th>
</tr>
</thead>
<tbody class="[&amp;_td:first-child]:pl-3 [&amp;_td:last-child]:pr-3 [&amp;_tr:last-child]:border-0">
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Generation</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Create new code from a set of specifications or descriptions given in natural language.</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class="">207 (20.7%)</div>
</td>
</tr>
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Fixing</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Identify and correct errors in existing code. For example, debugging, resolving syntax errors and fixing logical mistakes.</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class="">153 (15.3%)</div>
</td>
</tr>
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Understanding</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Explain, interpret or clarify existing code. For example, elucidating how certain code segments work, breaking down complex algorithms.</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class="">144 (14.4%)</div>
</td>
</tr>
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Modification</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Make changes or adjustments to existing code to meet new requirements or conditions. For example, altering functionality, updating or enhancing features.</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class="">102 (10.2%)</div>
</td>
</tr>
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Optimization</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Improve the performance of existing code. For example, enhancing efficiency, reducing resource consumption (like memory or processing time).</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class="">99 (9.9%)</div>
</td>
</tr>
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Learning</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Assist with learning or understanding programming concepts, languages or tools. For example, guidance on best practices, explanation of programming concepts.</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class="">96 (9.6%)</div>
</td>
</tr>
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Translation</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Convert code from one programming language to another with code structures, styles and idioms adapted to the best practices of the target language.</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class="">52 (5.2%)</div>
</td>
</tr>
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Recommendations</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Provide suggestions or advice on coding practices, tools, libraries or frameworks.</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class="">50 (5.0%)</div>
</td>
</tr>
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Commenting</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Add or improve comments in existing code.</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class="">49 (4.9%)</div>
</td>
</tr>
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Testing</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Develop, enhance or fix tests for existing code.</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class="">48 (4.8%)</div>
</td>
</tr>
</tbody>
</table>
<p>[coding-data-1]</p>
<p>Use Cases Distribution</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=529950a149f67a7bb750bccaf70337c4.webp&amp;f=webp&amp;q=75"></p>
<p>Coding Languages Distribution</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=d4d812f33d91a2ef01fa91b15cdb6b51.webp&amp;f=webp&amp;q=75"></p>
<p>The dataset was created by a group of coding annotators screened and selected for the project. They were selected on the basis of software development / programming / data science experience and qualifications, ensuring coverage of all required coding languages.</p>
<p>Dataset constructions steps:</p>
<ol>
<li><strong>Instructions creation.</strong> We generated an instruction document to guide human annotators contributors on how to generate the initial set of prompt-response pairs.<br>Controlled access. Contributors were prohibited from using any publicly available code or questions, including those from StackOverflow or public GitHub repositories. This prevents possible eval set contamination, ensuring the creation of novel problems the models haven&rsquo;t seen in their training sets.</li>
<li><strong>Initial attempts.</strong> Contributors were guided to create initial attempts, an initial prompt-response set, following a predetermined distribution by coding language and programming task.</li>
<li><strong>Quality Control.</strong> We designed a 2-stage pipeline:
<ol style="list-style-type: lower-alpha;">
<li>Based on initial attempts&rsquo; quality, the operators identified a set of trusted reviewers</li>
<li>Trusted reviewers performed multiple review rounds to evaluate the prompt-response pairs accuracy, coherence, and relevance.</li>
<li>In addition to standard reviews, code execution tests on the model answers and independent audits were performed. We filtered down the initial set of 2.5k prompts to 1.5k reviewed prompts-responses.</li>
</ol>
</li>
<li><strong>Independent final audit. </strong>The team commissioned a set of internal independent auditors with a final review of the quality of the prompts and answers. Low quality prompts or answers were filtered out. This final screen reduced the initial set down to 1,000 quality-validated prompt-response pairs.</li>
</ol>
<h2>Evaluation Taxonomy</h2>
<p>To capture a nuanced assessment, we created an evaluation taxonomy specific to coding tasks. Each model response was evaluated across a set of standalone criteria, covering each of the use cases, and side-by-side with another model response to measure preference ranking on a 7-point likert scale:</p>
<ol>
<li><strong>Prompt Adherence:</strong> Does the model strictly adhere to the given prompt and comprehend all its requirements?
<ul>
<li>Ratings: Yes, No</li>
</ul>
</li>
<li><strong>Correctness:</strong>
<ul>
<li>To what extent are the claims in the response truthful and correct? Verifying Correctness often requires external research.</li>
<li>If code was present in the response, does it execute and produce the correct output
<ol style="list-style-type: lower-alpha;">
<li>Evaluators are encouraged to use any means possible to test code (e.g., writing simple programs to test functions and code snippets), however, Code Correctness may not be measured if, for example, the code only functions when embedded inside a large, complex program that is not provided, or if it requires an external file/API dependency that is not provided.</li>
</ol>
</li>
<li>Ratings: Yes, No</li>
</ul>
</li>
<li><strong>Performance / Efficiency:</strong> Does the code execute without any performance concerns?<br>
<ul>
<li>Ratings: Yes, No</li>
</ul>
</li>
<li><strong>Readability / Documentation:</strong> Is the written explanation well-structured and visually organized? Does the response include necessary documentation aiding in code understanding? Is the code readable, employing proper formatting and mnemonic variable and function names?
<ul>
<li>Ratings: Yes, No</li>
</ul>
</li>
<li><strong>Overall Side-by-Side: </strong>Considering the dimensions above relevant to the specific task, how does the overall quality of the two model responses compare?<br>
<ul>
<li>Ratings: Score between 1 to 7, where 1 signifies that model 1&rsquo;s response was much better than model 2&rsquo;s response and 7 signifies the opposite.</li>
</ul>
</li>
</ol>
<p>Expert raters were required to provide further detailed justifications regarding (1) any model issues identified if they selected &ldquo;No&rdquo; in any of the above dimensions, and (2) the side-by-side rating, in an open-text format. Additionally, for all the responses that involve a nontrivial code snippet, we ask the annotators to compile and run the code in our tooling in order to fully catch compilation and runtime issues.</p>
<h2>Evaluation Methodology</h2>
<p>Each model is randomly paired with other models, and each pairing receives a randomly chosen prompt from the set of 1,000 prompts described above.</p>
<p>Each evaluation tasks consists of the following:</p>
<ol>
<li>Two models generate the responses for a prompt</li>
<li>Annotators provide a point-wise evaluation of each response</li>
<li>Annotators express their preference between the two scores on a 7-point likert scale</li>
</ol>
<p>To ensure thoroughness and reliability in the evaluation process, each task was executed in parallel 3 times by different human annotators. Then, the ratings were then reviewed in two stages: an initial review layer and a final review layer. The figure below provides an overview of the evaluation pipeline design. After finalizing the tasks, a team of internal independent auditors randomly selected and reviewed 10% of the tasks for quality control.</p>
<h4>&nbsp;</h4>
<h4><img src="https://img.plasmic.app/img-optimizer/v1/img?src=d8dd301f90348a8577382e4038caaef8.png&amp;f=webp&amp;q=75" alt=""></h4>
<div class="p-rich_text_section">Leaderboard rankings are determined using&nbsp;<strong data-stringify-type="bold">Rank (Upper Bound)</strong>, which reflects a model&rsquo;s statistical position based on confidence intervals. The ranking process follows these steps:</div>
<ol class="p-rich_text_list p-rich_text_list__ordered p-rich_text_list--nested" data-stringify-type="ordered-list" data-list-tree="true" data-indent="0" data-border="0">
<li data-stringify-indent="0" data-stringify-border="0">Count the number of models that are&nbsp;<strong data-stringify-type="bold">statistically significantly better</strong>&nbsp;than the target model.</li>
<li data-stringify-indent="0" data-stringify-border="0">Add&nbsp;<strong data-stringify-type="bold">1</strong>&nbsp;to this count to determine the model&rsquo;s rank.</li>
</ol>
<div class="p-rich_text_section">A model is considered&nbsp;<strong data-stringify-type="bold">statistically significantly better</strong>&nbsp;than another if its&nbsp;<strong data-stringify-type="bold">lower-bound score (95% confidence interval) is higher</strong>&nbsp;than the other model&rsquo;s&nbsp;<strong data-stringify-type="bold">upper-bound score</strong>.Models receive the&nbsp;<strong data-stringify-type="bold">same rank</strong> when the same number of models are statistically better than each of them. This approach groups models based on statistical significance rather than raw scores, ensuring rankings reflect meaningful performance differences.</div>
<h2>Evaluation Methodology - Pipeline Design</h2>
<p>Beyond producing overall ranking, this evaluation methodology enables slicing of the evaluation data by programming languages and use cases, to help highlight models&rsquo; strengths and weaknesses across different areas, to help answer questions like: how does a model perform compared to the reference model on SQL, Java, HTML/CSS, and C++ prompts? How competitive is a particular model in specific functions, from less code-intensive tasks like Understanding and Recommendations, to complex scenarios like Translation?</p>
<h2>Acknowledgments</h2>
<p>We extend our deepest gratitude to the dedicated team of annotators, operators and researchers who made this project possible.</p>
<p>Scale AI team: Dean Lee*, Edwin Pan, Johannes Mols, Mike Lunati, Antony Tokarr, Cristina Menghini, Daniel Berrios, William Qian, Kenneth Murphy, Summer Yue</p>
<h3>Appendix A - Prompt Diversity</h3>
<p>Controlling command diversity is essential for model evaluation because doing so exercises the versatility and comprehensiveness of the target models in handling a wide range of instructions or directives. A model with high command diversity can effectively understand and respond to various types of commands, including different task-specific instructions, requests, prompts for information, or actions to be performed. During prompt collection, frequent combinations like &ldquo;create-program&rdquo; were caught by automated checks and discouraged in favor of different wordings with similar meanings. To analyze command diversity for coding prompts, all code is removed such that only the natural language remains.</p>
<h3><img src="https://img.plasmic.app/img-optimizer/v1/img?src=c686354988213656182401835b86e72f.png&amp;f=webp&amp;q=75"></h3>

Alibaba (Qwen)

Amazon (Nova)

Anthropic

Cohere

Databricks

DeepSeek

Google DeepMind

Meta (Llama)

Microsoft (Phi)

Mistral

OpenAI

Scale’s SEAL Coding Leaderboard evaluates and ranks top LLMs on programming languages, disciplines, and tasks.

Coding

Performance Comparison