Methodology

Challenging LLMs at the frontier of human knowledge

Humanity's Last Exam

Humanity's Last Exam (Preview)

Humanity's Last Exam (Text Only)

Models evaluated on text-only HLE questions

Humanity's Last Exam Text Only (Preview)

Evaluate model honesty when pressured to lie

MASK

Evaluating model performance on complex, multi-step reasoning tasks

EnigmaEval

Assessing models across diverse, interdisciplinary challenges

MultiChallenge

Vision-Language Understanding benchmark for multimodal models

VISTA

Evaluating AI agents' ability to use enterprise tools effectively

Agentic Tool Use (Enterprise)

Assessing chatbots' proficiency in leveraging external tools

Agentic Tool Use (Chat)

Assessing models' ability to understand and generate programming code

Coding

Assessing performance on Arabic language understanding and generation

Arabic

Measuring capabilities in Korean language processing and comprehension

Korean

Testing models' proficiency in Japanese language tasks and cultural nuances

Japanese

Evaluating Spanish language skills across various linguistic challenges

Spanish

Evaluating language models' proficiency in Chinese language tasks

Chinese

Previously used for evaluating mathematical problem-solving capabilities

Math

Former benchmark for assessing models' ability to follow complex instructions

Instruction Following

Retired test for measuring models' resilience against adversarial inputs

Adversarial Robustness

<h2>Introduction</h2>
<p>The Precise Instruction Following Prompts Dataset is composed of 1,054 instruction following prompts aimed at assessing the ability of AI models to interpret and execute detailed commands, focusing on precision and specificity.</p>
<p>A popular method for assessing LLMs on instruction following tasks is the IFEval benchmark, which focuses on evaluating LLMs using prompts containing programmatically verifiable instructions. However, scenarios in this benchmark are limited due to the requirement of being automatically evaluable. Additionally, similar to other open source benchmarks, IFEval is prone to overfitting.</p>
<p>To address these limitations, we built the Scale AI Precise Instruction Following Prompts Dataset. This is a set of private instruction-following prompts, intended to be paired with human evaluations. This dataset includes 1,054 instruction following prompts grouped in 9 categories, including &ldquo;act as if&rdquo;, content creation and brainstorming, and covering real applications and use cases for instructions following tasks. It was generated by a diverse group of over 40 human annotators and developed through a five-step process to ensure the final prompts tested the model&rsquo;s capability to understand and execute instructions with specificity. The ultimate intent is to run human evaluations on models&rsquo; responses to this prompt set.</p>
<h2>Dataset Description</h2>
<p>The dataset comprises 1,054 prompts designed for single-turn instruction following, intended to evaluate nuanced directive comprehension by the model. It tests the model's ability to execute complex instructions with clarity and specificity.</p>
<p>The construction of this dataset posed challenges in maintaining prompt diversity and specificity, addressed by a review process to ensure each prompt's uniqueness. The dataset covers a broad spectrum of 9 categories, ensuring diversity in instruction-following tasks</p>
<table class="w-full caption-bottom text-sm">
<thead class="[&amp;_th:first-child]:rounded-[8px_0_0_8px] [&amp;_th:first-child]:pl-3 [&amp;_th:last-child]:rounded-[0_8px_8px_0] [&amp;_th:last-child]:pr-3 [&amp;_th]:bg-[rgba(18,18,18,0.6)] [&amp;_tr]:border-b-0">
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors">
<th class="h-12 px-1.5 text-left align-middle font-medium md:px-2">
<div class="">Category</div>
</th>
<th class="h-12 px-1.5 text-left align-middle font-medium md:px-2">
<div class="">Definition</div>
</th>
<th class="h-12 px-1.5 text-left align-middle font-medium md:px-2">
<div class=""># of prompts</div>
</th>
</tr>
</thead>
<tbody class="[&amp;_td:first-child]:pl-3 [&amp;_td:last-child]:pr-3 [&amp;_tr:last-child]:border-0">
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Generation - General Text Creation</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Tasks involve creating original content such as text messages, recipes, jokes, and essays.</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class="">385</div>
</td>
</tr>
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Generation - Content Creation</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Subdivided into poetry, narrative fiction, social media posts and other.</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class="">232</div>
</td>
</tr>
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Generation - "Act as if&hellip;"</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Tasks require responses from specific personas, enhancing the AI's adaptability and creative output.</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class="">89</div>
</td>
</tr>
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Brainstorming - Short answers</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Output a short list within 3-5 items, explicitly asking for a concise enumeration of ideas or options.</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class="">97</div>
</td>
</tr>
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Brainstorming - Long answers</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Require a longer list within 15-20 items, focusing on breadth and inclusivity of ideas or options, with each item being succinct.</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class="">96</div>
</td>
</tr>
<tr class="hover:bg-muted/50 data-[state=selected]:bg-muted border-b border-white/5 transition-colors h-14 font-inter font-medium" data-state="false">
<td class="px-1.5 align-middle md:p-2">
<div class="">Brainstorming - With a format</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div class=" ">Prompts specify the output formatting, such as dashes, bullet points, or enumeration, guiding the structure of the response.</div>
</td>
<td class="px-1.5 align-middle md:p-2">
<div>97</div>
</td>
</tr>
</tbody>
</table>
<p>[instruction-following-data-1]</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=0a1d3dbd07f8535b6b53e55d53af13db.png&amp;f=webp&amp;q=75">Breakdown of Categories (Percentage)<br><br></p>
<h2>Prompt Dataset Construction</h2>
<p>We assembled a diverse team of annotators with backgrounds in linguistics, cognitive psychology, and specific fields such as education and STEM, whom we think are best equipped for this task. Their diverse background was meant to test models' ability to understand and execute instructions across different fields.</p>
<p>To construct the dataset, we followed these steps:</p>
<ol>
<li>Content creation control: The annotators were barred from using public resources or LLMs in order to ensure the production of original human-generated prompts.</li>
<li>Initial attempts: The team of human annotators generated a set of more than 2,500 prompts, aimed at covering the 9 categories of instruction-following scope.</li>
<li>Full Set Review Process: The full initial set underwent multiple review stages, including qualitative analysis, automated grammar checks and final review by trusted reviewers. This</li>
<li>refined the set down to 1.5k prompts.<br>Final Audit (10% sample): An internal team of independent auditors conducted a final quality review on random samples of tasks; this resulted in a final review process, with the removal of any task scoring below 4-5 likert ratings, resulting in 1,054 finalized prompt-response pairs.</li>
</ol>
<p>This dataset is designed for research into LLMs' ability to follow detailed instructions, providing a resource for benchmarking and improvement.</p>
<h2>Quality Controls for Prompts</h2>
<p>Based on the evaluation of prompt effectiveness, we formulated guidelines for the annotators, emphasizing clarity, complexity, and specificity. A high-quality prompt includes distinct, non-repetitive elements that direct the model to a precise outcome, eliminating the possibility of vague or generic responses. Effective prompts define clear goals with specific conditions or constraints, challenging the model to employ deep reasoning and problem-solving skills.<br><br>In contrast, less effective prompts lack the specificity or challenge necessary to extend the model&rsquo;s capabilities beyond basic tasks. Our screening process therefore excludes prompts that are easily solvable via simple internet searches or do not require sophisticated AI responses.<br><br>To maintain quality we implemented a multi-stage review pipeline: each prompt underwent 2 expert reviews to ensure adherence to instructions. An internal team of independent auditors conducted a final quality review, correcting or discarding low-quality entries.</p>
<h2>Evaluation Taxonomy</h2>
<p>To capture a nuanced assessment, we created an evaluation taxonomy specific to precise instruction following tasks. Each model response was evaluated across a set of stand-alone criteria, covering each of the use cases, and side-by-side with another model response to measure preference ranking on a 7-point likert scale.</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=bce06369621e1c6420e005690c240a99.png&amp;f=webp&amp;q=75"></p>
<p>The above dimensions are broken down into 12 criteria that are rated with a Yes or No score.<br><br>After the stand-alone evaluation, responses are compared side-by-side using a Likert scale. This comparative assessment helps in identifying the preferable model response based on a detailed justification tied to the evaluation criteria.</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=b242ed35cfe283b77572fd2a646bc43b.png&amp;f=webp&amp;q=75"></p>
<h2>Evaluation Methodology</h2>
<p>Each model is paired with every other model at least 50 times, and each pairing receives a randomly chosen prompt from the set of 1,054 prompts described above.<br><br>Each evaluation tasks consists of the following:</p>
<ol>
<li>Two models generate the responses for a prompt</li>
<li>Annotators provide a point-wise evaluation of each response</li>
<li>Annotators express their preference between the two scores on a 7-point likert scale</li>
</ol>
<p><br>To ensure thoroughness and reliability in the evaluation process, each task was executed in parallel 3 times by different human annotators. Then, the ratings were then reviewed in two stages: an initial review layer and a final review layer. The figure below provides an overview of the evaluation pipeline design. After finalizing the tasks, a team of internal independent auditors randomly selected and reviewed 10% of the tasks for quality control.</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=d8dd301f90348a8577382e4038caaef8.png&amp;f=webp&amp;q=75"></p>
<p>Evaluation Methodology - Pipeline Design</p>
<h2>Acknowledgments</h2>
<p>This project was made possible by the dedicated efforts of a team of expert annotators. We extend our gratitude to everyone involved in the development and refinement of the dataset and the verification methodology.<br><br>Scale AI Team: Ernesto Hernandez*, Mike Lunati*, Dean Lee, Cristina Menghini, Diego Mares, Daniel Berrios, William Qian, Kenneth Murphy, Summer Yue, Darwin Hsu, David Guevara, Krill Chugunov</p>

Alibaba (Qwen)

Amazon (Nova)

Anthropic

Cohere

Databricks

DeepSeek

Google DeepMind

Meta (Llama)

Microsoft (Phi)

Mistral

OpenAI

Scale’s SEAL Instruction-Following Leaderboard assesses top models’ ability to interpret and execute detailed commands, focusing on precision and specificity.

Instruction Following

Performance Comparison