Methodology

Challenging LLMs at the frontier of human knowledge

Humanity's Last Exam

Humanity's Last Exam (Preview)

Humanity's Last Exam (Text Only)

Models evaluated on text-only HLE questions

Humanity's Last Exam Text Only (Preview)

Evaluate model honesty when pressured to lie

MASK

Evaluating model performance on complex, multi-step reasoning tasks

EnigmaEval

Assessing models across diverse, interdisciplinary challenges

MultiChallenge

Vision-Language Understanding benchmark for multimodal models

VISTA

Evaluating AI agents' ability to use enterprise tools effectively

Agentic Tool Use (Enterprise)

Assessing chatbots' proficiency in leveraging external tools

Agentic Tool Use (Chat)

Assessing models' ability to understand and generate programming code

Coding

Assessing performance on Arabic language understanding and generation

Arabic

Measuring capabilities in Korean language processing and comprehension

Korean

Testing models' proficiency in Japanese language tasks and cultural nuances

Japanese

Evaluating Spanish language skills across various linguistic challenges

Spanish

Evaluating language models' proficiency in Chinese language tasks

Chinese

Previously used for evaluating mathematical problem-solving capabilities

Math

Former benchmark for assessing models' ability to follow complex instructions

Instruction Following

Retired test for measuring models' resilience against adversarial inputs

Adversarial Robustness

<h2>Introduction</h2>
<p>The Scale AI Multilingual Prompts Dataset, composed of 1,000 prompts per language, is tailored to enhance models&rsquo; interaction capabilities across multiple languages. This dataset specifically aims to refine chatbots' proficiency in engaging with Spanish users from Spain, Mexico and the rest of Latin America, reflecting complexity of global communication.</p>
<h2>Dataset Description</h2>
<p>This dataset introduces single-turn prompts across a diversified range of scenarios, aiming to evaluate and improve models&rsquo; responses in both general and culturally nuanced conversations.</p>
<table>
<thead>
<tr>
<th>
<div>Category</div>
</th>
<th>
<div>Definition</div>
</th>
</tr>
</thead>
<tbody>
<tr data-state="false">
<td>
<div>Educational Support</div>
</td>
<td>
<div>Assist students with academic challenges in an easily understandable manner.</div>
</td>
</tr>
<tr data-state="false">
<td>
<div>Entertainment &amp; Recreation</div>
</td>
<td>
<div>Create engaging experiences with games, trivia, and interactive stories for all ages.</div>
</td>
</tr>
<tr data-state="false">
<td>
<div>Coding &amp; Technical Assistance</div>
</td>
<td>
<div>Provide clear, simple assistance with programming projects.</div>
</td>
</tr>
<tr data-state="false">
<td>
<div>Daily Life Assistance</div>
</td>
<td>
<div>Offer practical advice on everyday tasks like cooking and scheduling.</div>
</td>
</tr>
<tr data-state="false">
<td>
<div>Creative Expression</div>
</td>
<td>
<div>Encourage creativity in arts and writing with inspirational guidance.</div>
</td>
</tr>
<tr data-state="false">
<td>
<div>Idea Development &amp; Content Exploration</div>
</td>
<td>
<div>Help generate ideas and clarify thoughts for innovative minds.</div>
</td>
</tr>
<tr data-state="false">
<td>
<div>Information &amp; Learning</div>
</td>
<td>
<div>Simplify complex topics and support learning new skills across various subjects.</div>
</td>
</tr>
<tr data-state="false">
<td>
<div>Personal &amp; Professional Organization</div>
</td>
<td>
<div>Assist in organizing personal and work lives with actionable advice.</div>
</td>
</tr>
<tr data-state="false">
<td>
<div>Shopping &amp; Consumer Research</div>
</td>
<td>
<div>Provide insights for smart shopping decisions with clear comparisons and recommendations.</div>
</td>
</tr>
<tr data-state="false">
<td>
<div>Writing &amp; Communication</div>
</td>
<td>
<div>Enhance communication skills with tips on various forms of writing.</div>
</td>
</tr>
<tr>
<td>
<div>Travel Assistance</div>
</td>
<td>
<div>Simplify travel planning with straightforward advice on destinations and planning.</div>
</td>
</tr>
<tr>
<td>
<div>Workplace Productivity</div>
</td>
<td>
<div>Offer straightforward tips to streamline work processes and enhance professionalism.</div>
</td>
</tr>
<tr>
<td>
<div>Other</div>
</td>
<td>
<div>Address any additional tasks not specifically listed above, providing flexible support for various needs.</div>
</td>
</tr>
</tbody>
</table>
<p>[spanish-data-1]</p>
<h2>Construction Process</h2>
<p>Development of this dataset followed a structured approach:</p>
<ol>
<li>Original Content Requirement:&nbsp;Unique content generation was enforced, prohibiting the use of existing resources or models.</li>
<li>Initial Drafting:&nbsp;Initial creation exceeded 2,500 prompts to encompass a broad linguistic and cultural scope.</li>
<li>Review Stages:&nbsp;The content underwent qualitative and grammatical assessments.</li>
<li>Final Quality Audit:&nbsp;A final evaluation on a select sample refined the dataset to 1,000 prompts.</li>
</ol>
<p><br>Experts fluent in various languages with cultural knowledge were chosen to contribute, ensuring the dataset's relevance and authenticity.<br><br>Quality was maintained through:</p>
<ol>
<li>Multi-stage Reviews:&nbsp;Ensuring clarity, complexity, and cultural specificity.</li>
<li>Internal Benchmarking:&nbsp;Monitoring and assessing annotator performance.</li>
<li>Final Audits:&nbsp;Revising or removing prompts that did not meet quality standards.</li>
</ol>
<p>&nbsp;</p>
<h2>Evaluation Taxonomy</h2>
<p>To capture a nuanced assessment of both general language understanding and of regionalized styles from Spain, M&eacute;xico and the rest of Latin Am&eacute;rica, we created an evaluation taxonomy specific to evaluating this context. Each model response was evaluated across a set of stand-alone criteria, covering a diverse set of use cases, and side-by-side with another model response to measure preference ranking on a 7-point Likert scale.</p>
<p>Main stand-alone criteria and their sub-categories:</p>
<ol>
<li dir="ltr" aria-level="1">Honesty:
<ol>
<li dir="ltr" aria-level="3">Prompt Understanding - Language Consistency: The response uses the same language, unless explicitly requested.&nbsp;</li>
<li dir="ltr" aria-level="3">Localization: The model understands the context based on the language, it is appropriate for your locale, by aligning with the culture.</li>
</ol>
<ol>
<li dir="ltr" aria-level="3">The response is consistent as all statements are coherent and no sequential statements contradict prior statements.</li>
</ol>
<ol>
<li dir="ltr" aria-level="3">Claim Factuality: The response made false and inaccurate claims that can be fact-checked and refuted.&nbsp;</li>
<li dir="ltr" aria-level="3">Claim Verifiability: The response makes unverifiable claims and presents opinions as facts.</li>
</ol>
</li>
<ol>
<li dir="ltr" aria-level="2">Understanding:&nbsp;</li>
<li dir="ltr" aria-level="2">Consistency:</li>
<li dir="ltr" aria-level="2">Factual Accuracy:</li>
</ol>
<li dir="ltr" aria-level="1">Helpfulness:
<ol>
<li dir="ltr" aria-level="3">Response Relevance:</li>
<ol>
<li dir="ltr" aria-level="4">Point of interest: Assess if all model response content is relevant to address the prompt request.</li>
<li dir="ltr" aria-level="4">Common Pitfall: If the model fulfills all the prompt requests but does more than the prompt requires, the response has no issues in &ldquo;Prompt Adherence&rdquo; and solely issues in &ldquo;Prompt Relevance.</li>
</ol>
<li dir="ltr" aria-level="3">Main Request Adherence:</li>
<ol>
<li dir="ltr" aria-level="4">Point of interest: Evaluate how well the response adheres to the main request specified in the prompt.&nbsp;</li>
<li dir="ltr" aria-level="4">Common Pitfall: Does not include Factual Accuracy of the model response. The model can follow the instructions completely but simultaneously make inaccurate statements.</li>
</ol>
<li dir="ltr" aria-level="3">Constraint Adherence:</li>
<ol>
<li dir="ltr" aria-level="4">Point of interest: Evaluate if the response covers all constraints specified in the prompt.</li>
<li dir="ltr" aria-level="4">Common Pitfall: Incorrectly identify the constraints from other criteria, such as Request Adherence or Writing criteria and penalize this one. Failing the only constraint results in a more severe IF issue than failing 1 out of 5 constraints.</li>
</ol>
</ol>
<ol>
<li dir="ltr" aria-level="3">Mechanics: Are the spelling, grammar, punctuation, and syntax all correct?</li>
<li dir="ltr" aria-level="3">Structural Clarity: Is the response well-organized with clear, logical flow and appropriate formatting?</li>
<li dir="ltr" aria-level="3">Verbosity: Is the response concise as it does not repeat the same arguments and is not overly wordy in its explanation?</li>
<li dir="ltr" aria-level="3">Fluency: The response is easy to understand and sounds natural. There are no weird sentence constructions inherited by languages like English.</li>
<li dir="ltr" aria-level="3">Style and tone: Is the style and tone of the writing suitable for the topic and audience?</li>
</ol>
</li>
<ol>
<li dir="ltr" aria-level="2">Instruction Following:</li>
<li dir="ltr" aria-level="2">Writing:</li>
</ol>
<li dir="ltr" aria-level="1">Harmlessness:
<ol>
<li dir="ltr" aria-level="3">No: The response is free from harmful content and does not facilitate harmful actions.</li>
<li dir="ltr" aria-level="3">Yes: The response contains harmful elements or encourages harmful actions.</li>
</ol>
</li>
<ol>
<li dir="ltr" aria-level="2">Is the response free from harmful content and does it not facilitate harmful action?</li>
</ol>
</ol>
<p>Each dimension&rsquo;s sub-category is rated on whether it complies or not with the sub-category definition (i.e. &lsquo;yes&rsquo; or &lsquo;no&rsquo; binary score).</p>
<p>After the evaluation of stand-alone criteria, responses are compared side-by-side using a Likert scale. This comparative assessment helps in identifying the preferable model response based on a detailed justification tied to the evaluation criteria. We rank the models by the side-by-side elo scores for the leaderboard rankings.</p>
<h2>Evaluation Methodology</h2>
<p>Each model is randomly paired with other models, and each pairing receives a randomly chosen prompt from the set of 1,000 prompts described above.</p>
<p>Each evaluation tasks consists of the following:</p>
<ul>
<li>Two models generate the responses for a prompt</li>
<li>Annotators provide a point-wise evaluation of each response</li>
<li>Annotators express their preference between the two scores on a 7-point Likert scale</li>
</ul>
<p>To ensure thoroughness and reliability in the evaluation process, each task was executed in parallel 3 times by different human annotators. Then, the ratings were then reviewed in two stages: an initial review layer and a final review layer. The figure below provides an overview of the evaluation pipeline design. After finalizing the tasks, a team of internal independent auditors randomly selected and reviewed 10% of the tasks for quality control.</p>
<p>Evaluation Methodology - Pipeline Design<img src="https://img.plasmic.app/img-optimizer/v1/img?src=d8dd301f90348a8577382e4038caaef8.png&amp;f=webp&amp;q=75"></p>
<div class="p-rich_text_section">Leaderboard rankings are determined using&nbsp;<strong data-stringify-type="bold">Rank (Upper Bound)</strong>, which reflects a model&rsquo;s statistical position based on confidence intervals. The ranking process follows these steps:</div>
<ol class="p-rich_text_list p-rich_text_list__ordered p-rich_text_list--nested" data-stringify-type="ordered-list" data-list-tree="true" data-indent="0" data-border="0">
<li data-stringify-indent="0" data-stringify-border="0">Count the number of models that are&nbsp;<strong data-stringify-type="bold">statistically significantly better</strong>&nbsp;than the target model.</li>
<li data-stringify-indent="0" data-stringify-border="0">Add&nbsp;<strong data-stringify-type="bold">1</strong>&nbsp;to this count to determine the model&rsquo;s rank.</li>
</ol>
<div class="p-rich_text_section">A model is considered&nbsp;<strong data-stringify-type="bold">statistically significantly better</strong>&nbsp;than another if its&nbsp;<strong data-stringify-type="bold">lower-bound score (95% confidence interval) is higher</strong>&nbsp;than the other model&rsquo;s&nbsp;<strong data-stringify-type="bold">upper-bound score</strong>.Models receive the&nbsp;<strong data-stringify-type="bold">same rank</strong> when the same number of models are statistically better than each of them. This approach groups models based on statistical significance rather than raw scores, ensuring rankings reflect meaningful performance differences.</div>

Alibaba (Qwen)

Amazon (Nova)

Anthropic

Cohere

Databricks

DeepSeek

Google DeepMind

Meta (Llama)

Microsoft (Phi)

Mistral

OpenAI

Scale’s SEAL Spanish Leaderboard evaluates LLMs’ proficiency and interaction capabilities in Spanish.

Spanish

Performance Comparison