Methodology

Challenging LLMs at the frontier of human knowledge

Humanity's Last Exam

Humanity's Last Exam (Preview)

Humanity's Last Exam (Text Only)

Models evaluated on text-only HLE questions

Humanity's Last Exam Text Only (Preview)

Evaluate model honesty when pressured to lie

MASK

Evaluating model performance on complex, multi-step reasoning tasks

EnigmaEval

Assessing models across diverse, interdisciplinary challenges

MultiChallenge

Vision-Language Understanding benchmark for multimodal models

VISTA

Evaluating AI agents' ability to use enterprise tools effectively

Agentic Tool Use (Enterprise)

Assessing chatbots' proficiency in leveraging external tools

Agentic Tool Use (Chat)

Assessing models' ability to understand and generate programming code

Coding

Assessing performance on Arabic language understanding and generation

Arabic

Measuring capabilities in Korean language processing and comprehension

Korean

Testing models' proficiency in Japanese language tasks and cultural nuances

Japanese

Evaluating Spanish language skills across various linguistic challenges

Spanish

Evaluating language models' proficiency in Chinese language tasks

Chinese

Previously used for evaluating mathematical problem-solving capabilities

Math

Former benchmark for assessing models' ability to follow complex instructions

Instruction Following

Retired test for measuring models' resilience against adversarial inputs

Adversarial Robustness

<h2>Introduction</h2>
<p>The Scale AI Multilingual Prompts Dataset, composed of 1,000 prompts per language, is tailored to enhance models&rsquo; interaction capabilities across multiple languages. This dataset specifically aims to refine chatbots' proficiency in engaging with Japanese users, reflecting complexity of Japanese communication.</p>
<h2>Dataset Description</h2>
<p dir="ltr">This dataset introduces single-turn prompts across a diversified range of scenarios, aiming to evaluate and improve models&rsquo; responses in both general and culturally nuanced conversations.</p>
<p dir="ltr">Every prompt is designed to have a clear goal/request/instruction that specifies what needs to be achieved by the response and the objective is relevant to the category assigned to the task. In addition, the complexity of the prompts varies by adding different combinations of the following building blocks: Local Context, Limitations, Role, Audience, Style and Format.</p>
<p dir="ltr">For Limitations, every prompt can define one or several constraints. As for Role, specifying it ensures the model&rsquo;s responses are judged within a specific, well-defined context, which aligns the evaluation criteria with the expected output. This approach provides consistency, improves relevance, and allows for a more targeted assessment of the model's ability to emulate expertise in a given domain.&nbsp;</p>
<p dir="ltr">The evaluation process intentionally focuses on more complex use cases, which differ from the simpler and more straightforward conversations between a regular user and a chatbot. This approach more effectively differentiates models based on their varying underlying capabilities.</p>
<div dir="ltr" align="center">&nbsp;</div>
<table style="width: 100%; height: 516.687px;">
<thead>
<tr style="height: 18.4531px;">
<th style="width: 31.8003%; height: 18.4531px;">
<div><span style="font-weight: 400;">Category</span></div>
</th>
<th style="width: 66.9969%; height: 18.4531px;">
<div><span style="font-weight: 400;">Definition</span></div>
</th>
</tr>
</thead>
<tbody>
<tr style="height: 36.9062px;" data-state="false">
<td style="width: 31.8003%; height: 36.9062px;">Coding &amp; Technical Assistance</td>
<td style="width: 66.9969%; height: 36.9062px;">
<div>Assist with writing, understanding, and fixing code in a simple, clear manner for those working on programming projects.</div>
</td>
</tr>
<tr style="height: 36.9062px;" data-state="false">
<td style="width: 31.8003%; height: 36.9062px;">Creative Expression</td>
<td style="width: 66.9969%; height: 36.9062px;">
<div>Encourage creativity in music, writing, and art by providing easy-to-follow guidance and inspiration for artists.</div>
</td>
</tr>
<tr style="height: 36.9062px;" data-state="false">
<td style="width: 31.8003%; height: 36.9062px;">Daily Life Assistance</td>
<td style="width: 66.9969%; height: 36.9062px;">
<div>Offer advice on everyday tasks like cooking, booking services, and organizing schedules in a straightforward, practical way.</div>
</td>
</tr>
<tr style="height: 36.9062px;" data-state="false">
<td style="width: 31.8003%; height: 36.9062px;">Educational Support</td>
<td style="width: 66.9969%; height: 36.9062px;">
<div>Help students with their studies, homework, and any school-related problems in a way that's easy to understand.</div>
</td>
</tr>
<tr style="height: 36.9062px;" data-state="false">
<td style="width: 31.8003%; height: 36.9062px;">Entertainment &amp; Recreation</td>
<td style="width: 66.9969%; height: 36.9062px;">
<div>Create fun experiences with games, trivia, sports discussions, and interactive stories for people of all ages.</div>
</td>
</tr>
<tr style="height: 36.9062px;" data-state="false">
<td style="width: 31.8003%; height: 36.9062px;">Idea Development &amp; Content Exploration</td>
<td style="width: 66.9969%; height: 36.9062px;">
<div>Help generate new ideas, answer questions, and clarify thoughts in a clear, accessible way for creative and curious minds.</div>
</td>
</tr>
<tr style="height: 36.9062px;" data-state="false">
<td style="width: 31.8003%; height: 36.9062px;">Information &amp; Learning</td>
<td style="width: 66.9969%; height: 36.9062px;">
<div>Provide easy explanations of complex topics, help with learning new skills, and give advice on a wide range of subjects.</div>
</td>
</tr>
<tr style="height: 36.9062px;" data-state="false">
<td style="width: 31.8003%; height: 36.9062px;">Other</td>
<td style="width: 66.9969%; height: 36.9062px;">
<div>Cover any other tasks not listed above, providing a flexible and understandable approach for a variety of needs.</div>
</td>
</tr>
<tr style="height: 36.9062px;" data-state="false">
<td style="width: 31.8003%; height: 36.9062px;">Personal &amp; Professional Organization</td>
<td style="width: 66.9969%; height: 36.9062px;">
<div>Help people plan their personal and work lives, including events, budgets, and projects, with simple, actionable advice.</div>
</td>
</tr>
<tr style="height: 36.9062px;">
<td style="width: 31.8003%; height: 36.9062px;">Shopping &amp; Consumer Research</td>
<td style="width: 66.9969%; height: 36.9062px;">
<div>Guide on making smart shopping choices with clear comparisons, recommendations, and product insights.</div>
</td>
</tr>
<tr style="height: 36.9062px;">
<td style="width: 31.8003%; height: 36.9062px;">Travel Assistance</td>
<td style="width: 66.9969%; height: 36.9062px;">
<div>Make travel planning simple by providing easy-to-understand advice on destinations, planning, and local insights.</div>
</td>
</tr>
<tr style="height: 36.9062px;">
<td style="width: 31.8003%; height: 36.9062px;">Workplace Productivity</td>
<td style="width: 66.9969%; height: 36.9062px;">
<div>Help professionals streamline their work with tips on writing, presenting, organizing, and more, in a straightforward manner.</div>
</td>
</tr>
<tr style="height: 55.3594px;">
<td style="width: 31.8003%; height: 55.3594px;">Writing &amp; Communication</td>
<td style="width: 66.9969%; height: 55.3594px;">
<div>Improve communication skills with tips on writing everything from social media posts to professional emails, in a clear and engaging way.</div>
</td>
</tr>
</tbody>
</table>
<p>[japanese-data-1]</p>
<h2>Construction Process</h2>
<p dir="ltr">Development of this dataset followed a structured approach:</p>
<ul>
<li dir="ltr" aria-level="1">Original Content Requirement: Unique content generation was enforced, prohibiting the use of existing resources or models.</li>
<li dir="ltr" aria-level="1">Review Stages: The content underwent qualitative and grammatical assessments.</li>
<li dir="ltr" aria-level="1">Final Quality Audit: A final evaluation on a select sample refined the dataset to 1,000 prompts.</li>
</ul>
<p dir="ltr">Experts fluent in various languages with cultural knowledge were chosen to contribute, ensuring the dataset's relevance and authenticity.</p>
<p dir="ltr">Quality was maintained through:</p>
<ul>
<li dir="ltr" aria-level="1">Multi-stage Reviews: Ensuring clarity, complexity, and cultural specificity.</li>
<li dir="ltr" aria-level="1">Internal Benchmarking: Monitoring and assessing annotator performance.</li>
<li dir="ltr" aria-level="1">Final Audits: Revising or removing prompts that did not meet quality standards.</li>
</ul>
<h2>Evaluation Taxonomy</h2>
<p dir="ltr">To capture a nuanced assessment of both general language understanding and of regionalized styles from Japan, we created an evaluation taxonomy specific to evaluating this context. Each model response was evaluated across a set of stand-alone criteria, covering a diverse set of use cases, and side-by-side with another model response to measure preference ranking on a 7-point Likert scale.</p>
<p dir="ltr">Main stand-alone criteria and their sub-categories:</p>
<ol>
<li dir="ltr" aria-level="1">Honesty:
<ol>
<li dir="ltr" aria-level="2">Understanding:&nbsp;
<ol>
<li dir="ltr" aria-level="3">Prompt Understanding - Language Consistency: The response uses the same language, unless explicitly requested.&nbsp;</li>
<li dir="ltr" aria-level="3">Localization: The model understands the context based on the language, it is appropriate for your locale, by aligning with the culture.</li>
</ol>
</li>
<li dir="ltr" aria-level="2">Consistency:
<ol>
<li dir="ltr" aria-level="3">The response is consistent as all statements are coherent and no sequential statements contradict prior statements.</li>
</ol>
</li>
<li dir="ltr" aria-level="2">Factual Accuracy:
<ol>
<li dir="ltr" aria-level="3">Claim Factuality: The response made true and accurate claims that can be fact-checked and supported.&nbsp;</li>
<li dir="ltr" aria-level="3">Claim Verifiability: The response makes verifiable claims and does not present opinions as facts.</li>
</ol>
</li>
</ol>
</li>
<li dir="ltr" aria-level="1">Helpfulness:
<ol>
<li dir="ltr" aria-level="2">Instruction Following:
<ol>
<li dir="ltr" aria-level="3">Response Relevance: Assess if all model response content is relevant to address the prompt request.</li>
<li dir="ltr" aria-level="3">Main Request Adherence:&nbsp; Evaluate how well the response adheres to the main request specified in the prompt.&nbsp;</li>
<li dir="ltr" aria-level="3">Constraint Adherence: Evaluate if the response covers all constraints specified in the prompt.</li>
</ol>
</li>
<li dir="ltr" aria-level="2">Writing:
<ol>
<li dir="ltr" aria-level="3">Mechanics: Are the spelling, grammar, punctuation, and syntax all correct?</li>
<li dir="ltr" aria-level="3">Structural Clarity: Is the response well-organized with clear, logical flow and appropriate formatting?</li>
<li dir="ltr" aria-level="3">Verbosity: Is the response concise as it does not repeat the same arguments and is not overly wordy in its explanation?</li>
<li dir="ltr" aria-level="3">Fluency: The response is easy to understand and sounds natural. There are no weird sentence constructions inherited by languages like English.</li>
<li dir="ltr" aria-level="3">Style and tone: Is the style and tone of the writing suitable for the topic and audience?</li>
</ol>
</li>
</ol>
</li>
<li dir="ltr" aria-level="1">Harmlessness:
<ol>
<li dir="ltr" aria-level="2">Is the response free from harmful content and does it not facilitate harmful action?
<ol>
<li dir="ltr" aria-level="3">No: The response is free from harmful content and does not facilitate harmful actions.</li>
<li dir="ltr" aria-level="3">Yes: The response contains harmful elements or encourages harmful actions.</li>
</ol>
</li>
</ol>
</li>
</ol>
<p dir="ltr">Each dimension&rsquo;s sub-category is rated on whether it complies or not with the sub-category definition (i.e. &lsquo;yes&rsquo; or &lsquo;no&rsquo; binary score).</p>
<p dir="ltr">After the evaluation of stand-alone criteria, responses are compared side-by-side using a Likert scale. This comparative assessment helps in identifying the preferable model response based on a detailed justification tied to the evaluation criteria. We rank the models by the side-by-side elo scores for the leaderboard rankings.</p>
<h2>Evaluation Methodology</h2>
<p>Each model pairs with other models and&nbsp; receives a randomly chosen prompt from the set of 1,000 prompts described above.</p>
<p>Each evaluation tasks consists of the following:</p>
<ul>
<li dir="ltr" aria-level="1">Two models generate the responses for a prompt</li>
<li dir="ltr" aria-level="1">Annotators provide a point-wise evaluation of each response</li>
<li dir="ltr" aria-level="1">Annotators express their preference between the two scores on a 7-point Likert scale</li>
</ul>
<p>To ensure thoroughness and reliability in the evaluation process, each task was executed in parallel 3 times by different human annotators. Then, the ratings were then reviewed in two stages: an initial review layer and a final review layer. The figure below provides an overview of the evaluation pipeline design. After finalizing the tasks, a team of internal independent auditors randomly selected and reviewed 10% of the tasks for quality control.</p>
<p>Evaluation Methodology - Pipeline Design</p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=d8dd301f90348a8577382e4038caaef8.png&amp;f=webp&amp;q=75" alt="" width="960" height="348"></p>
<div class="p-rich_text_section">Leaderboard rankings are determined using&nbsp;<strong data-stringify-type="bold">Rank (Upper Bound)</strong>, which reflects a model&rsquo;s statistical position based on confidence intervals. The ranking process follows these steps:</div>
<ol class="p-rich_text_list p-rich_text_list__ordered p-rich_text_list--nested" data-stringify-type="ordered-list" data-list-tree="true" data-indent="0" data-border="0">
<li data-stringify-indent="0" data-stringify-border="0">Count the number of models that are&nbsp;<strong data-stringify-type="bold">statistically significantly better</strong>&nbsp;than the target model.</li>
<li data-stringify-indent="0" data-stringify-border="0">Add&nbsp;<strong data-stringify-type="bold">1</strong>&nbsp;to this count to determine the model&rsquo;s rank.</li>
</ol>
<div class="p-rich_text_section">A model is considered&nbsp;<strong data-stringify-type="bold">statistically significantly better</strong>&nbsp;than another if its&nbsp;<strong data-stringify-type="bold">lower-bound score (95% confidence interval) is higher</strong>&nbsp;than the other model&rsquo;s&nbsp;<strong data-stringify-type="bold">upper-bound score</strong>.Models receive the&nbsp;<strong data-stringify-type="bold">same rank</strong> when the same number of models are statistically better than each of them. This approach groups models based on statistical significance rather than raw scores, ensuring rankings reflect meaningful performance differences.</div>

Alibaba (Qwen)

Amazon (Nova)

Anthropic

Cohere

Databricks

DeepSeek

Google DeepMind

Meta (Llama)

Microsoft (Phi)

Mistral

OpenAI

Scale’s SEAL Spanish Leaderboard evaluates LLMs’ proficiency and interaction capabilities in Japanese.

Japanese

Performance Comparison