Methodology

Challenging LLMs at the frontier of human knowledge

Humanity's Last Exam

Humanity's Last Exam (Preview)

Humanity's Last Exam (Text Only)

Models evaluated on text-only HLE questions

Humanity's Last Exam Text Only (Preview)

Evaluate model honesty when pressured to lie

MASK

Evaluating model performance on complex, multi-step reasoning tasks

EnigmaEval

Assessing models across diverse, interdisciplinary challenges

MultiChallenge

Vision-Language Understanding benchmark for multimodal models

VISTA

Evaluating AI agents' ability to use enterprise tools effectively

Agentic Tool Use (Enterprise)

Assessing chatbots' proficiency in leveraging external tools

Agentic Tool Use (Chat)

Assessing models' ability to understand and generate programming code

Coding

Assessing performance on Arabic language understanding and generation

Arabic

Measuring capabilities in Korean language processing and comprehension

Korean

Testing models' proficiency in Japanese language tasks and cultural nuances

Japanese

Evaluating Spanish language skills across various linguistic challenges

Spanish

Evaluating language models' proficiency in Chinese language tasks

Chinese

Previously used for evaluating mathematical problem-solving capabilities

Math

Former benchmark for assessing models' ability to follow complex instructions

Instruction Following

Retired test for measuring models' resilience against adversarial inputs

Adversarial Robustness

<p dir="ltr">[hle-examples]</p>
<h2 dir="ltr"><strong>Introduction</strong></h2>
<p dir="ltr">AI capability is evaluated based on benchmarks, yet as their progress accelerates, benchmarks become quickly saturated, losing their utility as a measurement tool. Performing well on formerly frontier benchmarks such as <a href="https://arxiv.org/abs/2009.03300">MMLU</a> and <a href="https://arxiv.org/abs/2311.12022">GPQA</a> are no longer strong signals of progress as frontier models reach or exceed human level performance on them.</p>
<p dir="ltr">In partnership with the&nbsp;<a href="https://www.safe.ai/">Center for AI Safety</a>, we address the problem of benchmark saturation by creating Humanity&rsquo;s Last Exam (HLE):<strong> 2,700 of the toughest, subject-diverse, multi-modal questions designed to be the last academic exam of its kind for AI</strong>. HLE is designed to test for both depth of reasoning (eg. world-class mathematical problems) and breadth of knowledge across its subject domains, providing a precise measurement of model capability. Current frontier models perform poorly on HLE with low accuracies, and systematically exhibit uncalibrated overconfidence in their answers.</p>
<p dir="ltr">We <strong>publicly release Humanity&rsquo;s Last Exam</strong> for the research community to better understand model capabilities. Evaluation is low-cost, as questions are precise and unambiguous with closed-ended answers provided &ndash; allowing for automatic evaluation. To combat the serious problem of training data contamination and benchmark hacking, we have an additional held-out private set of HLE questions&nbsp; to periodically measure overfitting to the public dataset. More research on overfitting can be found <a href="https://scale.com/research/llm-performance-grade-school-arithmetic">here</a>.</p>
<p dir="ltr">High accuracy on HLE would demonstrate AI has achieved expert-level performance on closed-ended cutting-edge scientific knowledge, but it would not alone suggest autonomous research capabilities or &ldquo;artificial general intelligence.&rdquo;&nbsp;</p>
<p dir="ltr">See the linked <a href="https://arxiv.org/abs/2501.14249">full paper</a> and <a href="https://lastexam.ai">dataset.</a></p>
<h2 dir="ltr"><strong>Methodology</strong></h2>
<div class="p-rich_text_section">Leaderboard rankings are determined using&nbsp;<strong data-stringify-type="bold">Rank (Upper Bound)</strong>, which reflects a model&rsquo;s statistical position based on confidence intervals. The ranking process follows these steps:</div>
<ol class="p-rich_text_list p-rich_text_list__ordered p-rich_text_list--nested" data-stringify-type="ordered-list" data-list-tree="true" data-indent="0" data-border="0">
<li data-stringify-indent="0" data-stringify-border="0">Count the number of models that are&nbsp;<strong data-stringify-type="bold">statistically significantly better</strong>&nbsp;than the target model.</li>
<li data-stringify-indent="0" data-stringify-border="0">Add&nbsp;<strong data-stringify-type="bold">1</strong>&nbsp;to this count to determine the model&rsquo;s rank.</li>
</ol>
<div class="p-rich_text_section">A model is considered&nbsp;<strong data-stringify-type="bold">statistically significantly better</strong>&nbsp;than another if its&nbsp;<strong data-stringify-type="bold">lower-bound score (95% confidence interval) is higher</strong>&nbsp;than the other model&rsquo;s&nbsp;<strong data-stringify-type="bold">upper-bound score</strong>.Models receive the&nbsp;<strong data-stringify-type="bold">same rank</strong> when the same number of models are statistically better than each of them. This approach groups models based on statistical significance rather than raw scores, ensuring rankings reflect meaningful performance differences.</div>
<h2 dir="ltr"><strong>Leaderboard Results</strong></h2>
<p>HLE is a multi-modal benchmark, our main leaderboard includes only models with multi-modal capabilities. We present an additional benchmark of text-only models on the text-only split below.</p>
<p>We emphasize that as of release, most models do poorly on HLE, meaning small inflections in accuracy are not meaningfully differentiating for model rank within overlapping confidence intervals. Nevertheless, we observe models with inference time reasoning capabilities tend to do better than ones without reasoning.</p>
<div dir="ltr" align="left"><strong id="docs-internal-guid-f7cdaeda-7fff-143d-c22c-fea0bae70e08"></strong></div>
<h2 dir="ltr"><strong>Dataset Summary</strong></h2>
<p dir="ltr">Humanity&rsquo;s Last Exam includes questions across dozens of subjects across mathematics, humanities, and the natural sciences. We provide a high level visualization of the distribution of the benchmark categories &ndash; though there are many subjects within each summarized category.&nbsp;</p>
<p dir="ltr"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=cba736ca4b7e7e73a6e811ec8c9f1a16.png&amp;f=webp&amp;q=75&amp;w=3840" alt="" width="1866" height="814"></p>
<p dir="ltr">The benchmark is multimodal, with 13% of questions requiring comprehending a diagram or figure to answer the question. In addition, 24% of the questions are multiple choice.</p>
<h2 dir="ltr"><strong id="docs-internal-guid-a989f314-7fff-d96f-ef2a-10e377c49e92">Dataset Design</strong></h2>
<p dir="ltr">Humanity&rsquo;s Last Exam is a collaborative effort with questions from nearly 1000 subject expert contributors, affiliated with over 500 institutions across 50 countries &ndash; composed mostly of professors, researchers, and graduate degree holders. Participants competed for a $500,000 USD prize pool &ndash; $5,000 USD for each of the top 50 questions and $500 USD for the next 500 questions, along with the opportunity for optional co-authorship if any question is accepted in the final dataset. This structure incentives top questions from subject experts all around the world. More details can be found in our original announcement: <a href="https://scale.com/blog/humanitys-last-exam">https://scale.com/blog/humanitys-last-exam</a>.&nbsp;</p>
<p dir="ltr"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=66b3169fa2e6035142cc567428a6744e.png&amp;f=webp&amp;q=75&amp;w=3840" alt="" width="1866" height="497"></p>
<p dir="ltr">Submission: Submitted questions must stump several frontier LLMs for exact match questions, or allow only up to random chance across all LLMs for multiple choice questions to be considered for human review. This ensures the questions are of a necessary difficulty bar for the current generation of models, we further verify they are sufficiently difficult with human review. In total, we received over 70,000 submissions, with 13,000 passing this difficulty bar and forwarded to human review.</p>
<p dir="ltr">Human Review: We train experts sourced from&nbsp;<a href="https://scale.com/blog/new-era-outlier">Scale&rsquo;s Outlier platform</a> to review questions. All of the human reviewers have a graduate degree in their field. Reviewers score questions against a standardized rubric, providing feedback to help question creators iterate questions. A primary round is used to shortlist the best questions. A secondary review with both organizers and expert reviewers approves or rejects questions from the final dataset &ndash; resulting in 2,700 public questions and an additional set of private questions of equal quality and difficulty.</p>
<h2 dir="ltr"><strong>Metrics</strong></h2>
<p dir="ltr">We report both the accuracy on public questions of Humanity&rsquo;s Last Exam and use the model&rsquo;s own stated confidence to derive an&nbsp;<a href="https://github.com/hendrycks/outlier-exposure/blob/master/utils/calibration_tools.py">RMS calibration error</a> using the implementation from <a href="https://arxiv.org/abs/2112.05135">Hendrycks et al., 2022</a> with the default hyperparameters provided, reported in our paper for brevity. Models are ranked on the leaderboard using accuracy, however we want to emphasize calibration errors as an important metric in our paper.</p>
<p dir="ltr">A well-calibrated model should exhibit an average confidence similar to its accuracy on a benchmark - eg. 50% accuracy paired with 50% confidence. As of our initial publication, we observe systematic high calibration errors (greater than 80%) paired with low accuracy (less than 10%), which indicates strong evidence for confabulation/hallucination in all measured models.</p>
<p dir="ltr">Details on our evaluation methodology found below. At this time, we do not report any model performance metrics on the private held-out set.</p>
<h2 dir="ltr"><strong>Evaluation Methodology</strong></h2>
<p dir="ltr">Evaluation is automatic. Each model on the leaderboard is evaluated on all public questions of Humanity&rsquo;s Last Exam with temperature 0.0 when configurable or stated otherwise. Models are prompted to give a final answer and an estimation of confidence using the system prompts (or user prompt when not configurable) below depending on question type, following the setup from <a href="https://arxiv.org/abs/2411.04368">Wei et al., 2024</a>.</p>
<p dir="ltr">[hle-questions]</p>
<p dir="ltr">As HLE uses closed-form solutions, we use o3-mini-2025-01-31 as an automatic extractor and judge to compare the model response against the ground truth answer. We employ structured decoding to extract a JSON from the following prompt. We note small differences could arise from different judge models and prompts used on edge cases (eg. acceptable precision), hence we encourage the documentation of prompts and models used for evaluation on HLE. We document ours for this evaluation below.</p>
<p dir="ltr">[hle-prompts]</p>
<h2 dir="ltr"><strong>Acknowledgements</strong></h2>
<p dir="ltr">Humanity&rsquo;s Last Exam was a global collaborative effort developed in partnership with the <a href="https://www.safe.ai/">Center for AI Safety</a>. We extend our deepest gratitude to all participating question contributors and expert reviewers involved in creating and refining the Humanity&rsquo;s Last Exam dataset.</p>
<p dir="ltr">Scale AI Team: Ziwen Han, Josephina Hu, &dagger; Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, William Qian, Luis Esquivel, Caton Lu, Monica Mishra, Summer Yue, Alexandr Wang</p>
<p dir="ltr">&nbsp;</p>

<p><strong data-stringify-type="bold"><em data-stringify-type="italic">Rank (UB):</em></strong><em data-stringify-type="italic">&nbsp;1 + the number of models whose lower CI bound exceeds this model&rsquo;s upper CI bound.</em></p>
<p>-</p>

Humanity's Last Exam (Preview)

Performance Comparison