Methodology

Challenging LLMs at the frontier of human knowledge

Humanity's Last Exam

Models evaluated on text-only HLE questions

Evaluate model honesty when pressured to lie

Evaluating model performance on complex, multi-step reasoning tasks

Assessing models across diverse, interdisciplinary challenges

Vision-Language Understanding benchmark for multimodal models

Evaluating AI agents' ability to use enterprise tools effectively

Assessing chatbots' proficiency in leveraging external tools

Assessing models' ability to understand and generate programming code

Assessing performance on Arabic language understanding and generation

Measuring capabilities in Korean language processing and comprehension

Testing models' proficiency in Japanese language tasks and cultural nuances

Evaluating Spanish language skills across various linguistic challenges

Evaluating language models' proficiency in Chinese language tasks

Previously used for evaluating mathematical problem-solving capabilities

Former benchmark for assessing models' ability to follow complex instructions

Retired test for measuring models' resilience against adversarial inputs