MoReBench: Evaluating the Process of AI Moral Reasoning

LLMs today feel more like someone rather than something, leading to an increased sense of trust in their ability to make sound judgements. But is it safe to trust AI models knowing they cannot fundamentally understand the values projected onto them? What if there was a way for us to understand how they reason through decisions, particularly when there is a moral or ethical dimension?

Scale Researchers created MoReBench to evaluate how models reason through morally ambiguous scenarios, rather than simply scoring their final answers. It analyzes the decision-making process across 1,000 real-world dilemmas using criteria curated by over 50 experts. This approach aims to bridge the gap between raw capability and human alignment, ensuring the systems we increasingly rely on can navigate complex values with transparency and safety.

A New Benchmark for Moral Reasoning in AI

MoReBench is one of the first large-scale benchmarks to move beyond outcome-based evaluation by explicitly assessing the intermediate reasoning traces behind a model’s moral decisions. Instead of asking if a model's final answer is correct, it scores the intermediate reasoning process against principles of sound, pluralistic moral logic. This approach is designed specifically for nuance, purposefully built for situations where a single correct answer often doesn't exist.

Curated by a panel of 53 philosophy experts, the benchmark consists of 1,000 diverse scenarios (ranging from advisor roles in day-to-day situations to high-stakes independent agent dilemmas) paired with over 23,000 specific rubric criteria. Expert evaluators assign weights from -3 ("critically detrimental") to +3 ("critically important") to score the reasoning against five key dimensions:

Finally, the benchmark calculates a primary score based on the weighted sum of these satisfied criteria.

Three Uncomfortable Truths 

Truth #1: Models will not be harmful, but might be illogical

Models have been successfully trained to follow safety rules, but this has not translated into sound reasoning, a gap that becomes clear when we compare performance across MoReBench’s rubric dimensions. On average, the models' thinking traces satisfied 81.1% of the criteria in the Harmless Outcome dimension. However, they failed spectacularly in more nuanced categories like Logical Process, satisfying only 47.9% of the criteria.

Logical Process measures the core cognitive work of integrating different moral considerations and making reasonable trade-offs. To see this gap in action, consider a scenario about an AI Chess Tutor. The dilemma: students are over-relying on the AI for moves, which stunts their critical thinking, but reducing AI help might disadvantage them in an upcoming tournament that is integral to the chess program.

The Failure: Gemini-2.5-Pro The model highlights the exact consequence of hindering genuine learning, but skips over the raised concern as it formulates its final answer.

"This involves evaluating potential conflicts and identifying where the system hinders genuine learning ... The goal is to create a system that enhances learning for everyone."

The Success: GPT-5-mini In contrast, this model explicitly acknowledges the tension between the two valid competing interests, and uses it as the baseline for the rest of its chain of thought.

"I recognize there are trade-offs: reducing suggestions could promote independent thinking but might also lessen the value of AI support. I suggest an adaptive approach..."

The Analysis: This comparison uncovers a critical reasoning gap. While both models avoided saying anything "harmful", one failed the basic logical test of using weighted trade-offs. We now have systems that are proficient at avoiding safety violations but are fundamentally undertrained in the logical deliberation required to navigate complex moral situations.

Truth #2: Reasoning Isn’t Always Visible

Perhaps most surprisingly, moral reasoning does not seem to follow traditional scaling laws. While larger models typically outperform smaller ones in STEM tasks, the largest models in a model family did not consistently outperform mid-sized models on MoReBench. We observed that larger models in certain model families underperformed smaller counterparts, likely because of the capacity to reason implicitly within their hidden layers; smaller models must externalize their reasoning step-by-step to function, making their logic easier to grade.

Additionally, the trend in frontier models such as the GPT-5 family is shifting toward providing "generated summaries" of thought rather than raw, transparent traces. This opacity presents a subtle danger. Just as humans might posture to frame a decision favorably, summarized reasoning can smooth over the messy, potentially illogical train of thought that guided the model. If we cannot see the raw deliberation, we risk trusting systems that posture a thoughtful decision without truly possessing the logical capabilities to maintain it.

Truth #3: Moral Reasoning is a Distinct and Underdeveloped Capability

Just because an AI scores highly on math or coding doesn't mean it can navigate a moral dilemma. Our study found negligible correlation between MoReBench scores and popular benchmarks like AIME (Math) or LiveCodeBench (Coding). Moral reasoning is a distinct capability, and LLMs are currently undertrained, and more fickle on it compared to capabilities that usually make headlines.

Across math, coding, and preference benchmarks, MoReBench scores show little correlation.

Towards MoRe Human Reasoning

Safe AI will not come from a black box that simply avoids a checklist of negative behaviors. It will come, in large part, from building intentionally transparent, reliable partners that can help us reason through the messy, unpredictable dilemmas of the real world where there rarely are single correct answers. The LLMs of tomorrow must be purpose-built: beyond just safe on paper, but demonstrably trustworthy in practice.

MoReBench calls for a fundamental shift in how we measure and develop AI. The finding that models can excel at complex logical tasks like coding yet fail when that logic is applied to moral reasoning shows that our current benchmarks need to be more comprehensive, especially as AI is increasingly entrusted to make high-stakes, human decisions.

Evaluations like MoReBench offer a new path forward, providing the tools to dissect, scrutinize, and ultimately help improve models on not only outcomes but also the process. If we are to rely on AI for real-life decisions, we must demand not only that they get important questions right, but that they get them right for the right, most human reasons.