Scale and Singapore's IMDA Advance AI Evaluation with New Legal Reasoning Benchmark

As AI systems become more capable, they are increasingly being used to answer questions and support decisions in areas where accuracy matters. Law is one example of a regulated industry that is adapting to using AI. A model may be able to explain a law or summarize a court decision, but applying that law to a person's circumstances requires understanding which facts matter, recognizing when information is missing, and knowing when to ask questions before reaching a conclusion.

Today, Scale AI and Singapore's Infocomm Media Development Authority (IMDA) are introducing SG-LegalClarify, a large-scale benchmark for evaluating legal reasoning and ambiguity recognition under Singapore law. Developed as the first research initiative under Scale's collaboration with IMDA, the benchmark evaluates 19 AI models across 1,860 legal scenarios spanning 10 domains of Singaporean law.

The research marks an important step in our broader work with IMDA to advance AI evaluation that reflects national and regional contexts. As governments consider how to deploy and oversee increasingly capable AI systems, they need ways to independently assess whether those systems perform reliably against their own laws, institutions, and standards.

Challenges for Existing Legal Benchmarks

AI models are trained on information from around the world, but much of today's evaluation infrastructure focuses on a relatively small number of languages, legal systems, and jurisdictions.

Singapore presents a particularly interesting challenge. Its legal system draws on English common law while incorporating Singapore-specific statutes and legal principles. A model that has learned general common-law concepts may still reach the wrong conclusion if it fails to recognize where Singapore law differs.

Many existing legal benchmarks focus on whether models can answer a question correctly when all the necessary information has already been provided. Real legal questions rarely arrive that neatly.

Consider someone asking an AI assistant what penalty they might face after being caught speeding on an expressway in Singapore. To answer correctly, the model needs to know what type of vehicle they were driving, how fast they were traveling, and how many demerit points they already have.

An AI assistant that immediately provides a penalty without asking for those details may give an answer that sounds authoritative but is legally incorrect.

How We Evaluated the Models

SG-LegalClarify contains 1,100 original questions written by qualified legal professionals with expertise in Singapore law. The questions cover 10 legal domains, including criminal law, employment, property, family law, and judicial processes.

Of those questions, 760 deliberately omit information needed to determine the correct answer. Each is paired with a version that includes the missing facts, creating 1,860 evaluation scenarios in total. Models can ask a simulated user for missing facts, allowing the benchmark to test how they gather information before answering.

The benchmark compares models’ performance when the same facts must be gathered and when they are provided upfront. A three-model AI judge panel, selected through comparison with human expert assessments, evaluates responses against 12,546 expert-written grading criteria across four dimensions:

• Issue identification: Does the model recognize the relevant legal questions and variables?
• Rule recall: Does it correctly identify the applicable law?
• Application: Can it apply the law to the specific facts?
• Conclusion: Does it reach the correct legal outcome?

The study tested 19 models from leading AI developers, including OpenAI, Anthropic, Google DeepMind, DeepSeek, and others, both with and without access to web search.

Key Findings

1. The benchmark leaves substantial room for improvement

Without access to web search, the highest-performing model satisfied just 69.8% of mandatory grading criteria, while the average across all 19 models was 53.8%.

Providing web search improved performance, raising the average to 63.0%. The highest-performing model with web access satisfied 77.8% of mandatory criteria.

The research also identified recalling the correct legal rules as a particular weakness. With web access, models satisfied only 56.5% of applicable rule-recall criteria.

This matters because an AI system can recognize the general nature of a legal question and still reach the wrong answer if it relies on an incorrect or incomplete understanding of the law.

2. Models frequently fail to recognize when they need more information

Without web search, approximately 39% of scenarios with missing facts received no clarifying question.

Models also asked unnecessary questions in approximately 28% of scenarios where enough information had already been provided.

When information was missing, models asked about an average of just 39% of the relevant facts identified by legal experts.

These findings highlight a critical challenge for AI assistants: knowing when they have enough information to provide a reliable answer.

3. Asking the right questions does not guarantee the right answer

The research also found that models did not always reach a correct conclusion even after asking about all the missing facts.

Among web search-enabled evaluation trials with missing facts where models asked about every relevant missing fact, only 55.6% reached the correct legal conclusion. Another 32.1% reached an incorrect conclusion, while 12.3% provided no conclusion.

Why Evaluation in Local Contexts Matters

SG-LegalClarify is the first benchmark to emerge from Scale's collaboration with IMDA, announced in July 2026, and illustrates the value of evaluating AI in national contexts.

As AI systems are deployed across borders, governments need the ability to independently evaluate how those systems perform against their own laws, languages, and institutional requirements. Strong performance on international benchmarks does not necessarily translate to reliable performance in every national context.

Collaborations like ours with IMDA are critical to building that capability. IMDA brings deep expertise in Singapore's regulatory environment and AI governance, while Scale contributes expertise in developing rigorous benchmarks and evaluating frontier models. Together, we are helping build the evaluation infrastructure governments need to make informed decisions about AI adoption and oversight.

While SG-LegalClarify focuses on Singapore law, the approach offers a model for other countries seeking to evaluate AI systems against their own requirements.

Scale's collaboration with IMDA

SG-LegalClarify is the first research milestone in Scale's collaboration with IMDA. We're also exploring additional work in areas including agentic AI, model evaluation methods, and AI safety.

We plan to release a public subset of the benchmark to support further research and independent analysis. As AI systems become more widely deployed, we're committed to working with governments and researchers to develop evaluations that reflect the environments where these systems are used.