Introducing DualChem

Most existing AI benchmarks evaluate general capability, such as reasoning accuracy, coding performance, or factual recall, all of which do not accurately capture a model’s behavior in ambiguous, high-stakes environments. DualChem was designed to measure the latter: determining whether a model is capable of providing the same answer to support either legitimate scientific research or harmful activity.

For government agencies tasked with assessing increasingly capable frontier models, it is important to be able to better understand whether these systems can recognize harmful intent. DualChem is one of the first benchmarks designed to evaluate whether large language models provide assistance in such dual-use chemistry scenarios, and where current safeguards related to this ability may fall short, which in the real world would have to be balanced against not inhibiting beneficial uses.

Scale’s findings are captured in our recent research paper, “DualChem: Measuring LLM Uplift and Dual-Use Risk Awareness in Chemistry ”.

This effort represents the first benchmark resulting from a Collaborative Research and Development Agreement signed with the NIST Center for Advancing Innovation and Standards for Super Intelligence (CAISSI) in February 2025, combining Scale's evaluation expertise with input from CAISSI and U.S. Government subject-matter experts, who informed the scope and question taxonomy underpinning the benchmark.

The Problem DualChem Was Built to Solve

Chemistry is one of the clearest testbeds for understanding dual-use risk in frontier AI systems. The same chemistry guidance that helps a graduate student troubleshoot an experiment could also lower the barrier for harmful misuse. Existing safety benchmarks often focus on explicit harmful requests, but in the real world, the same question may have both legitimate and harmful applications depending on context.

The DualChem benchmark consists of 600 expert-curated questions (reviewed by PhD-level expert evaluators before inclusion) spanning multiple laboratory chemistry domains, including molecular design, synthesis procedures, purification, validation, and information management. The questions were developed by chemistry experts, most with advanced degrees, to reflect realistic dual-use ambiguity where the same request could support either legitimate research or harmful misuse.

Rather than relying on synthetic or explicitly adversarial prompts, which can fail to capture how real chemistry questions are asked, the benchmark focuses on realistic chemistry questions with ambiguous dual-use applications.

We evaluated 31 state-of-the-art frontier models on the benchmark; you can see how they performed here.

Research Findings

• Frontier models demonstrate high performance on dual-use chemistry questions. Models have an average accuracy of ~50% , with top-performing models reaching ~67%. LLM assistance raised the performance of participants with bachelor’s degrees in chemistry on dual-use chemistry tasks from around ~32% to around ~48%. This means a frontier model can provide relevant chemistry assistance, which is a meaningful capability gap with potential real-world implications.
• Models can see the risk and still answer. Models frequently identified questions as potentially dual-use, but still provided substantive answers in most cases. Across evaluated models, risk identification and refusal behavior appeared only weakly connected.
• Models differ from experts in their judgements of how dual-use a question actually is. When tested on their ability to assign severity scores to dual-use scenarios — using expert-annotated examples labeled as benign or harmful — model predictions showed only weak alignment with human judgments and significant variability across cases. This indicates that current models can flag surface-level sensitivity without being able to reason about what actually makes a question dual-use.

DualChem contributes something to the field that has largely remained unexplored: a rigorous, expert-validated framework for measuring a specific and consequential failure mode, one that doesn’t show up in standard capability or refusal benchmarks.

The findings also suggest that refusal-based safety mechanisms alone may be insufficient for handling complex dual-use scenarios. Independent assessments can help policymakers and procurement teams better understand how models behave in ambiguous dual-use scenarios and where safeguards may fall short.

Building Better Evaluation Frameworks

This research highlights the importance of collaboration between industry, researchers, and government institutions in developing shared evaluation methodologies for frontier AI systems.

It is critical that public and private partnerships can enable evaluation frameworks that support institutional oversight, including adversarial stress-testing, robustness analysis under distribution shifts, and measurement approaches aligned with evaluators’ needs.

Scale Labs will continue partnering with governments and national AI safety institutes to advance research on frontier AI evaluation and deployment. The goal is to help institutions better understand how models behave across different contexts and what safeguards, testing, and evaluation mechanisms support reliable deployment.