<p dir="ltr">In September, we <a href="https://scale.com/blog/mcp-atlas">introduced</a> MCP-Atlas, a benchmark measuring how well LLMs handle real tool use via the Model Context Protocol. We showed that even top models failed nearly half of realistic multi-tool tasks. Today, we're open-sourcing the benchmark so you can measure this yourself. As agents increasingly rely on tools to act in the real world&mdash;querying databases, calling APIs, and coordinating multi-step workflows&mdash;evaluating tool use has become as important as evaluating reasoning or language generation.</p>
<h2 dir="ltr">What We're Releasing</h2>
<p dir="ltr">Three artifacts, everything you need to run MCP-Atlas evaluations:</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation"><strong><a href="https://static.scale.com/uploads/674f4cc7a74e35bcaae1c29a/MCP_Atlas.pdf">Research Paper</a>:</strong> Full methodology and detailed analysis.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation"><strong><a href="https://huggingface.co/datasets/ScaleAI/MCP-Atlas">Public Dataset</a>:</strong> 500 human-authored tasks on Hugging Face</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation"><strong><a href="https://github.com/scaleapi/mcp-atlas/tree/main">Evaluation Environment</a>:</strong> A containerized setup exposing real MCP servers.</p>
</li>
</ul>
<p>The remaining 500 tasks are held out as a private validation set to preserve <a href="https://scale.com/leaderboard/mcp_atlas">leaderboard</a> integrity and prevent overfitting.</p>
<h2 dir="ltr">What Makes MCP-Atlas Different</h2>
<p dir="ltr">MCP-Atlas is designed to measure performance on realistic, multi-step workflows that handle real-world tool use through the Model Context Protocol (MCP).</p>
<p dir="ltr">Real MCP servers: All tasks run against real MCP implementations. Agents must handle real schemas, real errors, and real API behavior, not simplified mocks.</p>
<p dir="ltr">Natural language prompts without tool names: Prompts describe goals in plain language and avoid naming tools or servers. Agents must discover what to use, not follow instructions.</p>
<p dir="ltr">Controlled tool exposure with distractors: Each task exposes a limited set of tools that includes both required tools and distractors. This tests whether agents can select and parameterize the right tools under noise.</p>
<h2 dir="ltr">Updated Results</h2>
<p>Since September, we've evaluated additional models. The leaderboard has shifted significantly (<a href="https://scale.com/leaderboard/mcp_atlas">see full results here</a>):</p>
<table style="border-collapse: collapse; width: 100%; height: 276.796px;" border="1"><colgroup><col style="width: 33.3333%;"><col style="width: 33.3333%;"><col style="width: 33.3333%;"></colgroup>
<tbody>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;"><strong>Model</strong></td>
<td style="height: 18.4531px;"><strong>Pass Rate</strong></td>
<td style="height: 18.4531px;"><strong>Mean Coverage</strong></td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">claude-opus-4-5-2025-1101</td>
<td style="height: 18.4531px;">62.30%</td>
<td style="height: 18.4531px;">78.5%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">gemini-3-pro-preview</td>
<td style="height: 18.4531px;">54.10%</td>
<td style="height: 18.4531px;">73.2%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">GPT-5</td>
<td style="height: 18.4531px;">44.50%</td>
<td style="height: 18.4531px;">61.75%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">claude-4-5-sonnet</td>
<td style="height: 18.4531px;">43.80%</td>
<td style="height: 18.4531px;">62.17%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">o3</td>
<td style="height: 18.4531px;">43.60%</td>
<td style="height: 18.4531px;">66.91%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">Claude Opus 4.1</td>
<td style="height: 18.4531px;">40.90%</td>
<td style="height: 18.4531px;">64.99%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">Claude Sonnet 4</td>
<td style="height: 18.4531px;">35.60%</td>
<td style="height: 18.4531px;">57.35%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">GLM 4.5 Air*</td>
<td style="height: 18.4531px;">34.00%</td>
<td style="height: 18.4531px;">60.59%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">Kimi K2 Instruct*</td>
<td style="height: 18.4531px;">23.90%</td>
<td style="height: 18.4531px;">50.41%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">Qwen3-235B-A22B*</td>
<td style="height: 18.4531px;">12.00%</td>
<td style="height: 18.4531px;">29.06%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">Gemini 2.5 Pro</td>
<td style="height: 18.4531px;">8.80%</td>
<td style="height: 18.4531px;">30.77%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">GPT-4o</td>
<td style="height: 18.4531px;">7.20%</td>
<td style="height: 18.4531px;">28.53%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">Gemini 2.5 Flash</td>
<td style="height: 18.4531px;">3.40%</td>
<td style="height: 18.4531px;">17.83%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">Llama 4 Maverick*</td>
<td style="height: 18.4531px;">0.80%</td>
<td style="height: 18.4531px;">13.03%</td>
</tr>
</tbody>
</table>
<p dir="ltr"><sup>*Evaluations for these models were run using Fireworks AI for inference</sup></p>
<p dir="ltr">Claude Opus 4.5 now leads at 62.3%, compared to GPT-5&rsquo;s 44.5% at launch. But the broader pattern holds: even top models fail roughly 40% of tasks that require coordinating multiple tools.</p>
<h2 dir="ltr">Where Models Still Break</h2>
<p dir="ltr">Baseline results reveal consistent failure patterns across models:</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation"><strong>Tool usage is the dominant bottleneck.</strong> Errors most often come from selecting the wrong tool, using incorrect parameters, or sequencing calls incorrectly.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation"><strong>Many failures involve not using tools at all.</strong> Agents frequently stop early or fail to recognize that tools are required to solve the task.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation"><strong>When tools are used correctly, synthesis is rarely the issue. </strong>The main challenges are discovery, parameterization, and completing multi-step workflows reliably.</p>
</li>
</ul>
<p dir="ltr">MCP-Atlas is designed to make these behaviors measurable and comparable across agents.</p>
<h2 dir="ltr">How to use MCP-Atlas</h2>
<p dir="ltr">You can use MCP-Atlas to:</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Reproduce published baseline results</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Evaluate your own MCP-capable agent</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Compare models under identical tool conditions</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Track regressions as you improve planning, tool retrieval, or execution</p>
</li>
</ul>
<h2 dir="ltr">Get started</h2>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Read the <a href="https://scale.com/research/mcpatlas">paper</a></p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Explore the dataset on <a href="https://huggingface.co/datasets/ScaleAI/MCP-Atlas">Hugging Face</a></p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Run the environment and harness on <a href="https://github.com/scaleapi/mcp-atlas">GitHub</a></p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">View rankings on the <a href="https://scale.com/leaderboard/mcp_atlas">leaderboard</a></p>
</li>
</ul>
<p dir="ltr">We&rsquo;re releasing MCP-Atlas to help establish a shared, realistic standard for evaluating tool-using agents. We welcome feedback and contributions as the benchmark evolves, and we&rsquo;re especially interested in seeing how new planning, retrieval, and execution strategies perform under MCP-Atlas.</p>
<p>&nbsp;</p>

MCP-Atlas is now open source: a benchmark measuring how AI agents use real tools across complex, multi-step tasks.