<p dir="ltr">LLMs can write poetry, explain quantum physics, and debug code. But ask them to book a hotel, analyze a spreadsheet, and compute the total cost with tax? Many fail at the basics: picking the right tool, calling it correctly, and combining results into a useful answer.</p>
<p dir="ltr">As organizations deploy AI agents for real work, including customer support, data analysis and various forms of workflow automation, tool use competency is the difference between a helpful assistant and an expensive disappointment.</p>
<p dir="ltr">However, existing evaluations do not reflect realistic and multi-step workflows. We introduce <strong>MCP-Atlas</strong>, an agentic leaderboard evaluating models on their ability to use tools via the Model Context Protocol (MCP).</p>
<h2 dir="ltr">What We Built: A Real-World Tool Use Leaderboard</h2>
<p dir="ltr">MCP-Atlas evaluates how well models handle <strong>single-turn requests that require multiple tools</strong> across real MCP servers. Think: <em>"Look up Microsoft's IPO price in 1986, find the IPO prices of Apple, Amazon, and Google, then calculate which company had the lowest IPO price and by what percentage it was lower than the highest."</em></p>
<p dir="ltr">The Setup:</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation"><strong>1,000</strong> tasks spanning 40+ MCP servers and 300+ tools</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation"><strong>Real tools</strong> across buckets such as search engines, databases, file systems, APIs and development tools</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation"><strong>Realistic complexity:</strong> 3-6 tool calls per task, with distractors to test tool discovery</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation"><strong>Scoring:</strong> Pass/fail based on whether the final answer matches ground truth, plus detailed diagnostics</p>
</li>
</ul>
<h2 dir="ltr">Our Approach</h2>
<h3 dir="ltr">1. Scale &amp; Diversity</h3>
<p dir="ltr">Our environment exposes 300+ tools across 40+ servers. We test against real filesystem operations, search engines, databases (MongoDB, Airtable), financial APIs, development tools (Git, GitHub), and specialized services (weather, maps, academic papers). This breadth mirrors the tool diversity agents encounter in production.</p>
<h3 dir="ltr">2. Realism</h3>
<p dir="ltr">Every task is human-written with real-world data. Our Airtable tasks query real business datasets that Scale has acquired; search tasks pull live results.&nbsp;</p>
<p dir="ltr">Each task includes 10+ available tools, but only 3-7 are needed. Models must discover the right tools among distractors and near-synonyms (Exa Search vs DuckDuckGo Search vs Brave Search), testing models&rsquo; ability to pick the right tool for the right job.</p>
<h3 dir="ltr">3. Complexity</h3>
<p dir="ltr">Multi-tool workflows with 3-6 steps with conditional logic, cross-server coordination (65% of tasks), and branching decisions. A task like "Look up Microsoft's IPO price in 1986, find the IPO prices of Apple, Amazon, and Google, then calculate which company had the lowest IPO price and by what percentage it was lower than the highest" requires financial data retrieval and arithmetic computation in sequence.</p>
<h2 dir="ltr">Results</h2>
<p dir="ltr">We ran <strong>11 leading models </strong>across our private leaderboard set and report <strong>pass rate </strong>as the share of tasks that reached a correct ground truth answer.</p>
<h3 dir="ltr">Performance</h3>
<ul>
<li data-stringify-indent="0" data-stringify-border="1">The top performer, GPT-5, passed&nbsp;<strong data-stringify-type="bold">44.5%</strong> of tasks (445/1000)</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">The median model, Kimi-K2, landed at <strong>23.9%&nbsp;</strong></p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Half of the models scored between&nbsp;<strong>8%</strong> and <strong>38.3%</strong> pass rate</p>
</li>
</ul>
<h3 dir="ltr">Where Agents Stumble: The Diagnostic Picture</h3>
<p dir="ltr">These aren't abstract reasoning puzzles; they're the kinds of multi-step workflows that define practical AI utility. When the best model fails more than half the time, there's a significant opportunity for improvement.</p>
<p dir="ltr">Beyond pass/fail rates, we categorized every failure to understand where models struggle most:</p>
<table style="border-collapse: collapse; width: 100%; height: 92.2655px;" border="1"><colgroup><col style="width: 50%;"><col style="width: 50%;"></colgroup>
<tbody>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;"><strong>Failure Category</strong></td>
<td style="height: 18.4531px;"><strong>% of All Tasks (Across All Models)</strong></td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">Tool Usage</td>
<td style="height: 18.4531px;">34-52%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">Task Understanding</td>
<td style="height: 18.4531px;">23-38%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">Response Quality</td>
<td style="height: 18.4531px;">3-8%</td>
</tr>
<tr style="height: 18.4531px;">
<td style="height: 18.4531px;">Logical Error</td>
<td style="height: 18.4531px;">1-2%</td>
</tr>
</tbody>
</table>
<p dir="ltr">The dominant failure mode is Tool Usage, accounting for between a third to a half of all task failures (depending on the model). This breaks down into three primary issues:</p>
<ol>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation"><strong>Tool Discovery Problems:</strong> Models frequently select plausible, but incorrect tools when faced with similar options. For example, choosing web search instead of local search for location-based queries, or selecting MongoDB when Airtable would be more appropriate for the task structure.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation"><strong>Parameter Construction Issues:</strong> Even when models pick the right tool, they struggle with correctly invoking them. Common problems include incorrect date calculations, wrong units or data types, enum value mistakes, and schema compliance failures that cause API calls to fail.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation"><strong>Orchestration Gaps:</strong> Models often demonstrate poor workflow management by stopping prematurely before completing all required steps, failing to combine intermediate results from multiple tools, or abandoning tasks entirely after encountering a single error rather than implementing retry strategies.</p>
</li>
</ol>
<p><strong id="docs-internal-guid-74d859c0-7fff-24ac-072b-92dd647a5b26">The bottom line: </strong>even state-of-the-art frontier models leave a lot on the table once real tools and schemas are involved. The headroom for improvement is large, and now, measurable.</p>
<h2 dir="ltr">What This Means for AI Teams</h2>
<h3 dir="ltr">For Model Developers</h3>
<p dir="ltr">The failure analysis points to three high-impact improvement areas. <strong>Tool selection accuracy</strong> shows models struggle with distinguishing between similar tools, particularly when schemas or descriptions are ambiguous. <strong>Parameter validation</strong> shows frequent issues with type conformance, date handling, and enum values. <strong>Error recovery </strong>represents a significant gap, as models often abandon tasks after single failures rather than attempting repairs or retries</p>
<h3 dir="ltr">For Product Developers</h3>
<p dir="ltr">When evaluating models for production deployment, consider testing them on multi-tool workflows from your specific domain. Key questions include whether models can recover from tool errors or give up after single failures, and how accurately they discover relevant tools among distractors.</p>
<p dir="ltr">Watch for models that excel at conversation, but struggle with structured output, systems that work in demos, but fail on edge cases, and Agents execute tools without stating <strong>why this tool</strong> was chosen. These patterns may indicate poor real-world reliability.</p>
<h2 dir="ltr">What's Next?</h2>
<p dir="ltr">This initial release includes our private leaderboard and key findings. We plan to release a public dataset for reproducible research, an open-source evaluation harness with all MCP servers, and a research paper with comprehensive methodology and analysis.</p>
<p>View the live <a href="https://scale.com/leaderboard/mcp_atlas" target="_blank" rel="noopener">leaderboard</a>.</p>

MCP-Atlas benchmarks AI agents on real multi-tool tasks, exposing tool-use gaps and a leaderboard with diagnostics data.