Insights Generator: Diagnosing Agent Failures

Too many companies are learning the hard way: successful pilots often do not lead to successful deployments. As engineers take their agents from v1 to v2, diagnosing failure patterns becomes a major engineering cost. We created Insights Generator to increase engineer productivity when doing this work. In one study, engineers using Insights Generator improved their agents nearly twice as much as those using leading tools.

Coming to the Scale GenAI Platform, Insights Generator reads across an agent's full run history at once, catching what no single trace reveals.

For a closer technical look, read the paper or the full write-up on Scale Labs.

Results

We evaluated Insights Generator across four settings, varying who did the judging and what was measured. Two results stand out:

• Engineers using Insights Generator nearly doubled the improvement of Claude Code with subagents. In a controlled study, Scale engineers given IG reports lifted agent performance by +30.4 percentage points over baseline in a 90-minute window. The same engineers given Claude Code with subagents reports achieved +16.2 points in the same window. The diagnostic report was the only variable.
• Insights Generator produced the strongest diagnostic reports under automated evaluation. In a pairwise tournament using an automated judge, IG's diagnostic reports won 77.9% of comparisons across SpreadsheetBench and Humanity's Last Exam, compared with 62.4% for reports from the next-best system. IG's largest advantages were in explaining the mechanisms behind failures and grounding findings in specific trace evidence, giving engineers a clearer path from diagnosis to fix.

Most agent debugging tools work one trace at a time. Observability platforms let you inspect individual runs, but you still have to do the reading. Other tools look across many runs but flag patterns too vaguely to fix. Insights Generator reads across the full set of an agent's runs, tests theories about what's going wrong, and produces a report that names each pattern, how often it occurs, and which specific runs to look at.

Failure Compound

Agents often don’t know when they fail. They produce wrong answers with confident self-assessment, and the errors compound across runs. By the time anyone notices, the behavior is baked in. Insights Generator surfaces these failures at scale. A few examples from corpora we analyzed:

• On AppWorld, a benchmark where agents complete multi-step tasks across apps like Spotify and Venmo, 50 of 51 incorrect runs marked themselves as successfully completed, often with celebratory language
• On SpreadsheetBench, 128 of 250 runs generated a self-contained solution that the runtime silently discarded. Agents usually recovered on the next turn, but the bug wasted at least 128 turns.
• IG uncovered a benchmark-contamination channel on Humanity’s Last Exam. Four of 391 runs found copies of the benchmark on Hugging Face that included the answers, and all four were graded correct. One agent downloaded the benchmark’s complete answer key; another copied an exact answer from a search result and immediately returned it.

While a single trace can present an error mode, full corpus analysis identifies and prioritizes the most important error modes for improving agent performance. All of them were obvious once the full corpus was analyzed together. Reading a handful of traces gives you anecdotes, but reading across the corpus gives you the patterns that actually explain how your agent behaves in production.

Coming to Scale GenAI Platform

Insights Generator is rolling out to Scale GenAI Platform customers as a diagnostic layer for production agents. Instead of engineers manually pulling traces and hunting for patterns when something goes wrong, IG surfaces and quantifies issues across the full trace corpus, so teams spend their time fixing what's breaking rather than finding it.

The result is a shift in how agent improvement works. It stops being a manual, one-off engineering project and starts becoming a continuous loop, where better diagnostics produce better outcomes on both sides, whether the fix comes from a person or a system. That's how enterprises close the gap between a successful pilot and a reliable deployment.