Matthew Siegel

In order to create and maintain the best AI agents available, enterprises need to be constantly evaluating and improving them. Doing this manually is both slow and expensive, so at Scale we solve this problem using LLMs as judges. The way we do this is simple: we use the LLM judges to compare two agents that are identical except for one small variable. This allows us to reliably determine what actually causes a shift in behavior, which is the foundation of making principled, incremental improvements.
These evaluations, however, have a critical flaw: they are not repeatable over time. Our internal experiments revealed that the exact same test can produce metrics that change by as much as 10-15% from one day to the next. This level of instability makes it impossible to trust the results, turning a principled process into a game of chance. In this post we&rsquo;ll share the data that shows how this is an industry-wide problem, explain its root cause, and detail our practical &ldquo;cohort of judges&rdquo; method for solving it.&nbsp;
This foundation of reliable measurement enables all of our advanced agent improvement work, including our research in <a href="https://scale.com/blog/enterprise-agent-reinforcement-learning">reinforcement learning</a>.
<h2 dir="ltr">An Industry-Wide Problem</h2>
Our investigation began with a project to evaluate a student assistant chatbot, where the aim was to assess its performance across key dimensions like avoiding direct answers, resisting jailbreak attempts, and staying on topic. When we used LLM Judges, the results quickly showed that the exact same evaluations were not repeatable.
<ul>
<li dir="ltr" aria-level="1">
On jailbreak resistance, the same model scored 77% one day and 63% the next.
</li>
<li dir="ltr" aria-level="1">
On refusal fidelity, it swung from 71% to 81% across consecutive runs.
</li>
</ul>
This variance occurred in our tests of models from OpenAI, Google, and Anthropic, with the following ranges of variance, suggesting a systemic challenge for current LLM provider APIs:
<ul>
<li dir="ltr" aria-level="1">
OpenAI (GPT-4 variants): &plusmn;10&ndash;12%
</li>
<li dir="ltr" aria-level="1">
Anthropic (Claude variants): &plusmn;8&ndash;11%
</li>
<li dir="ltr" aria-level="1">
Google (Gemini variants): &plusmn;9&ndash;14%
</li>
</ul>
The variance in evaluation metrics had a margin of error large enough to invalidate any A/B test. This makes principled, incremental improvement nearly impossible. To achieve a 50% improvement in an agent's performance, a team might make ten small changes, each expected to contribute a ~5% gain. However, an A/B test cannot reliably detect a 5% signal when the day-to-day noise of the measurement tool is 10-15%. Positive changes can appear negative, and vice versa, making the development process inefficient and unreliable.
<h2 dir="ltr">Why Might this Happen?</h2>
There are a few potential reasons why this happens, some of them simple, others a bit more complicated. The simpler reason is because provider APIs have constantly shifting components; in other words, builders are always tweaking their models, so you&rsquo;re likely using a model with slightly different components today than you were yesterday. This is a key factor for next-generation frontier models as well, as users often lack the fine-grained control to select a specific, static model version.
The deeper technical reason is a consequence of the architecture used by most frontier LLMs and involves two key concepts: Sparse Mixture of Experts (MoE) and batched inference.
<ol>
<li dir="ltr" aria-level="1">
Sparse Mixture of Experts (MoE): Instead of being one massive, monolithic network, models tend to be composed of many smaller, specialized "expert" sub-networks. For any given input, the model dynamically routes the request through only a fraction of these sub-networks.
</li>
<li dir="ltr" aria-level="1">
Batched Inference: To operate efficiently, providers process many user requests simultaneously in a "batch."
</li>
</ol>
When these two techniques are combined, the results become unpredictable because the composition of a batch can determine which expert your query gets routed to. The mix of queries in a batch is not deterministic. For example, your query might be a math question, but if most other queries from other users in the same batch are related to psychology, your math query could be incorrectly routed to a psychology expert. This means the specific path your request takes through the model is influenced by the other requests being processed alongside it, making the output consistent only at the batch-level, not for your individual query.
For a more detailed analysis of this phenomenon, we recommend<a href="https://152334h.github.io/blog/non-determinism-in-gpt-4/"> this technical breakdown</a>.
<h2 dir="ltr">The Solution: A Cohort of Judges for Improved Reliability</h2>
Instead of relying on a single, noisy judge, we use a panel of three, which we call the &ldquo;cohort of judges.&rdquo; Each judge is given a slightly different prompt for the same task that is semantically the same. By aggregating the outputs from this cohort, we successfully smooth out the provider-side variance and produce a stable, repeatable evaluation metric. With this method, the variance in our evaluation results was reduced by at least 50% across the different dimensions we tested. This allows us to distinguish true performance changes from statistical noise, enabling us to trust our A/B test results and make principled, incremental improvements with confidence.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=cce83318d48707f054309a6d376169a2.png&amp;f=webp&amp;q=75" alt="" width="1200" height="675">
<h2 dir="ltr">The Requirement for Reliable Evaluation</h2>
The underlying models we build on are not as fundamentally stable as one might imagine. This variance is a structural, industry-wide challenge that can introduce uncertainty into the evaluation process, creating risks for product development and roadmap planning. By adopting a cohort-of-judges approach, we can produce more reliable, repeatable measurements needed for decision-making. This practical step moves organizations toward a more robust MLOps practice, ensuring that the hard work of improving agents is measured on a solid foundation of trustworthy data.

LLM eval variance breaks A/B testing. Our new method ensures stable, repeatable results for reliable evals,

Matthew Siegel

Smoothing Out LLM Variance for Reliable Enterprise Evals