How We Engineer World-Class Data at Scale

It doesn't matter how powerful the chips or how sophisticated the algorithms; if the data is poor, the model is poor. Models learn what “correct” looks like from the data they train on, and higher quality training data has helped drive the rapid improvements in AI capabilities over the past few years. Producing this data for frontier labs has long been the heart of our work, and today, the labs we work with accept 97% of our data on the first pass.

That translates to models that are more accurate, reliable, and ready to perform sooner; when a new frontier model launches and is noticeably better than the last one, data is a big reason why.

This post details the three levels of quality assurance backing our work: task-level review, which verifies individual annotations; dataset-level evaluation, which gauges a dataset's collective potential to improve model performance; and contributor-level assessment, which measures the skill, integrity, and fit of human annotators.

Task-Level Quality

Every task (a discrete assignment like ranking model responses or rewriting a flawed answer) produces an annotation that shapes the model it trains, so task-level review catches any errors before they reach it.

We call our system for that work Archie, short for Automated Review & Checking with Human-In-the-loop Evaluation. In this system, AI agents act as impartial judges, checking each annotation against the project's quality rules. Human experts review edge cases and set the standards the model is trained to meet.

These agents can handle tasks ranging from a simple check by an AI model to complex assignments that require using tools, searching the web, or performing multi-step reasoning (e.g., multi-agent debate). This flexibility lets Archie match the right level of scrutiny to every task, and every component can be configured to meet the specific standards of each customer's project.

Archie is designed around several key properties:

• Convenient Setup: Our "auto task generation" feature translates high-level project specifications directly into runnable evaluation tasks, bypassing the complexities of prompt engineering. This can reduce setup time from days to minutes.
• Automated Human Alignment: Archie uses a process called auto-hillclimbing, an iterative optimization technique that refines evaluation prompts by testing them against a calibration dataset of ideal examples. This keeps an AI agent's judgment aligned with human nuance, ensuring decisions are consistently aligned with human standards while significantly reducing manual tuning time.
• Failure Pattern Analysis: Going beyond simple scoring, Archie automatically analyzes failure patterns to provide actionable insights. It generates reports that summarize common error types (failure modes), helping teams quickly diagnose where and why quality issues are occurring so they can improve their data pipelines.

Archie includes prebuilt, customizable agents to evaluate key criteria like instruction-following, truthfulness, and consistency. Before deployment, we require Archie's performance to match that of expert human reviewers on pre-delivery checks. This ensures human-level accuracy while allowing us to scale detailed reviews across every task.

Dataset-Level Quality

Even when every label (every rating, ranking, or correction the contributors produce) is right, gaps in the dataset can still leave the model unable to learn what it needs to. Catching and filling those gaps is the job of dataset-level evaluation.

Our model data quality framework measures how much a dataset will improve a customer's model. It analyzes prompts, rubrics, and ground truth across the full dataset, combining multiple methods, including LLM-as-verifier checks, fine-tuned models, and, where appropriate, heuristic and statistical approaches.

Key metrics include:

• Diversity: Topic and task variety to ensure broad coverage and reduce bias.
• Relevance & Naturalness: Alignment with reference sets and human-like phrasing.
• Difficulty & Similarity: Complexity to challenge models while minimizing redundancy.
• Generalization: Detecting issues like reward hacking (where models learn to game evaluation metrics rather than genuinely improve) to ensure genuine learning.

These metrics combine into a composite quality score that predicts a dataset's likely impact on model performance before it ever touches training, delivered via reports and monitoring pipelines so teams can act on them early. The actionable metrics from the Model Data Quality system drive continuous improvement across each iteration of the data lifecycle. This dataset-level focus complements task-level quality, and the toolkit is continually expanding into rubric and agentic data, along with code, math, and multimodal formats like audio and video.

Contributor-Level Quality

World-class data comes from world-class human contributors. Sustaining that quality at scale means actively verifying both the skill behind every contribution and the authenticity of the effort.

We do that through contributor-level review. A system of intelligent agents verifies the integrity and authenticity of every contribution. Operating within the Archie framework, it combines several powerful sub-systems to assess different signals of genuine, high-effort work:

• Human/LLM Likeness: Detects the statistical likelihood that a response was generated by another LLM, helping to confirm that human-in-the-loop work is truly human.
• Behavioral Risk: Analyzes user interaction patterns to detect anomalies that indicate low-effort or inauthentic work.
• Spam/Gibberish: Automatically flags and filters out nonsensical or egregiously low-quality content.

Beyond integrity, a contributor-project fit model measures their ability to produce high-quality data on that specific task. It combines a contributor’s background, skills, and profile, along with the work they have performed on the Outlier platform to offer them projects that best align with their skills.

The contributor-project fit model improves through a feedback loop that runs on every project:

For example, when a contributor with a computer science degree joins Outlier they may work on a basic coding evals project and progress to more complex agentic coding projects as they provide evidence of their tasking abilities within the domain. The result is world-class contributors working on the projects where they are most likely to deliver.

Consistent Quality Metrics

Treating data quality as a system across these three levels is what makes a 97% acceptance rate at first pass possible, and it's why the labs we work with trust us with the data that trains their models. We’ve earned that trust across a decade of work alongside many of the labs building the most powerful models in the world. Our role is making sure those models are powerful in the right way: accurate, reliable, and ready for the work people need them to do.