In An Agentic World Where Automation Gets Cheap, Which Work Is Worth Routing to a Human?

Where Human Judgment Retains Its Value

Large companies often leave money on the table in accounts payable. Vendor statements routinely contain credits, overpayments, and duplicate charges that are recoverable, but identifying them requires manual reconciliation against enterprise records. Review capacity is the binding constraint. Statement volume exceeds the bandwidth available to the accounts-payable team, so only a portion of vendor statements can be reviewed in any period, and recoverable balances remain outstanding.

Working with the finance team at a global media technology company (Media Co.), Scale AI built an agentic system for this workflow. It ingests and classifies vendor inquiries, identifies credit items within vendor statements, obtains credit memos from vendors, and ultimately posts to the enterprise resource planning (ERP) system. The system is projected to recover millions of dollars in credits that would otherwise remain unclaimed.

One of the questions in building it was when to let the agent act on its own and when to put a person in the loop. That question, and how to answer it in dollars, is the subject of this article.

With each wave of automation technology comes a familiar question: how much of a given job can be transferred to a machine? Agentic systems complicate that framing. A language-model agent will attempt nearly any knowledge task at negligible marginal cost, and its reliability can vary from task to task. Answering the question in dollars, though, requires a domain where both the cost of a mistake and the cost of a person’s time can actually be measured.

Finance is often such a domain, particularly credit recovery. A credit is either recovered or it is not, and an analyst’s hour is either spent on one statement or another. Both sides are already denominated in dollars, so the boundary at which human review becomes worthwhile can be set directly. Not every decision works this way. A CFO certifies quarterly statements because the law requires it, not because their review is the cheapest way to catch an error, and some decisions inside this workflow are similarly governed by regulation, internal policy, or the vendor relationship. Those are handled as rules rather than priced.

Automation also often comes with a second-order effect that merits attention. As the marginal cost of a task approaches zero, the volume of work that is economically worthwhile does not remain fixed. It expands. Work that did not justify a human’s time becomes worth an agent's, and previously out-of-scope work becomes worth pursuing. Jevons (1865) observed the analogous pattern in coal consumption, where gains in steam-engine efficiency raised total consumption by rendering previously uneconomic applications viable.

Maximizing the share of work that is automated is no longer the binding objective. The question is which slice of an expanded universe of work deserves human attention. We argue that in finance workflows this boundary can be located quantitatively, by pricing human review against the expected value of the decisions that review improves. The resulting boundary is risk-adjusted rather than fixed: as the potential impact of an error increases, the confidence required for autonomous action should increase with it. We develop this framework using an accounts-payable credit-recovery workflow as the motivating case and show how it can be used to allocate limited human-review capacity toward the decisions where intervention has the greatest expected value.

Related Work

The closest work to ours is what enterprises already run in production. AP automation follows a consistent shape: extract fields from the document, validate them against rules, and route for approval when a rule fails or a spend threshold is crossed. Varun and Islam (2026) describe a no-code agentic architecture in which an LLM reads invoices, retrieves tax rules from a vector database, and writes results back through database functions exposed as tools. Simcox (2026) describes the same sequence inside cloud ERP, with capture, extraction, validation against purchase orders, and threshold-based approval routing. Both are well built for the happy path. Neither addresses what happens when the model is wrong. There is no confidence attached to a decision, no account of what an incorrect autonomous action costs, and no criterion for when a person should be pulled in. Approval routing in these systems is triggered by rule failures and spend limits, not by how likely the system is to be right.

Research on rule-based automation establishes the ceiling of deterministic approaches. Jamithireddy (2025) implements exception-based Robotic Process Automation (RPA) across accounts payable, accounts receivable, general ledger, and bank reconciliation in SAP, reporting shorter reconciliation cycles, less manual intervention, and improved compliance detection. That architecture assigns confidence scores to exceptions, though the scores derive from fixed rule tolerances and predetermined breach pathways. Therefore, performance degrades on transactions the rule set did not anticipate, which is the category into which vendor credits often fall.

Machine learning applied to documents comes closer to our work. Malla (2025) reports a production invoice-processing system combining optical character recognition (OCR), natural language processing, and a confidence-driven human review loop. Its measure of confidence answers a narrower question than ours, though: whether a field was read correctly off the page, rather than whether a full reconciliation decision is right. Dadopoulos and Moschidis (2026) use an LLM to score how often the correct record appears among three candidates offered to an analyst. That metric assumes a correct record exists for every line, but in credit reconciliation, the most common outcome is that no recoverable credit exists at all. Both systems decide what to put in front of a person; neither asks whether that person's time is worth more elsewhere.

A separate line of research asks how to split work between a model and a person. Selective prediction lets the model decline to answer when it is not confident and pass those cases to someone else; the fewer it answers, the higher its accuracy on the ones it does (Chow 1970). Human-in-the-loop pipelines apply the same idea with a confidence cutoff. These approaches do price the error, but at one flat rate: abstaining is worth the same on a $50 line as on a $50,000 one. In recovery work, where value concentrates in a few large credits, that assumption does not hold.

Whether a model's expressed confidence corresponds to its empirical accuracy is a question in its own right. Accuracy is observed only after the correct answer is known, whereas confidence is an estimate made at decision time. Xiong et al. (2024) find that large language models asked to report their own certainty are systematically overconfident. A second approach treats agreement across responses as evidence of reliability, but its usefulness depends on the temperature, the setting that governs how much randomness the model uses when producing text (Lin, Trivedi, and Sun 2023). Neither approach yields a figure usable in an economic calculation until it has been calibrated, meaning that decisions assigned a score of 0.9 prove correct close to 90 percent of the time. Guo et al. (2017) sort predictions into buckets by confidence and plot each bucket's average confidence against the share of it that proved correct. Points below the 45-degree line indicate overconfidence, summarized as expected calibration error (ECE). Calibration matters here because the threshold in Section 3 depends on a real rate of error, not just a measure of how certain the model reports being.

Defining High-Value Work

In alignment with the classic economics framework of Marginal Revenue Product of Labor (MRPL), high-value human work may be defined as work for which a person's expected contribution, measured in dollars, exceeds the cost of that person's time. For a single decision inside an automated workflow, that contribution is the loss a reviewer prevents, so review is warranted when

P(error) × Impact(error) > Cost(review)

Work failing this test is a candidate for automation or for exclusion from the pipeline. Applying the test in practice requires two criteria.

Criterion 1: Expected value. In the base case each term is directly observable. Impact is the value of the credit at stake, C. The probability of error is one minus the agent's confidence q that its disposition is correct, so P(error) = 1 − q. Review cost is the cost R of the analyst time required to check it. Review is therefore warranted when

(1 − q) × C > R

For the majority of credit line items this comparison is sufficient and the resulting threshold is straightforward to compute.

Criterion 2: Asymmetric downside. The expected-value calculation above assumes the cost of an error is symmetric with the benefit of a correct decision. In recovery workflows, this assumption does not always hold. Consider a credit that the agent incorrectly determines has already been collected and therefore closes without escalation. The recoverable balance is forgone in full, and where the vendor is a significant supplier the same misclassification is likely to recur across subsequent statements, compounding the loss and introducing friction into the commercial relationship. The opposing error, escalating a credit that proves invalid, consumes several minutes of analyst time. The cost of a false negative is thus neither equal nor opposite to the benefit of a true positive.

The asymmetry can be made explicit by separating the face value of a credit from the share of it actually lost when the agent makes a mistake and no person intervenes:

Impact(error) = s × C, where 0 < s ≤ 1

Here C is the credit's face value and s is a severity factor governed by reversibility. This calculation reflects the mechanism by which errors do or do not surface. A silently closed credit carries s near 1 precisely because nothing downstream catches it, which is why such balances are typically discovered only by audit. An invalid credit that is posted behaves in the opposite way: the vendor disputes it and a future reconciliation reverses it, so the realized loss is a fraction of face value and s is small.

Introducing severity allows the review threshold to vary by error type. Silent errors, which are unlikely to be detected or corrected downstream, should trigger human review at lower credit values. Reversible errors can tolerate higher values before requiring review because much of their potential impact will be caught and corrected later. This directs human review toward errors where intervention matters most.

Together, the two criteria yield a threshold. Under a fixed review budget, they also provide a way to rank, which matters more in practice. Assuming analyst capacity is held constant, the real question is not whether a given credit merits review in isolation but which credits receive the minutes available. Ranking decisions by expected loss, that is by P(error) × s × C, and allocating capacity in that order produces the threshold as a by-product: it falls where the budget is exhausted. The framework therefore functions as a prioritization rule as much as a cost test, directing scarce expert attention toward the decisions that are simultaneously least certain and most consequential.

Application: Recovering Unclaimed Credits at Media Co.

Media Co. operates a procure-to-pay function in which credits are typically only recovered if vendors proactively send a credit memo. The manual review of vendor statements to identify credits is simply too time-intensive for the team to conduct at scale, so credits are rarely identified actively today.

The agentic system developed for this workflow ingests each document, classifies vendor statements and credit memos as relevant compared to other documents, extracts credit line items from vendor statements, and returns one of two dispositions: the “happy path” that expedites the posting of a credit or the manual path that escalates to an analyst for deeper review. Across one month of pre-deployment testing, the agent surfaced several times more recoverable credits than the existing process had found, with an end-to-end disposition accuracy of at least 92 percent across classification, extraction, and reconciliation.

The substantive contribution of this article is the mechanism governing escalation within this workflow. The boundary for confidence is risk-adjusted, not fixed: as the financial impact of an error increases, the confidence required for autonomous action also increases. The rest of this section describes how that boundary is computed and what it produces on the tested credits.

The agents themselves are developed and validated through evaluation, but even a well-evaluated LLM-based system remains probabilistic, and aggregate evaluation performance cannot determine whether any individual output is correct. The confidence mechanism described here operates on top of those evaluations, providing a runtime estimate of uncertainty for each disposition. Because a model's self-reported confidence is unreliable, as Section 2 describes, the system proceeds in three stages, shown in Figure 2.

First, confidence is decomposed into distinct signals corresponding to the points at which the process can fail: whether source data was read correctly, whether the correct enterprise records were retrieved, and how strongly the reasoning supports the disposition. Second, those signals are combined into a single score and calibrated against observed outcomes, with the target that a score of 0.9 corresponds to empirical correctness of approximately 90 percent. Third, two thresholds divide calibrated confidence into three operational bands: high-confidence dispositions proceed with low touch, low-confidence dispositions are reprocessed or escalated, and cases in between are sent for human review. The thresholds balance the cost of analyst review against the expected cost of an incorrect decision.

We demonstrate evaluating the boundary at the observed accuracy of 92 percent to illustrate its effect on the tested credits. We assume a review cost of $150, estimated as the opportunity cost of an analyst’s time that could have been spent identifying the next credit. This figure is deliberately not a salary-based cost. A fully loaded hourly rate would price the analyst's hour far lower; what is relevant here is the value of the recovery that hour displaces. The $150 value is used to illustrate the tradeoff between manually reviewing and prioritizing high-potential cases based on the agent’s analysis, and can be replaced with an organization-specific estimate in practice.

Combining the two criteria from Section 3 gives the condition for review as (1 − q) × s × C > R. Solving for the credit value at which review becomes warranted:

C > R / ((1 − q) × s)

Evaluated at the observed population accuracy q = 0.92, a review cost R = $150, and s = 1 for an irreversible error:

C > $150 / (0.08 × 1) = $1,875

At that confidence, cost, and severity, the agent would escalate credits worth above $1,875 for an analyst to review. At 98 percent confidence, the threshold would be $7,500; at severity s = 0.1, the threshold would become $18,750.

Illustrative. Cost, confidence, and severity values are assumption-driven.

Two aspects of the evaluation warrant description. Hand-labeled reconciliation data is costly to produce, and the labeled corpus available was chosen to be augmented with synthetic data in order to expand effective sample size and to achieve coverage of failure modes that occur infrequently in production. A taxonomy of failure cases was constructed, and real statements were perturbed along those dimensions to characterize where and why the agent degrades. Separately, the system was stress-tested against unresponsive APIs, database dependency failures, and rate limiting. Under these conditions the system logs its current state, retries failed calls, and paces vendor inquiry ingestion so that volume spikes do not propagate into failure.

Generalization Beyond Accounts Payable

Accounts payable is a convenient setting for this work, but the framework does not depend on its specifics. Other workflows in which automation reduces the marginal cost of a task will exhibit the same expansion in the volume of work worth performing, and will pose the same question regarding the placement of human review. Adjacent workflows including order-to-cash recovery, procurement review, and tax and coding validation share the relevant structure: high transaction volume, material financial value concentrated in exceptions, and errors whose costs are unevenly distributed. In each case the boundary can be established by the same two criteria: weighing the expected value of review against its cost and adjusting for asymmetric downside.

One further property generalizes. Human review capacity is slow to adjust in either direction. It is added one hire at a time, requires months of training and accumulated institutional knowledge, and is paid for in quiet periods as well as busy ones. Agent capacity is elastic, scaling from a handful of documents to many thousands without a corresponding change in cost structure. A framework that prices review correctly will therefore tend to recommend a human team aimed at the highest-severity decisions, with the agent absorbing the volume and the variance that a fixed team cannot economically carry.

Conclusion

The reduction in the cost of knowledge work produced by agentic systems expands the volume of work worth performing and shifts the strategic question from the extent of automation to the placement of human judgment within it. This boundary cannot be determined by accuracy alone, because the value of human review depends on both the probability of an error and the consequence of that error. The framework developed here locates the boundary based on calibrated confidence, which is evaluated against the financial consequences of error and converted into a dollar threshold at which the agent would escalate. The result is a system that directs human attention toward the decisions in which it creates the greatest value, rather than applying a uniform review threshold.

Work continues with Media Co. on the two pieces this depends on most: refining the calibration of the confidence score against production outcomes, and extending coverage beyond the initial vendor population. The same framework should then be applied to adjacent processes.

References

Chow, C. K. 1970. "On Optimum Recognition Error and Reject Tradeoff." IEEE Transactions on Information Theory 16(1): 41–46.

Dadopoulos, M., and Moschidis, S. 2026. "Beyond Fuzzy Matching: A Dual-Augmentation RAG System for Robust Product Reconciliation in Accounting." Journal of Risk and Financial Management 19(6): 402.

Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. 2017. "On Calibration of Modern Neural Networks."  Proceedings of the 34th International Conference on Machine Learning: 1321-1330.

Jamithireddy, N. H. 2025. “Robotic Process Automation in SAP ERP: Enhancing Financial Transaction Reconciliation and Compliance Monitoring.” Archives for Technical Sciences 33(2): 279–296.

Jevons, W. S. 1865. The Coal Question. London: Macmillan.

Lin, Z., Trivedi, S., and Sun, J. 2023. "Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models." arXiv:2305.19187.

Malla, P. 2025. “A Scalable Enterprise Framework for AI-Driven Invoice Processing Using Document Intelligence.” International Journal of AI, BigData, Computational and Management Studies 6(4): 181–186.

Simcox, L. 2026. “From AP Automation Software to Cloud ERP: How Finance Leaders Scale More Intelligently.” SAP.

Varun, K. G., and Islam, M. A. 2026. “Turning Invoice Compliance into a Scalable, Auditable Workflow with Private Agent Factory.” Oracle Database Insider.

Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. 2024. "Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs." International Conference on Learning Representations.

Thank you Ryan Kappedal, Shreyas Subramanian for reviewing and proving feedback.