Back to Insights
AI Governance7 min read

When AI Sells Work, the Right KPI Is the Accepted Case

TokenShift Executive Note

When AI Sells Work, the Right KPI Is the Accepted Case

An assistant can impress with a good answer. An AI tasked with a share of the work has to deliver a case the business can accept, defend and audit. As offerings shift from copilot to executed work, the COMEX needs to change its unit of measure: fewer answers produced, more cases accepted without rework.

The Work Being Sold Changes the Unit of Control

In an essay published on 5 March 2026, Julien Bek of Sequoia Capital frames the shift this way: some AI players will no longer sell the tool, but the work done. His contrast between copilot and autopilot is an investor's thesis, not proof that autonomy is already reliable in every line of work. It does, however, point to a useful shift for executives: the customer is no longer judging software alone; they are judging an operational outcome (Sequoia Capital, 2026).

For a bank, an insurer or a payment provider, that outcome may be a prepared fraud case, an investigated complaint or a documented regulatory control. The accepted case is not necessarily a file. It is a complete unit of work: required data present, applicable rule identified, conclusion supported, exception flagged and action authorised.

This distinction also protects ROI. An agent can produce twice the output and create more rework. Volume rises; useful work falls.

A preprint by Deep Mehta illustrates this risk at experimental scale. Across 140,000 generations, spanning 7 question-answering tasks, 5 families of prompt formats and 4 models, average sensitivity to format varied by more than 30-fold depending on the model. The ability to extract the answer correctly was strongly associated with measured accuracy (arXiv, 2026). The study is limited to four models and one benchmark protocol; it does not measure an enterprise failure rate. Its operational lesson is narrower: an output can look correct to a human reader and remain unusable by the next system in line.

The Acceptable Case Method

The Acceptable Case turns a demo into an operating contract. It has five tests, applied before the model is finally chosen.

1. Define acceptance before the prompt

Start with a five-column grid: expected outcome, mandatory evidence, applicable rule, grounds for rejection and validation authority. Every case must be acceptable or rejectable with a stable reason code.

For a fraud alert, for instance, "relevant summary" is too vague. "Transaction IDs cited, period covered, internal rule and version, conflicting elements, authorised recommendation" is verifiable.

2. Require a chain of reasoning

A final answer easily hides a shortcut. The case must separate:

  1. the recommended proposal or action;
  2. the facts that support it;
  3. the rule that links those facts to the proposal;
  4. the level of reservation;
  5. the elements that could rebut it.

This structure follows the Toulmin model. A 2026 preprint applies it to a retinal-diagnosis support architecture, distinguishing claim, evidence, warrant, qualifier and rebuttal (Marginean and Groza, arXiv, 2026). It is a research proposal, not a clinical validation. For the enterprise, the value is architectural: forcing the system to hand back the parts of its reasoning, not just its conclusion.

3. Separate recommendation from action

The level of autonomy should depend on impact and reversibility. A correctable internal classification can be executed within a defined range. A customer rejection, a payment or a decision affecting rights calls for validation backed by real authority.

For the high-risk systems concerned, Articles 12 to 14 of the EU AI Act require, among other things, automatic event logging, information that makes outputs interpretable and effective human oversight. The person in charge of oversight must be able to disregard, override or reverse an output, and even stop the system (European Union, Regulation 2024/1689). Not every enterprise use is high-risk; even so, these requirements remain sound design criteria for consequential work.

4. Test the variations that break the workflow

Do not validate a single prompt on clean cases. Vary the order of the items, the length of the case, the requested format and the presence of a missing field. Introduce an outdated rule, two conflicting sources and an unreadable document.

The expected behaviour must be defined before the test: refuse, request the item or escalate. Silently filling in a missing piece of information should count as a failure, even if the resulting sentence looks plausible.

5. Measure the accepted outcome

The dashboard should bring quality, cost and time together:

  • first-pass acceptance rate;
  • minutes of human rework per accepted case;
  • completeness rate of evidence and rule versions;
  • number of actions corrected after execution;
  • rate of justified escalations;
  • median time and cost per accepted case.

Measure these indicators on the existing workflow before deployment. Otherwise, a model improvement will be confused with a change in volume, team or policy.

Mini-case: Tuesday morning's fraud alert

Take the fictional case of a payment institution. Its agent gathers transaction history, checks internal rules and prepares a recommendation for the compliance analyst.

In the first pilot, the team tracks the number of alerts processed. The summaries read smoothly, but the analyst has to reopen several applications to find the transactions cited. The rule version does not appear. When an item is missing, the agent produces a conclusion anyway. Throughput looks good; the accepted case stays rare.

The workflow is then redesigned. Each proposal contains the source IDs, the rule version, the missing information, a conflicting element and the escalation reason. The agent cannot close the alert. If two rules conflict or a mandatory item is missing, it hands back an explicitly flagged incomplete case.

After a measurement cycle, five markers decide the move to production: first-pass acceptance up on the baseline, rework time down, no closure without evidence, relevant escalations holding steady and end-to-end time reduced. If only the number of summaries goes up, the pilot has not improved the work.

Mistakes to Avoid

The autonomy-rate fetish

Chasing the highest possible share of cases handled without intervention encourages the agent to escalate less. Also measure the exceptions it should have raised.

The JSON false positive

A valid schema guarantees neither the right rule nor the right evidence. Format compliance is a technical check; business acceptance is a separate one.

The human rubber stamp

A validator with no time, no items and no power to stop is not a control. Measure their corrections, their rejections and the time they actually have.

The orphan log

Keeping the prompt and the answer is not enough. Without source IDs, rule versions, triggered actions and override decisions, the case cannot be reconstructed.

The Decision to Put on the Next COMEX Agenda

Ask for one page per workflow: unit of work, acceptance grid, baseline, six KPIs, escalation thresholds and the owner of the business decision. Have that page tested on historical cases—normal, incomplete and conflicting. Authorise production when the quality and consequence thresholds are met, not when the demo looks smooth.

Procurement should carry the same criteria into the specification: accepted-case quality, rework, traceability, incidents and change procedure. If a vendor sells the work, a per-user or per-token price is no longer enough to steer its performance.

An AI that produces a lot is not productive. It becomes productive when the business accepts its work without having to rebuild its evidence, its rules or its effects.

Sources

Continue reading

View all insights