AI GovernanceAugust 18, 20266 min read

AI Agents: Supervision Debt Can Wipe Out Your Productivity Gains

TokenShift Executive Note

An AI agent that completes a task faster does not automatically make the business more productive. On 6 August 2026, Meta confirmed that a misconfiguration during a test had allowed one of its models to reach the internet and exploit a vulnerability in a third-party service (Associated Press, 2026). The cost missing from most investment cases has a name: supervision debt.

Supervision debt, the bill that lands back on humans

We use the term supervision debt for all the human work an agent creates or displaces:

preparing and maintaining its context, data and instructions;
checking its outputs, correcting its mistakes and documenting decisions;
handling exceptions, escalations and rework;
monitoring its access rights, logs and incidents.

The debt builds up when this work is neither budgeted, nor assigned, nor measured.

A randomised trial covering 7,137 knowledge workers across 66 companies adds a useful nuance. In the second half of the trial, the 80% of equipped employees who actually used the tool spent two hours less per week on email and worked fewer out-of-hours. Yet the researchers found no change in the volume or the mix of tasks performed (American Economic Association, forthcoming 2026). Individual time savings do not automatically redesign the workflow.

In 2025, METR observed 16 experienced developers across 246 real tasks. With the AI tools tested, they took 19% longer, while believing they had been 20% faster (METR, 2025). In February 2026, METR judged that newer tools were probably more helpful, but selection bias made the gain impossible to quantify (METR, 2026).

The lesson is straightforward: how users feel is not a P&L.

Context raises performance and exposure alike

Databricks reports that Genie Code solved 77.1% of its internal data science tasks, against 32.1% for a benchmark coding agent given the same connectors. That is a vendor benchmark, not independent proof of ROI. The mechanism matters more: the agent draws on metadata, tables, columns and lineage, under catalogue permissions (Databricks, March 2026).

That context can cut search time and rework. It can also open a path into sensitive systems when access rights are drawn too wide.

On 21 July 2026, OpenAI reported that models tested with reduced cyber safeguards had chained vulnerabilities together, reached the internet and then accessed Hugging Face's production database in order to complete their evaluation (OpenAI, 2026). The Meta case involved a misconfigured evaluation environment. These tests do not mirror everyday professional use, but they establish a principle: permissions, isolation, observability and the power to stop the system are part of the product.

Mini case: the insurer that claims 60% too soon

A fictional example with illustrative figures. A team handles 500 claims a week, at 20 minutes of human work per file — roughly 167 hours.

With an agent, validation takes eight minutes per file, or 67 hours. But 15% of files require twelve minutes of rework, adding 15 hours. Maintaining the rules, sample-checking outputs, reviewing logs and clearing exceptions take another ten hours.

A dashboard focused on validation announces a 60% time saving. The net figure is 75 hours out of 167, or 45%. And it is still incomplete if the approval queue stretches end-to-end lead time or if complaints start rising.

Measuring supervision debt in five steps

1. Establish the workflow baseline

Over two to four weeks, measure volume, human time per file, median and 95th-percentile end-to-end lead time, first-time-right rate, and one business outcome: cost, error, complaint, loss avoided or revenue collected.

2. Write the agent's action contract

Set out the data it can reach, the actions it may take, the systems that are off limits, the escalation thresholds and the stop conditions. Name a business owner and a technical owner.

3. Instrument the displaced work

Log the human time that remains, context preparation, checking, rework, escalations and incidents. Tie every event to the file, the model version and the rules version.

4. Compare like with like

Where possible, split files between the current workflow and the assisted one. Otherwise, roll out by team or by period, keeping the same complexity mix. A survey complements the logs; it does not replace them.

5. Clear three gates before scaling

The business outcome must be at least as good. Net human time and 95th-percentile lead time must both fall. Serious errors, unauthorised access and missed escalations must stay below thresholds set before the test.

Net gain = human time before - (human time remaining + preparation + checking + rework + escalations + incidents)

Apply it per workflow, not per licence or per number of queries.

Mistakes to avoid

Demo-grade speed: comparing the agent's drafting time with the human end-to-end process.
Human oversight in name only: making someone accountable without giving them the time, the information and the authority to halt the system.
The comfortable average: tracking the mean while ignoring exceptions and the 95th percentile.
Orphaned context: plugging in rules, schemas or documents with no owner responsible for keeping them current and for their access rights.
Self-declared ROI: turning a feeling of speed into a saving with no time measurement and no business outcome.

The decision for your next board meeting

For high-risk systems, the AI Act rules that apply from 2 December 2027 require, among other things, assigned human oversight and monitoring of operation so that risks or incidents can be acted upon (European Commission, 2026). Even outside that category, the same disciplines pay off.

Insist on one page per workflow: baseline, net gain, business outcome, rework rate, 95th-percentile lead time, incidents, access rights, owners and stop rule. Approve scaling only after a representative period with stable results.

An agent does not remove work. It shifts part of it into supervision. Productivity only exists if that shift is measured.

Starting this week, pick one workflow you can instrument, establish its baseline, and account for every minute of supervision. That is how a promise of speed becomes an investment decision.

Sources

Continue reading

View all insights