Scaling an AI pilot: the evidence to look for in real work
TokenShift Executive Note
AI-translated from the original French version.

News of more capable models can be a good reason to review an AI pilot again. On its own, though, it does not show that the company can take more cases through to closure. Before approving a wider rollout, the executive committee should ask three things: what is improving in real work, what is still blocking it, and who controls the actions the system takes.
Measure the completed case
Take a hypothetical example from insurance. An assistant drafts a reply to a request to amend a policy and proposes an update to the CRM. The demonstration may end with the drafted reply. The actual work also involves checking which policy applies, chasing missing documents, authorising the amendment and recording it.
The business owner should define what a correctly completed case looks like, then compare the current process with the assisted process on that basis. If a case handler has to track down the information the system used or re-enter its output, that rework belongs in the pilot's assessment.
Feeling faster does not prove a gain. In an experiment published in 2025, METR reports that 16 experienced developers took on average 19% longer to complete 246 tasks with the AI tools studied, even though they believed they were saving time. The finding concerns software development, not insurance. An update published in 2026 also highlights how difficult it is to measure the effect of more recent tools. The lesson for the executive committee is one of method: measure completed work in its own process, without carrying over the result of this study.
Find the cause of rework before changing model
In the hypothetical example, a reply that needs correcting may stem from a misreading of the policy, an out-of-date document, a duplicate customer record or an ambiguous approval rule. These causes do not call for the same investment.
The analyses by HubSpot on CRM modernisation and by Christophe Bouron on AI adoption in mid-sized companies draw attention to data, working practices and the choice of targeted use cases. They set out approaches; they do not demonstrate the return of the pilot in question.
The team can then review the cases that needed rework and attribute each intervention to its cause. A document that cannot be found points to document access; an approval with no owner points to the organisation; a misreading despite a complete file points to a fresh comparison of models. It is the observed blockage that gives the technical evaluation its meaning.
Prove that actions remain under control
NVIDIA presents OpenShell as an environment designed to govern the actions of agents. Its policy documentation describes, among other things, restrictions on access to files, network destinations and requests. This technical capability deserves a look if the pilot hands actions to an agent; it proves neither that the deployed configuration is correct nor that the use complies with the applicable obligations.
In the hypothetical insurance case, blocking a write to an unauthorised application is useful. Allowing a write to the CRM, however, does not prove that the amendment complies with the policy. The executive committee needs to see both checks: tests showing which actions are actually blocked or permitted, and business controls on the proposed amendments. It also needs to know who takes over an ambiguous or interrupted case. Without an available owner, the control results in a wait, not a completed case.
Decide on the result and its full cost
The assessment should relate the costs of the tested scope to the cases completed and accepted, counting abandoned attempts, corrections and human rework. The AI service, operations and integration all belong in this calculation; the initial investment and the recurring cost are best presented separately.
This cost should be read alongside the end-to-end lead time, errors found after closure and the share of cases that needed rework. Requests of comparable difficulty must be compared: a good result on simple cases proves nothing about exceptions. Minutes freed up may create capacity, but they become a budget saving only if some expenditure falls. The expected benefit must therefore be named and observed.
If the work improves, its cost is acceptable and the actions are under control, a bounded extension can be considered, with an owner and suspension criteria. If a piece of evidence is missing, the priority is to establish what is blocking the chosen use case before funding a new model, a process redesign or new infrastructure. The pilot can stay within its current scope while this is checked.