AI evaluation: A good score can conceal a flawed method

Imagine an insurer that wants to help its claims handlers identify claims needing expert assessment as soon as a file is opened. To test its AI, it uses old claims files whose outcomes are known. The score looks convincing.

But the dataset also contains information added after the expert assessment. If the AI uses it, the test gives it information the claims handler will not have when making the decision.

This example is hypothetical. It shows why a good score alone is not enough to authorise deployment: the insurer needs to understand how the score was obtained and what it can actually establish.

An AI that already knows how the story ends

In our example, the insurer is trying to anticipate the need for expert assessment. By giving the model information gathered after that assessment, it makes the task artificially easy. The test no longer reflects the work the model is expected to do.

This is a form of “data leakage”. Here, the term does not mean that information has been stolen. It means that data the model should not be able to access influences its development or evaluation. The scikit-learn documentation explains how this flaw can produce an overly optimistic estimate of performance. scikit-learn, documentation accessed in October 2026.

A preprint from Centrum Wiskunde & Informatica, submitted on 9 October 2026, illustrates the problem. The researchers studied AI agents carrying out data science work across 220 tasks. In the configurations examined, some forms of data leakage were associated with higher predictive scores. A flawed method could therefore appear to perform better. This work does not measure how often the problem occurs in companies. CWI, 2026.

For the insurer, the check involves reconstructing each claims file as it stood when it was opened. It must separate what was known at that point from what was learned later. The date the dataset was extracted does not answer this question: a dataset extracted today may combine several stages in the life of a claim.

Until the test respects that chronology, the score cannot show whether the tool will help claims handlers with new files.

Being able to trace what was tested

Now suppose the insurer has corrected its dataset and run the evaluation again. The committee receives a presentation reporting “satisfactory quality”. To assess that result, it needs to be able to identify the claims files examined, those excluded, the definition of a correct answer and the version of the tool used.

Without these details, it remains difficult to distinguish a sound test from an incomplete one.

The same difficulty arises in research. In a study published in the BMJ on 13 October 2026, David Chen and his co-authors examined, among other things, 68 reports of clinical trials using large language models. They found missing information about model versions, settings and how input and output data were handled. Chen et al., 2026, University of Birmingham record.

A poorly described method is not necessarily a bad one. But missing information prevents a proper assessment. These findings concern clinical research; they do not provide a failure rate that can be applied to insurance or industry.

In our example, we recommend having the test reviewed by a competent person who is separate from the team improving the model. That person must be able to trace the data, instructions, documents and tools used, then request another test if necessary. The choice of supplier should therefore also take account of access to these records and how the supplier reports changes to its system.

The committee can fund this verification. It simply needs to keep the distinction clear between a validated result, an observed failure and a question that remains unanswered.

Track the consequences through the teams’ work

Once the test can be checked, the question remains: what does its score mean for claims handlers?

In our example, the AI may miss a claim that needs expert assessment. It may also send a straightforward claim to an expert unnecessarily. These errors have different consequences. Combining them in an overall score conceals the trade-off the business owner must accept.

The evaluation should therefore report important claims missed and expert assessments requested in error separately, along with the extra work those errors would create. It should also show which incomplete claims files were excluded from the test. If they were not evaluated, the insurer needs a way to identify and handle them differently.

The same logic applies to the economic benefit. Faster sorting would not demonstrate a net saving if experts received more unnecessary referrals. The finance director needs to compare the time saved with the cost of preparing data, human checks, corrections and running the tool.

Time freed up may allow teams to handle more claims. It does not, by itself, reduce expenditure.

Authorisation can then be specific: the tool may suggest a priority order to claims handlers, using only information available when a file is opened, for the claim types and teams covered by the test. The claims handler retains the decision. This approval would not, for example, permit the same tool to be used to deny coverage.

The business owner accepts this scope, the technical team describes the version evaluated, and the person responsible for validation records the test’s limitations. If evidence is missing, the next decision may be to fund its production. Any expansion of use will wait for the results.

Sources

• Centrum Wiskunde & Informatica, Xue Li et al., 9 October 2026. The Silent Failures in Agentic Data Science, preprint.
• David Chen et al., BMJ, 13 October 2026. Reporting completeness of clinical trials evaluating large language model interventions in medicine: systematic review, record and abstract on the University of Birmingham website.
• scikit-learn, documentation accessed in October 2026. Data leakage and its consequences for evaluation.