Plausible Is Not Representative: The Quality Control Your AI Deployments Are Missing

Your teams feed 40,000 customer verbatims into a model and get back a clean list of the top five objections. The list is coherent, well phrased, ready to drop straight into a board deck. Nobody in the room knows whether it is representative. That is precisely where the move from pilot to governed production is decided: not in the choice of model, but in the ability to demonstrate that its output matches the input data.

What "plausible but not representative" looks like in a real case

A French insurer has the verbatims from its annual satisfaction survey analysed. The model returns five drivers of dissatisfaction, all stated with equal confidence. A manual recount on a sample tells a different story: the first driver appears in roughly a third of responses, the fifth in fewer than 2%. Both are presented as equals. Part of the quarterly product roadmap is arbitrated on an objection that barely exists.

Nothing was invented. The model did not hallucinate a driver that was absent — it promoted one to the status of a trend. That is exactly what a system trained to produce a satisfying answer will do. Anthropic's work on model sycophancy (Sharma et al., 2023) shows that five leading assistants exhibit this bias on open-ended generation tasks, and that both human evaluators and preference models pick a convincing but wrong answer over a correct one in a non-trivial share of cases. A summary is a narrative task; a frequency is a counting task. Conflating the two is the first crack in the system.

Having AI check AI: what the measurements actually support

The instinctive fix is to add a second model as a judge. The approach is sound, within documented limits. The reference study on the subject (Zheng et al., NeurIPS 2023) establishes that a model judge reaches over 80% agreement with human evaluators — the same level humans reach with each other. The same study catalogues the biases that come with it: position bias, verbosity bias, and a preference for its own output.

The scale of those biases has since been quantified. An analysis covering 15 judges, some forty evaluated models and more than 150,000 verdicts (Shi et al., arXiv:2406.07791) concludes that position bias is not random noise, that it varies sharply from one judge to another, and that it worsens when the candidates being compared are close in quality. Put plainly: the automated judge is least reliable in exactly the borderline cases that justified deploying it.

A benchmarking preprint published in April 2026 and accepted at the AIED 2026 conference shows what that means in practice. Three frontier models grade the same body of scientific work: two of them agree almost perfectly (Spearman's rho of 0.907), the third comes apart entirely (rho of roughly 0.32). Same corpus, three judges, two incompatible verdicts. A single-judge setup would have produced a ranking, a decision, and no trace of the uncertainty.

Six steps to put this quality control in place

1. Separate classification from narration. Have each unit classified against a closed taxonomy you have defined, then aggregate the codes in SQL. The model handles one verifiable decision at a time; the count comes out of a database, not a paragraph.
2. Build a hand-annotated reference set. Draw 200 to 300 units at random and have two people from the business annotate them independently. This is your ground truth; without it, no quality measurement exists.
3. Measure agreement, not satisfaction. Calculate agreement between your two human annotators, then between the model and the ground truth. The Landis and Koch (1977) scale remains the usual reference: kappa between 0.61 and 0.80 for substantial agreement, above 0.81 for almost perfect agreement.
4. Use two different model families as judges. A model should never grade its own output. When the two judges disagree, the case goes to human review; the disagreement rate becomes your grey-zone indicator.
5. Appoint an independent validator with the power to stop. Banking has done this since 2011 under the name effective challenge (SR 11-7, Federal Reserve): critical challenge by a competent party with no reporting line to the team that built the model. Apply the same rule to generative AI output.
6. Re-run the sample on every change. New model, new version, revised prompt: the reference set goes through again. It is regression testing, exactly as it would be for software.

France's data protection authority makes a similar recommendation for the annotation phase, calling for a documented, continuous verification procedure including "analysis of random samples of annotated data" (CNIL, AI: annotating data). The principle applies identically to production output.

Mistakes to avoid

• The single judge. One evaluator model, often from the same family as the generator. You are measuring its internal consistency, not its accuracy.
• The flattering sample. Reviewers go back over the cases the system handled well and never the tail of the distribution, where the risk and the complaints actually sit.
• The average score. An overall 4.2 out of 5 that hides the spread and connects to no operational decision.
• Zero challenge. A 0% human correction rate does not signal a flawless system; it signals a validator signing without reading. Article 14 of the EU AI Act has a name for this: automation bias.
• Review without a mandate. A reviewer with no time, no authority and no button to reject an output is not a control — it is a signature.

The indicators that prove the control works

Four markers belong on the monthly committee agenda: model-versus-human-ground-truth kappa, with its acceptance threshold written down; the disagreement rate between judges, which sizes the human review workload; the human correction rate, healthy between 5% and 20%, alarming at 0% as much as at 40%; and the median time between a drift and its detection. Add the cost of control as a share of inference cost: above 30%, the use case deserves a fresh look.

What the regulatory calendar changes

Article 14 of the EU AI Act requires that the person overseeing a high-risk system be able to correctly interpret its output, disregard or override it, and halt the system. The simplification package adopted by the Council on 29 June 2026 pushes back the obligations for Annex III systems from 2 August 2026 to 2 December 2027, and those for Annex I to 2 August 2028 (Gibson Dunn, 2026). The Article 50 transparency obligations, however, stay on the 2 August 2026 deadline.

Sixteen extra months is one budget cycle, not a reprieve. Building a reference set, accumulating an agreement track record and documenting stop decisions takes precisely that long. Organisations that start in 2027 will file a case with no history behind it.

Where to start

Take the most advanced AI use case in your organisation. Ask its owner three questions: which hand-annotated sample serves as the ground truth, what agreement rate was measured on that sample last month, and who has the authority to stop production. If all three answers do not arrive within one meeting, you do not have a production deployment; you have a pilot that grew.

---

TokenShift works with European executive teams on exactly this control layer: defining the reference set, designing the inter-judge agreement protocol, and building an oversight dashboard fit for the audit committee.