Back to Insights
AI6 min read

Local AI: Keeping Data on the Device Is Not Enough

TokenShift Executive Note

Local AI: Keeping Data on the Device Is Not Enough

The most important signal this week is not a new performance record. It is a shift in where execution happens: models capable of processing text, images or audio are moving from the data centre to the device. For an executive committee, the question becomes very concrete: who decides what runs where, with which data and with what evidence?

Local AI is becoming a genuine architectural option

On 13 July 2026, Google introduced Gemma 4 E2B running natively on the Pixel 10 TPU, with offline demonstrations across text, images and audio. On 14 July, PrismML stated that its 1-bit version of Bonsai 27B occupied 3.9 GB, versus around 54 GB at 16-bit precision, and reached 11 tokens per second on an iPhone 17 Pro. Google Developers Blog, 2026, PrismML, 2026

These results come from the suppliers and must be replicated on the organisation’s own tasks. Their convergence is nonetheless meaningful: local AI is no longer limited to spell-checking or basic classification.

It does not replace the cloud. It adds an execution zone.

| Zone | Good candidate | Trade-off to manage | | --- | --- | --- | | Device | Sensitive data, bounded task, offline use | Heterogeneous device fleet, updates, fragmented logs | | Private infrastructure | Shared business context, central rules, audit | Capacity, operations, fixed cost | | External cloud | Complex task, elastic capacity, advanced model | Transfers, variable cost, supplier dependency |

Keeping data local is not governance

A review published in 2026 in Expert Systems with Applications reaches a useful conclusion: federated learning keeps raw data local, but does not solve model opacity. The authors also note the absence of standardised measures for the quality of explanations, their stability, information leakage and computational overhead. Federated learning is not local inference, but the warning applies: moving computation is not enough to control the decision. Gholizade et al., 2026

The EDPB recalled in February 2026 that data protection by design is a continuous obligation, with default minimisation and regular reassessment. For high-risk AI systems, the European regulation notably requires automatic logging, human oversight, and accuracy, robustness and cybersecurity in Articles 12, 14 and 15. EDPB, 2026, European Union, Regulation 2024/1689

Local execution can therefore reduce one exposure. It does not itself provide a legal basis, human oversight or an audit trail.

The Governed Routing method

Governed Routing is a verifiable policy that assigns each stage of a workflow to an execution zone. It is built in five steps.

  1. Break down the workflow. Separate intake, extraction, matching, recommendation, decision and execution. A single case can use three zones without one application being declared entirely local or cloud-based.
  2. Classify each stage. Document the data category, business impact, reversibility and required latency. A high-impact local task still requires oversight; a cloud task using minimised data may be acceptable if its purpose and safeguards are established.
  3. Set the route and fallback. For each stage, authorise the device, private infrastructure or cloud. Specify permitted data, prior transformation, the model and its version, the confidence threshold and offline behaviour. A fallback to the cloud must never be implicit.
  4. Test on real work. Build a versioned dataset representative of languages, devices, customer segments and difficult cases. Measure errors by field, refusals, human corrections and disparities across populations. A supplier’s average score does not validate your workflow.
  5. Assign ownership and instrument. The business owner is accountable for the outcome; IT for execution; the risk function validates controls; security manages updates and revocation. Each hand-off records the route, version, data class, output status and human intervention, with proportionate retention.

Mini-case: a motor claim split across three zones

At a fictional European insurer, Claire leads claims operations. The mobile application locally analyses photos and voice comments, extracts the useful fields and masks identifiers. Private infrastructure then matches those elements against the policy and business rules. Only ambiguous, pseudonymised narratives are sent to a cloud model; no rejection or payment is executed without the claims handler’s approval.

The pilot dashboard must make the following observable:

  • the share of cases for which no raw data left the device, by data class;
  • the rate of fallback to the cloud and its reason;
  • the rate of human correction by field, language and device;
  • the 95th-percentile turnaround time and cost per completed case;
  • log completeness, route violations and the rate of reopened cases.

The outcome is achieved only if turnaround time or cost falls without an increase in corrections, disputes or policy deviations.

Mistakes to avoid

  • The false local model: the model runs on the device, but telemetry, backups or attachments send raw data upstream.
  • The silent fallback: low confidence triggers a cloud call without filtering or traceability.
  • The average benchmark: overall accuracy masks errors for one language, device or population.
  • Compliance by hosting location: staying on the device or within the Union does not address purpose limitation, individuals’ rights or human oversight.
  • Token economics: comparing API price alone ignores the device fleet, updates, support and fallbacks. The right denominator is the completed case.

Decision for the next executive committee

Choose one workflow, require its routing map on one page, appoint its business owner and approve the fallback rules. Then establish a pre-pilot baseline for turnaround time, cost, quality, incidents and human interventions.

Data that does not leave the device reduces an exposure; it creates neither accountability, nor evidence, nor control.

If the team cannot reconstruct a case’s route and explain why each item of data crossed a boundary, it does not yet have governed local AI.

Sources

Continue reading

View all insights