Open-weight AI model: what evidence is needed before migrating a process?

Meta’s announcement on 15 October 2026 presents Muse Spark Open as a model with downloadable weights, also available through several inference providers. It creates the option of choosing another operator. It does not prove that moving an existing process would be beneficial. Meta, 2026.

To authorise a migration, a business leader must start with the process in question: what benefit is expected, what service must be maintained and under what conditions could the company revert? The alternative must be tested within that scope.

An alternative must be usable and practical to operate

Downloadable weights make it possible to consider running the model on infrastructure of the company’s choosing. That freedom first depends on the rights attached to the intended use. Under the licence associated with Muse Spark Open, a separate agreement with Meta is required when the licensee or its affiliates provide models as a service and their combined revenue exceeds USD 20 million over twelve consecutive months. The text distinguishes this case from internal use and integration into a product that does not constitute such a service. The legal team must verify this interpretation and the other licence conditions for the specific project. Meta, licence of 15 October 2026.

The hosting location alone does not establish control over data. The application may transmit logs or call external tools: data flows, their destinations and administrative access must be examined. The company must also assign responsibility for maintaining the service, responding to incidents and providing the necessary capacity. Without these elements, it does not yet have a defined solution to compare with its current provider.

The comparison covers the entire process

A compatible interface can ease part of the integration without guaranteeing the same behaviour. The vLLM documentation describes an interface compatible with OpenAI APIs, along with additional parameters and default settings that may come from the model repository. To interpret the tests, the IT team must therefore specify the model version, its configuration and the functions actually used. vLLM project, documentation.

The business team can then assess representative cases according to the consequences of an error. It can check whether decisive information at the end of a long document is taken into account, whether a response missing a required field is rejected and what happens when a tool is unavailable. These tests help define which deviations are acceptable in real work; an average quality score alone does not show whether a particular error is acceptable.

Consider a hypothetical example in insurance. An assistant prepares a request for an additional document. After human approval, the application sends it to the customer but does not receive technical confirmation and retries the operation. The expected outcome is one request sent, with a verifiable status and a record of the retry. A duplicate may confuse the customer; an error reported incorrectly may create unnecessary work for the case handler. The quality of the generated text cannot capture these effects.

AWS describes the principle of idempotent APIs: a request can be repeated without producing an additional effect if the service recognises its identifier, among other things. This protection belongs in the application and the services that execute the action, not in an instruction to the model. AWS, 2021. The test must also verify that the status can still be retrieved after an interruption. In the hypothetical example, returning to the previous provider is useful only if the case handler can tell which request was sent and approved. Reconnecting the old interface is not enough to restore that information.

Cost depends on the level of service achieved

Quality and recovery requirements determine what needs to be costed. Stanford HAI reports that between November 2022 and October 2024, the cost of querying a model reaching a level comparable to GPT-3.5 on the MMLU test fell from USD 20 to USD 0.07 per million tokens. This measure from a particular test is a reason to reassess inference options; it demonstrates neither an economic advantage for an open model nor savings on the process being considered. Stanford HAI, AI Index 2025.

The useful comparison is the cost per case processed correctly, against criteria agreed with the business team. It includes inference, any hosting, operations, retries and human checks. The costs of migration, adaptation, qualification, temporary operation of both solutions and preparation for reverting must be shown separately. If a solution that costs less per query requires more corrections or cannot retrieve statuses, its price alone does not justify moving the process.

Authorise the trial before deciding on migration

If an alternative appears relevant to a defined process, the next decision is to fund a reversibility trial with a defined scope. Its mandate specifies the expected benefit, permitted data, acceptance criteria, budget, duration and stop conditions. A responsible lead brings together the results: the business team approves acceptable deviations; IT documents the configuration, recovery steps and route back; procurement and legal verify the rights and exit terms; security examines access and data flows.

Migration may then be authorised only for the scope in which the service delivered, full cost and operating conditions have been established. If the trial leaves the status of an action uncertain after an interruption, or the benefit remains unproven, the current service can remain in place while that evidence is gathered.