What Changes When the Model Changes

A new model performs better on a public benchmark. Your team expects it to improve an architecture assistant that reads project context and proposes design options. Before changing the default, someone needs to answer a narrower question: does the candidate perform acceptably on this workflow, with these tools and these review expectations?

The model is one part of that workflow. Prompts, effort settings, context handling, retrieval, tool definitions, and output evaluation also affect the result. A useful change review identifies what is changing and tests the behavior the team actually depends on.

1. Describe the complete change

Record the incumbent and candidate model identifiers, the serving platform, the relevant settings, and the prompts and tools used for the comparison. If the candidate needs a different prompt or tool arrangement, include that difference in the evaluation rather than describing the exercise as a model-only swap.

Configuration changes deserve attention even when a model name stays the same. Anthropic's April 2026 Claude Code postmortem described quality problems involving reasoning-effort defaults, context handling, and a verbosity instruction. The company explicitly stated that its API and inference layer were unaffected. The lesson is about the complete application configuration: behavior can change without changing the underlying model service.

For the architecture assistant, the change record might identify a new model, the same source documents, an unchanged tool set, and a revised effort setting. That gives the comparison a defined object.

2. Decide what acceptable means before reviewing results

Choose tasks that represent the intended work. Include ordinary cases, difficult cases, and cases where the correct behavior is to identify missing information or decline a proposed action. A test collection made only of successful demonstrations will underrepresent those boundaries.

For an architecture workflow, assess whether the response uses the supplied constraints, cites relevant sources, identifies conflicting requirements, compares viable options, and preserves unanswered questions. Score material errors separately from writing quality. A more fluent response should not hide a missing constraint.

Set acceptance criteria before comparing results. The candidate need not win every category: improved cost, availability, or latency may be worth a small, explicitly accepted tradeoff elsewhere. Some requirements should remain hard limits. The decision should distinguish those limits from preferences.

3. Compare repeated runs with appropriate review

Run both configurations on the same cases and retain the inputs and outputs. Where behavior varies, repeat the cases enough to understand that variation. Report the spread and the important failure modes rather than selecting the most persuasive response from each model.

Use exact checks for properties that can be checked exactly, such as required fields or permitted tool arguments. Use qualified review for the quality of an architecture option or the adequacy of its rationale. If a model helps score the results, examine its rubric and check samples yourself. Do not make the candidate the sole judge of its own work.

An illustrative result might show that the candidate produces clearer option comparisons but misses a constraint in one class of incomplete briefs. The next decision could be to improve the workflow and repeat those cases, or to restrict adoption to a narrower task. An average score should not make that failure disappear.

4. Stage the rollout and define the recovery point

Start in an environment where the candidate's outputs can be inspected without committing consequential effects. Shadow evaluation is useful only if the shadow path cannot duplicate writes or send messages. Then expand use within an agreed scope while retaining the previous configuration where feasible.

Define the signals that stop expansion and the person responsible for acting on them. Recovery must account for any effects already committed; changing a model setting does not undo a record that has already been altered.

If the incumbent is being retired, record that constraint. The choice may be between several replacements rather than between changing and staying put. A defensible decision reflects the options actually available.

5. Keep the evidence with the decision

A reusable change record includes the compared configurations, evaluation cases, scoring criteria, material failures, accepted tradeoffs, reviewer, rollout scope, and recovery conditions. Preserve enough context for another person to understand why the change was accepted.

Then continue checking representative behavior after adoption. A passing pre-release comparison is evidence about that comparison; it is not a guarantee about every future task or configuration.

Archangel's Decision Trace connects evidence, constraints, rationale, review, and downstream work around a decision. A model-change evaluation can supply evidence for that record. The evaluation harness and rollout controls still need to be designed and operated by the team; recording the decision does not execute them.

Before the next model change, ask for a short review that answers five questions: what changed, what acceptable means, what the comparison found, how rollout will be controlled, and where the decision can be inspected. Those answers are more useful than a benchmark headline.