LLM model migrations without regressions: deprecations, evals, shadow traffic

Every LLM feature you ship has an expiry date you did not choose. Providers retire models on a schedule, and the replacement is never a drop-in: request parameters get rejected, prompts that were tuned for the old model behave differently, and the tokenizer may count your inputs differently. If Claude Sonnet 4.5 is in your stack, the clock is visible right now: Anthropic deprecated it on 30 September 2026 and retires it on 30 November 2026, with Claude Sonnet 5.5 as the recommended replacement.

This article is a migration playbook: how deprecation timelines work at OpenAI, Anthropic and Google, how to find every call site, what a regression suite for a model swap should measure, and how to roll the new model out with shadow traffic and a canary instead of a big-bang switch.

Why a model swap breaks things

Three different things change when you move to a new model, and each needs its own check:

• The API contract. Parameters that worked on the old model return a 400 on the new one. The Claude Opus 5.5 migration guide lists several: disabling thinking, thinking budgets, forced tool choice, non-default sampling parameters and assistant prefills are all rejected.
• Behaviour. The same prompt produces different outputs. Sclar et al. measured accuracy differences of up to 76 points from meaning-preserving formatting changes alone, and found that which format works best only weakly correlates between models. A prompt tuned for model A is not tuned for model B.
• Cost and limits. Anthropic's guide notes that the tokenizer introduced with Opus 4.7 may use roughly 1x to 1.35x as many tokens as earlier models for the same text. Your max_tokens, compaction triggers and cost estimates move with it.

There is also a fourth case that is not a migration at all: the same model name changing behaviour. Chen, Zaharia and Zou compared the March and June 2023 versions of GPT-4 and found accuracy on identifying prime numbers dropping from 84% to 51%, along with more formatting mistakes in generated code. Their conclusion, that the behaviour of the "same" LLM service can change substantially in a short time, is the argument for running your regression suite on a schedule, not only when you switch.

Deprecation timelines at OpenAI, Anthropic and Google

How much warning you get depends on the provider and on the kind of model. From each provider's documentation:

Two practical consequences. Never put a preview model or a -latest alias behind a business-critical feature: OpenAI's docs say outright that they do not recommend preview models for such workloads unless you can migrate on short notice. And put the retirement date of every model you use into a calendar the day you adopt it. Anthropic currently lists tentative dates as "not sooner than" a given day, which is a floor, not a promise.

Pin model IDs and know where they are used

A migration you cannot scope is a migration you cannot finish. Before any testing:

1. Pin exact model identifiers in production. Stable names on Gemini, exact IDs on Claude and OpenAI. An alias moves without a deploy, which means behaviour can change without a commit.
2. Centralise the model ID in configuration, one entry per use case ("order-extraction", "support-triage"), not a string literal per call site. A gateway such as LiteLLM or OpenRouter makes this a routing rule.
3. Inventory actual usage. Code search finds the call sites you know about. Anthropic's Console usage export (CSV by API key and model) and your gateway logs find the ones you forgot: the cron job, the internal tool, the prototype that became production.
4. Record the request settings per use case: max tokens, thinking or effort settings, tool choice, structured output schema, sampling parameters. These are what the new model may reject.

The migration pipeline

I run every model swap through the same five stages. Each stage has an exit criterion, and any failure sends you back one step, not forward with a fix in production.

Stage 2: fix the contract before you test behaviour

Contract changes are deterministic, so handle them first and separately. Read the provider's migration guide for your exact source and target model. As a concrete example, these are some of the changes the Claude Opus 5.5 guide lists for code coming from earlier models:

The exit criterion for this stage is simple: every recorded request shape from the inventory goes through against the new model without an error. Anthropic also ships a /claude-api migrate skill for Claude Code that applies the model ID swap and breaking parameter changes across a codebase and produces a checklist of items to verify by hand. Tools like that speed up stage 2; they do not replace stage 3.

Stage 3: the regression suite for a model swap

Anthropic's eval guide describes regression evals as the ones that should sit near 100% and catch backsliding. For a model swap, that suite runs both models on the same inputs and compares. Build it from production, not from imagination: sample real requests per use case, add every past incident, and keep the expected outcome or a reference answer next to each one. Then measure more than accuracy:

Score open-ended outputs pairwise: show a judge the old and the new answer for the same input, in random order, and ask which better meets the rubric. Pairwise comparison is more stable than absolute scores for "is this as good as before", and it surfaces the cases worth reading. The harness itself is small:

import random, statistics

def compare(cases, old_model, new_model, run, judge, checks):
    """run(model, case) -> Result(text, tokens, ms, refused); judge(case, a, b) -> 'a'|'b'|'tie'"""
    rows = []
    for case in cases:
        old, new = run(old_model, case), run(new_model, case)
        flip = random.random() < 0.5                      # hide which answer is new
        verdict = judge(case, *( (new.text, old.text) if flip else (old.text, new.text) ))
        new_wins = (verdict == "a") == flip if verdict != "tie" else None
        rows.append({
            "id": case.id,
            "checks_failed": [c.name for c in checks if not c(case, new)],   # schema, refs, rules
            "new_wins": new_wins,
            "refused": (old.refused, new.refused),
            "len_ratio": len(new.text) / max(1, len(old.text)),
            "token_ratio": new.tokens / max(1, old.tokens),
            "ms": (old.ms, new.ms),
        })
    losses = [r for r in rows if r["new_wins"] is False or r["checks_failed"]]
    print("losses to read:", [r["id"] for r in losses])
    print("median length ratio:", statistics.median(r["len_ratio"] for r in rows))
    return rows

Run each case more than once if the feature is agentic or sampled; one run per model hides variance. And keep the losses list as the real output of the stage: the decision to proceed is made by reading those cases, not by a single aggregate score. Promptfoo and similar tools can run the same side-by-side comparison from configuration if you prefer not to maintain a harness; the method is covered in more depth in LLM evals for product features.

Re-tune prompts, effort and old workarounds

When the suite shows losses, resist adding instructions on top of the old prompt. Prompts accumulate workarounds for the previous model's quirks ("never use bullet points", "ALWAYS return the field even if empty"), and a newer model may follow them too literally. That fits the Sclar et al. finding that prompt-format preferences transfer poorly between models. My order of work:

1. Remove workarounds first and re-run. Many losses disappear.
2. Re-tune the reasoning setting. On Claude, effort replaces thinking budgets, and defaults differ between models (medium on Opus 5.5). Sweep low, medium and high on the suite and pick per use case.
3. Replace format coaxing with structured outputs where the provider supports them.
4. Only then add new instructions, one at a time, each tied to a failing case.

Stages 4 and 5: shadow traffic and canary

Offline suites only contain the cases you thought of. Shadow traffic sends a copy of live requests to the new model, logs its answer next to the served one, and never shows it to users. Run it for a few days across the weekly traffic pattern, then run the same checks as in stage 3 on the shadow pairs. Two cautions: shadow calls cost money, so sample if volume is high; and never let the shadow path execute side effects. For agents, shadow only the read-only part or point tools at a sandbox.

The canary then serves the new model to a small share of traffic behind a feature flag, with the same dashboards you use for the old model: error rate, refusal rate, user feedback, cost per task and latency. Widen in steps (5%, 25%, 100%) with a fixed observation window per step, and keep the old configuration deployable until the retirement date, so rollback is a flag flip rather than a release. Tracing with OpenTelemetry makes the side-by-side comparison per model a query instead of a project.

Keep the suite running after the migration

The regression suite is the expensive part to build and the cheap part to keep. Run it weekly against the production configuration, and on every prompt or model change. That catches drift behind a stable name (the Chen et al. case), it gives you a baseline when the next deprecation notice arrives, and it turns "should we switch to the new, cheaper model?" into a measurement. The same suite is what makes model routing for cost safe: a cheaper model is only cheaper if it passes.

Leaderboards help you pick candidates, as I discussed for the current Artificial Analysis ranking, but they cannot tell you whether your extraction prompt still finds the delivery date. Only your suite can.

Migration checklist

1. Calendar the retirement date of every model you use; avoid preview models and -latest aliases in production.
2. Pin exact model IDs, centralise them per use case, and inventory real usage from provider exports and gateway logs.
3. Read the migration guide for your exact source and target model; fix every rejected parameter first.
4. Re-measure token counts and adjust max_tokens, budgets and compaction triggers.
5. Run old and new model side by side on a golden set built from production and past incidents.
6. Gate on success, parse failures, refusals, length, cost per task and latency, and read every loss.
7. Remove old prompt workarounds before adding new instructions; re-tune effort per use case.
8. Shadow live traffic without side effects, then canary behind a flag with rollback until the retirement date.
9. Keep the suite running weekly after the migration.

If you have a retirement date coming up and want a second pair of hands on the migration, see AI engineering.

Sources

1. Claude API docs: Model deprecations
2. OpenAI API docs: Deprecations
3. Gemini API docs: Models and version name patterns
4. Claude API docs: Claude Opus 5.5 migration guide
5. Chen, Zaharia, Zou: How is ChatGPT's behavior changing over time? (2023)
6. Sclar et al.: Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design (2023)
7. Anthropic: Demystifying evals for AI agents (Jan 2026)