When a new AI model comes out: 4 things to check before you switch

These days there is news of a new AI model almost every week. On October 7, Anthropic announced Claude Haiku 5.5, and OpenAI's documentation now lists the GPT-6 family as the recommended replacement for earlier models. Every announcement says some version of "faster, cheaper, smarter." So should you swap the model behind your service right away?

The short answer: there are plenty of reasons to switch, but few reasons to switch *immediately*. A new model may be better on average, but there is no guarantee it is equally better on your service's questions and output formats. This post covers what to check when a new model arrives and in what order to switch. It is part 7 of our AI feature development series, following demo to production, API costs, structured output, streaming, prompt caching, and eval sets.

Check 1: Re-run your eval set

The first thing to do is run the eval set you already use against the new model. An eval set is a test sheet that collects "for this question, the answer should look like this" (we covered how to build one in our previous post, "Check AI feature quality with an eval set, not a gut feeling").

When comparing, look at per-item results rather than the overall score. Even if the overall score rises slightly, if items that used to pass have started failing, first check whether those items matter for your service. New models often change tone or answer length, so with the same prompt quality can even appear to drop. In that case, also run a version of the prompt lightly adjusted for the new model.

Check 2: Measure price and response speed with real requests

You can check a new model's price right away on the official pricing page. For example, Claude Haiku 5.5 costs $0.10 per million input tokens and $0.50 per million output tokens, much lower than the previous Haiku 4.5 ($1 input · $5 output). However, long prompts over 100,000 tokens fall into a different price tier: $0.50 input · $2.50 output.

That is why cost is only accurate when calculated with your own requests.

Tokens are the units AI uses to process text. If a new model writes longer answers, the cost per request can stay about the same or even rise despite a lower unit price.

Check 3: Does the output format still match?

If you receive AI answers as JSON and pass them to a screen or another system, format is the part that breaks most quietly. Differences that look trivial to a person — field names, how empty values are handled, list order — become errors in code.

• If you use structured output (a specified schema), whether the new model supports the same approach
• Whether all format checks (code-based grading) in your eval set pass
• If you stream responses, whether the order and chunking on screen stay the same
• If you use prompt caching, whether the cache still hits properly with the new model

Check 4: Your current model's deprecation schedule

Alongside the new model, look at how long the model you use now will be supported. Each provider publishes model deprecation schedules in its official documentation.

• Anthropic: notifies customers using a public model at least 60 days before retirement, and labels model status as Active, Legacy, Deprecated, or Retired. For example, Claude Sonnet 4.5 was announced for deprecation on September 30 and is scheduled to retire on November 30.
• OpenAI: announces generally available models at least six months in advance, listing the shutdown date along with a recommended replacement model.

If your model has a set retirement date, you need to switch — no deliberation required. If your current model remains supported, switch when the results of the three checks above are good enough.

Switch in stages

Once you have done the four checks, move over in stages rather than all at once.

1. Pin the model version by its exact name. If you let it automatically follow the latest model, it is hard to know when it changed
2. Apply the new model to some users or some features first
3. Compare errors, response times, and user reactions for a few days
4. If there are no problems, expand; if problems arise, roll back with one line of configuration

Summary

New models usually arrive improved, but you only know whether they are better for your service by checking directly. Look at four things — eval set results, price and speed based on real requests, output format, and deprecation schedules — then switch in stages with the version pinned, and model migrations become far less nerve-racking.

References

• Anthropic, Introducing Claude Haiku 5.5 (2026-10-07): https://www.anthropic.com/claude-haiku-5-5
• Claude documentation, Pricing: https://platform.claude.com/docs/en/about-claude/pricing
• Claude documentation, Model deprecations: https://platform.claude.com/docs/en/about-claude/model-deprecations
• OpenAI documentation, Deprecations: https://developers.openai.com/api/docs/deprecations

SOL LABS AI

We build web apps, mobile apps and business systems, then add and run the AI they need. We build and operate our own meeting notes, contract and company analysis AI.

See our projects