A service with AI features can run smoothly most of the time, then one day answers suddenly slow down or an error screen appears. A model API is, in the end, an external service: when usage spikes or our requests exceed a limit, responses fail.
This post covers how to make sure users never stare at a blank screen in those moments. It's a sequence for sorting errors by type and deciding whether to retry, wait, or route around the problem.
First, sort errors into three groups
Applying the same "retry" to every error can actually make things worse. Grouping the error codes from the official documentation into three broad buckets keeps your response simple.
| Group | Typical codes | Response |
|---|---|---|
| Errors that clear if you wait | 429 (rate limit), 500, 504 (timeout), 529 (overloaded) | Retry with growing intervals |
| Errors that need a fixed request | 400 (request format), 413 (request size) | Don't retry; log and fix |
| Errors that need a settings check | 401 (key), 402 (billing), 403 (permission) | Alert immediately; admin checks |
Anthropic's documentation describes 529 as the API being temporarily overloaded and 429 as hitting a rate limit. However, it notes that when a 429 comes from hitting a monthly spend cap, no retry time (retry-after) is returned and resending will keep failing. In that case, stop retrying and notify an administrator.
Retry "with growing intervals and a fixed limit"
For errors that clear if you wait, retrying is the answer. But don't fire requests back to back. OpenAI's official documentation explains that failed requests also count toward your per-minute limit. If you keep sending without pause, you take longer to get out from under the limit.
So follow two principles.
- Exponential backoff (wait longer each time): Increase the wait after each failure, like 1 second, 2 seconds, 4 seconds. Add a small random delay (jitter) so many requests don't pile in at once.
- retry-after first: If the response includes a retry time, use it as the minimum wait.
Always cap both the number of retries and the total wait time. Anthropic's official SDK (software development kit) retries connection errors, rate limits, and server errors twice by default. If you call the API directly without the SDK, you need to build the same rules into your own code.
Set timeouts by "how long users will wait"
A timeout (the maximum time to wait for a response) is a number about screen experience before it's a technical number. A feature where users wait in front of the screen, like chat, needs a different standard from a feature that organizes documents overnight.
- On-screen features: Streaming (sending the answer piece by piece) shows the first words quickly, so the wait feels shorter. Anthropic's documentation also recommends streaming or batch processing for long requests, especially those that may take more than 10 minutes.
- Background features: Don't make users wait for results; put the job in a task list and send a notification when it's done.
If retries don't work, keep another path open
Once you've passed the retry limit, show users another path instead of a blank screen. Prepare these in order, depending on your service.
- Fallback model: Send the request once more to a different model or another provider's API that can do the same job. The tone or format of answers may differ, so it's best to check with your evaluation set in advance.
- Saved answers: For frequently asked questions, show a recently verified answer, along with the time it was checked.
- "Try again shortly" screen: Briefly explain what didn't work, whether the user's input was saved, and when to try again.
- Do you log error codes in the three groups?
- Are there caps on the number of retries and the total wait time?
- Is the user's input still there after a failure?
- Does the screen format hold when you switch to a fallback model?
Summary
Handling LLM failures is a sequence: sort the errors, retry with growing intervals, and if that still fails, hand off to a fallback model, saved answers, or a guidance screen. We can't control the state of a model API, but we can control what users see when it fails.
References
- Anthropic, Errors: https://platform.claude.com/docs/en/api/errors
- OpenAI, Rate limits: https://developers.openai.com/api/docs/guides/rate-limits
We review your current system and data, then outline the migration order and scope.
We build web apps, mobile apps and business systems, then add and run the AI they need. We build and operate our own meeting notes, contract and company analysis AI.
See our projects