LLM API Timeouts and Retries — What Actually Breaks on One Container

When a model provider gets slow, your app doesn't return errors — it stops responding, and everything looks fine from the outside. The default in the OpenAI Python SDK is a 10-minute timeout with two automatic retries (openai-python README), so a single stuck call can occupy one of your workers for close to half an hour. On a platform where your app is one container with no autoscaling and no health probe, a handful of those is the whole app. The fix is unglamorous and takes about ten lines: set a timeout far shorter than the default, decide which errors are worth retrying, and give the user's request a total budget you're willing to honour.

What the SDK already does for you#

Worth knowing exactly, because most people are surprised by both halves. The OpenAI Python SDK documents:

"By default requests time out after 10 minutes."

"Certain errors are automatically retried 2 times by default, with a short exponential backoff."

And the retried set is "connection errors … 408 Request Timeout, 409 Conflict, 429 Rate Limit, and >=500 Internal errors".

Two consequences fall out of that.

Your own retry loop multiplies, it doesn't replace. Wrap that client in a three-attempt loop and you have up to nine requests, not three, with the outer loop unaware of the inner one. If you want your own retry policy, turn the built-in one off first — max_retries=0 — rather than stacking them.

Worst case is measured in tens of minutes. Three attempts at up to ten minutes each is a long time for one request to be alive. Nothing about that is unreasonable for a batch job; it's disastrous for a request a person is waiting on.

Why one container changes the calculus#

On a bigger setup, slow upstream calls degrade throughput. Here they can take the app out entirely, for reasons specific to how it runs:

• There's one container. No second instance is absorbing traffic while this one is tied up — an app is a single container, and you scale it vertically or not at all.
• Your worker count is small, because your RAM is. If you're running four workers and four requests are parked on a hung provider call, the fifth visitor waits with them.
• Nothing probes your app. The platform's view of health is whether the container is running, so a fully wedged process is indistinguishable from a healthy idle one. There are no health checks — you build the watchdog.
• A restart doesn't fix an upstream outage. The container comes back and immediately makes the same failing call. Crashing on provider errors gets you a restart loop that fixes nothing.
• Queued requests hold memory. Each one holds its request body, its context, its partial response. Enough of them and you hit the hard RAM cap and get OOM-killed, which reads as a mystery crash rather than a provider problem.

Set a timeout that matches the wait you'll accept#

Start from the person, not from the API. How long will your user actually wait? Thirty seconds? Then the whole operation — attempts, backoff, and all — has to fit in thirty seconds.

client = OpenAI(timeout=20.0, max_retries=0)   # per-attempt, no hidden retries

Then budget explicitly: two attempts at twenty seconds with a short pause between them fits; three attempts at thirty does not. Per-attempt timeout times attempts must be less than the total budget, and if it isn't, the extra attempts are ones your user will never see the result of.

For work nobody is waiting on — a nightly summariser, a queue consumer — you can be far more generous. That's the case the ten-minute default was written for.

Streaming needs a different timeout. A total deadline is wrong for a stream, because a healthy long generation legitimately takes minutes. What you want is an idle timeout: if no token has arrived for fifteen seconds, the connection is dead regardless of how long the whole thing has been running. If you're streaming to a browser, that idle check is the thing that distinguishes "still thinking" from "hung".

Which errors are worth retrying#

Retrying everything wastes both time and money. The split is clean:

Retry these — connection failures, timeouts, and 5xx. They're transient by definition, and the same request will probably work in a second.

Retry with a real wait — 429. This is the one people get wrong, because retrying a rate limit immediately is how you stay rate-limited. If the provider sends a Retry-After header, honour it; that number is better than any backoff you'd invent. And be aware the SDK's default policy retries 429 with only a short backoff, which is usually not what you want.

Never retry — 400, 401, 404, and content-policy refusals. Your prompt is too long, your key is wrong, your model name is a typo. Three attempts produce three identical failures, more slowly.

Add jitter to whatever backoff you use. Without it, everything that failed at the same moment retries at the same moment, which is the shape of a self-inflicted spike.

Fail usefully instead of hanging#

When you've exhausted the budget, you still owe the caller an answer. In rough order of how much they'll like it:

• Fall back to another model. If your model name is an environment variable rather than a constant, a fallback is a few lines and no redeploy.
• Serve something cached. For summaries and descriptions, slightly stale beats absent.
• Queue it and reply later. Fine for a bot — "I'll get back to you on this" and a follow-up message is a genuinely good experience.
• Say so plainly. "The AI service isn't responding right now, try again in a minute" is a better outcome than a spinner that never resolves.

What you must not do is let the request hang. A user who gets a clear failure in twenty seconds retries; a user watching a spinner for four minutes leaves.

An in-process circuit breaker is enough here#

If the provider is down, the tenth request doesn't need to discover that independently. Count consecutive failures, and after a threshold, stop calling for thirty seconds and fail immediately.

The nice part is that on this platform the simple implementation is also the correct one: there's one container, so a counter in memory is a complete picture of your app's state. No shared store, no coordination — you almost certainly don't need Redis for this. Just remember the counter resets on every deploy, which is harmless for a breaker and would not be for anything you actually care about.

Log enough to tell the two apart#

The question you'll be asking at 2am is "is it them or is it me", and only your logs can answer it. On each attempt, record the model, the attempt number, the elapsed time, and the status or error class.

Keep it structured and keep it small. Logs are capped at roughly 10 MB before older lines are dropped, and logging whole request and response bodies will evict the lines you needed. A count and a duration are worth more than a payload — see viewing logs for what you actually get in the panel.

FAQ#

What's the default OpenAI SDK timeout?#

Ten minutes, with two automatic retries, so a stuck request can live for around thirty minutes. Set both explicitly.

Should I write my own retry loop?#

Only if you disable the SDK's built-in retries first. Otherwise the two multiply and you make far more requests than you think.

Which errors should I retry?#

Connection errors, timeouts and 5xx. Back off properly on 429. Never retry 400, 401, 404 or a content-policy refusal.

How long should my timeout be?#

Work backwards from how long your user will wait. Per-attempt timeout times number of attempts has to fit inside that budget.

Does the platform restart my app if the provider is down?#

No. There's no health probe — the container is running, so it's considered fine. And a restart wouldn't help, since the next call fails the same way.

Why does my app go down entirely instead of returning errors?#

Requests are parked on hung provider calls and holding your workers and memory. Short timeouts free both.

Wrapping up#

Nothing here is specific to AI except how long the calls are. Every rule about calling a slow dependency applies — it's just that a model call can legitimately take a minute, which makes it hard to tell "working" from "wedged", and the defaults are tuned for batch work rather than for someone refreshing a page. Pick a timeout from the user's patience, retry only what's actually transient, keep a breaker so an outage costs you one slow request instead of hundreds, and log enough to know whose fault it was. That's a couple of hours of work that turns your provider's bad afternoon into a slightly worse one for you, instead of an outage.