Retry prompt on HTTP error code(s)

[zikajk]

When LLM returns 429, we should repeat the request (and cancel it at any time).
A good default is to do it 10 times with the interval increasing by 2 times each time (2s, 4s, 8s, ...).

It also makes sense to have the model name readily available so we can eventually use different HTTP error codes for different models, if this is not enough.

To upvote this issue, give it a thumbs up. See this list for the most upvoted issues.

[ericdallo]

Interesting, I think one thing we can already do is mention the status code always on every errors, because I don't think I faced too many 429 errors, or I just don't know it was a 429

[zikajk]

We do mention the status code.

429 is the one that gets retry in most agents.
There might be some problems, e.g. Claude Code sub uses 429 when your limit is gone but there is no harm in retrying (10 times max) and allowing user to cancel IMHO.

It will be very useful with Gemini through proxy, Kimi-K2 and other models that returns smth like "Usage is too high, try again later" quite often.

[ericdallo]

Makes sense, maybe the status code could be configurable? dunno... with 429 as default

[ericdallo]

Ok I have something working for this! example of a simulated error of 429.
I don't intend to have configs to customize that yet, let's see what users face, but I think the default of retrying on 429, 500, 502 and 503 is pretty good for most cases.

[zikajk]

Nice, now I wonder if the same error doesn't happen when you exceed your quota. But even if it does this shouldn't hurt, right?

[ericdallo]

@zikajk yeah, from what I researched the quota can happen on some 429 and yes we try again which I don't see a issue