Provider rate-limit 429

429 Too Many Requests is not a provider outage but a signal that you hit your own limit (RPM/TPM). One 429 is normal, the SDK retries it itself. But when there are dozens per minute — half the requests go into backoff, user latency grows, and it’s time either to throttle your own load or ask the provider for a higher tier.

A full outage is caught by availability monitoring; this is specifically about 429s in a rolling window.

Count 429s around the client call

We wrap the SDK call: catch the rate-limit error, put a timestamp into a rolling window, and alert when the threshold per minute is exceeded.

import os, time, collections, requests
import openai   # or anthropic — the principle is the same

NOTIFLY_URL   = os.environ["NOTIFLY_URL"]
NOTIFLY_TOKEN = os.environ["NOTIFLY_TOKEN"]

WINDOW_SEC = 60     # window width
THRESHOLD  = 10     # this many 429s per window = alert
COOLDOWN   = 300    # don't repeat the alert more often than once per 5 minutes

client = openai.OpenAI()
_hits = collections.deque()   # timestamps of recent 429s
_last_alert = 0.0

def notify(title, msg, prio):
    requests.post(f"{NOTIFLY_URL}/message",
                  params={"token": NOTIFLY_TOKEN},
                  json={"title": title, "message": msg, "priority": prio},
                  timeout=5)

def _record_429():
    global _last_alert
    now = time.time()
    _hits.append(now)
    while _hits and _hits[0] < now - WINDOW_SEC:   # drop stale entries
        _hits.popleft()
    if len(_hits) >= THRESHOLD and now - _last_alert > COOLDOWN:
        _last_alert = now
        notify(
            f"🚦 Rate-limit: {len(_hits)} × 429 за {WINDOW_SEC}с",
            "Провайдер режет по лимиту. Сбавьте темп или поднимите tier.",
            priority=7,
        )

def chat(**kwargs):
    """Wrapper around the call: counts 429s, retries with backoff."""
    for attempt in range(5):
        try:
            return client.chat.completions.create(**kwargs)
        except openai.RateLimitError as e:
            _record_429()
            # respect Retry-After if the provider sent it
            wait = float(getattr(e, "response", None)
                         and e.response.headers.get("retry-after", 0)) or 2 ** attempt
            time.sleep(wait)
    raise RuntimeError("rate-limited: retry count exceeded")

# usage — like a normal call, but with 429 accounting
resp = chat(model="gpt-4o-mini", messages=[{"role": "user", "content": "ping"}])

Key details: a deque with stale entries dropped gives a real rolling window; COOLDOWN keeps your phone from being flooded while the limit holds; Retry-After from the header is respected, not ignored.

Separate 429 from 529 / 503

With Anthropic, overload arrives as 529 Overloaded — that’s about the provider’s side, not your limit, and is fixed differently (wait, don’t throttle yourself). Count them with a separate counter and send a separate alert, otherwise you’ll confuse “I’m too fast” with “it’s hot on their end”.

Multi-process variant — a counter in Redis

If requests come from several workers, an in-memory deque in one process won’t see the whole picture. Keep the rolling window in Redis:

import redis, time
r = redis.Redis()

def record_429_shared():
    now = time.time()
    key = "llm:429"
    r.zadd(key, {f"{now}": now})
    r.zremrangebyscore(key, 0, now - WINDOW_SEC)   # clean stale entries
    return r.zcard(key)

The threshold is checked against the returned count — this way all workers together decide when it’s time to send the push.

What to put in the alert text

• which provider/model is throttling (each has its own limit);
• the number of 429s per window and the current trend;
• the Retry-After value, if the provider sent it;
• a link to the limits page in the dashboard, to raise the tier in one click.