Skip to content

Webhook retry storm

Most platforms retry failing webhooks for hours to days. If you miss that window — the event will be lost forever. A simple retry counter:

import os, time, requests, collections
W = collections.defaultdict(list) # delivery-id → [timestamps]
@app.post("/hook/<provider>")
def hook(provider, request):
delivery_id = request.headers.get("X-GitHub-Delivery") or \
request.headers.get("Stripe-Delivery-Id") or \
request.headers.get("X-Hook-Id")
if delivery_id:
W[delivery_id].append(time.time())
if len(W[delivery_id]) >= 3:
push(f"🌪️ Webhook retry storm: {provider}",
f"Delivery {delivery_id} получен {len(W[delivery_id])}× за "
f"{int(time.time()-W[delivery_id][0])} сек",
priority=8)
process(provider, request.json)
def push(t, m, p):
requests.post(f"{os.environ['NOTIFLY_URL']}/message",
params={"token": os.environ["NOTIFLY_TOKEN"]},
json={"title": t, "message": m, "priority": p}, timeout=5)

Notifly has built-in webhooks — create a dynamic-webhook to act as a “proxy”: the external service sends to it, it forwards to push and simultaneously records the delivery-id.