API-key usage spike (possible leak)

A leaked provider key is not an abstract risk: sk-... ends up in a public commit, a log dump, a built frontend bundle — and an hour later someone is mining embeddings for their own startup on your dime. The first clear signal is usually the invoice at the end of the month. A push on a sudden spike catches it within minutes.

Three independent signals, each worth an alert:

• volume spike — the key makes many times more requests/tokens than in a typical hour;
• new IP / country — requests started coming from an address you have never seen;
• spend spike — the provider usage API shows spend above the daily budget.

Option 1: rate-count per key in your own gateway

If all traffic to the provider goes through your proxy/gateway, you see who burns how much. A sliding window on Redis + a per-key threshold:

import os, time, requests, redis

R = redis.from_url(os.environ["REDIS_URL"])
NOTIFLY_URL   = os.environ["NOTIFLY_URL"]
NOTIFLY_TOKEN = os.environ["NOTIFLY_TOKEN"]

WINDOW   = 60          # window in seconds
LIMIT    = 300         # requests per key per window above which it's suspicious

def notify(title, msg, prio):
    requests.post(f"{NOTIFLY_URL}/message",
                  params={"token": NOTIFLY_TOKEN},
                  json={"title": title, "message": msg, "priority": prio},
                  timeout=5)

def on_request(key_id: str, ip: str):
    now  = int(time.time())
    rkey = f"rate:{key_id}:{now // WINDOW}"
    n    = R.incr(rkey)
    R.expire(rkey, WINDOW * 2)

    # remember the set of IPs the key is used from
    seen = f"ips:{key_id}"
    is_new_ip = R.sadd(seen, ip) == 1
    R.expire(seen, 86400)

    flag = f"alerted:{key_id}:{now // WINDOW}"
    if n > LIMIT and R.set(flag, 1, nx=True, ex=WINDOW):
        notify("🔑 Всплеск по API-ключу",
               f"Ключ {key_id}: {n} запросов за {WINDOW}с (порог {LIMIT}).\n"
               f"Последний IP: {ip}\nВозможна утечка — отозвать ключ?",
               prio=9)

    if is_new_ip and R.scard(seen) > 5:
        notify("🔑 Ключ ходит с новых IP",
               f"Ключ {key_id}: за сутки уже {R.scard(seen)} разных IP, "
               f"новый: {ip}.",
               prio=8)

priority=9 — because the cost of a miss is high (a leak = money + access to data), so this push should be loud even if it turns out to be a false alarm.

Option 2: poll the provider usage API on a timer

No proxy key, only the provider itself? Many expose usage/costs via API. A rolling comparison against a baseline in a state file so you don’t alert on every tick. Drop it into a scheduled YC function:

import os, json, datetime, requests

STATE = "/tmp/apikey-spend.json"
DAILY_BUDGET = float(os.environ.get("DAILY_BUDGET_USD", "20"))

def handler(event, context):
    today = datetime.date.today().isoformat()
    # OpenAI, for example, has /v1/usage?date=YYYY-MM-DD; fields depend on the provider
    r = requests.get("https://api.openai.com/v1/usage",
                     params={"date": today},
                     headers={"Authorization": f"Bearer {os.environ['PROVIDER_KEY']}"},
                     timeout=15)
    spent = r.json().get("total_usage", 0) / 100.0    # cents → dollars

    st = json.load(open(STATE)) if os.path.exists(STATE) else {}
    already = st.get(today, {}).get("alerted", False)

    if spent > DAILY_BUDGET and not already:
        push("💸 Расход по ключу выше бюджета",
             f"Сегодня потрачено ${spent:.2f} при бюджете ${DAILY_BUDGET:.2f}.\n"
             f"Если это не вы — ключ мог утечь.",
             prio=9)
        st[today] = {"alerted": True}
        json.dump(st, open(STATE, "w"))
    return {"statusCode": 200}

def push(t, m, prio):
    requests.post(f"{os.environ['NOTIFLY_URL']}/message",
                  params={"token": os.environ["NOTIFLY_TOKEN"]},
                  json={"title": t, "message": m, "priority": prio}, timeout=5)

Timer */10 * * * ? * — once every 10 minutes is more than enough to react before the bill grows by an order of magnitude.

What to do on the alert

1. Revoke/rotate the key in the provider console — this is the first action.
2. Check where it could have leaked: git log -p | grep -i sk-, the public bundle, environment variables in CI.
3. Issue a new key already restricted by IP/budget, if the provider supports it.

What to put in the alert text

• the key id (not the secret itself!) and its purpose;
• which metric fired: RPS / spend / new IP;
• the value and the threshold, so you grasp the scale immediately;
• a link to the key-revocation page in the provider console.