A new model shipped — worth benchmarking

The flip side of deprecation: new models also appear silently. While you are heads-down in features, the provider ships a model that is cheaper/smarter/faster, and you keep paying for the old one for six months. This is not an incident — so a low-priority push (0–3, silent), just so you don’t miss it.

The same scheduled /models poll, but the logic is inverted: you care about new ids that were not there last time.

Skeleton: diff the model list

A state file keeps the set of previously seen ids. Anything not in it is a benchmark candidate.

import os, json, requests

STATE = "/tmp/known-models.json"

# family prefixes you care about (so you don't get noise from service ids)
WATCH_PREFIXES = ("gpt-", "o1", "o3", "claude-")

def notify(title, msg, prio):
    requests.post(f"{os.environ['NOTIFLY_URL']}/message",
                  params={"token": os.environ["NOTIFLY_TOKEN"]},
                  json={"title": title, "message": msg, "priority": prio},
                  timeout=5)

def list_model_ids():
    r = requests.get("https://api.openai.com/v1/models",
                     headers={"Authorization": f"Bearer {os.environ['PROVIDER_KEY']}"},
                     timeout=15)
    r.raise_for_status()
    return {m["id"] for m in r.json()["data"]
            if m["id"].startswith(WATCH_PREFIXES)}

def handler(event, context):
    current = list_model_ids()
    known   = set(json.load(open(STATE))) if os.path.exists(STATE) else set()

    # first run — just remember, don't spam the whole catalog
    if not known:
        json.dump(sorted(current), open(STATE, "w"))
        return {"statusCode": 200, "body": "seeded"}

    fresh = sorted(current - known)
    if fresh:
        notify("🆕 Новые модели у провайдера",
               "Появились в /models:\n" + "\n".join(f"• {m}" for m in fresh) +
               "\n\nСтоит прогнать eval и сравнить цену/качество.",
               prio=2)                       # quiet informational push
        json.dump(sorted(current | known), open(STATE, "w"))

    return {"statusCode": 200, "body": json.dumps({"new": fresh})}

priority=2 — this is “when you have a minute”, with no sound or vibration. Exactly what a “there’s something to benchmark” news deserves, not an “everything is broken”.

A once-a-day timer (0 10 * * ? *) — new models don’t appear every minute.

An automatic mini-benchmark right in the alert

It is useful not just to say “a model shipped” but to give a couple of numbers right away. Run a tiny eval on the new id and put the result in the push:

def quick_bench(model_id: str) -> str:
    cases = [("2+2?", "4"), ("Столица Франции?", "Париж")]
    ok = 0
    for q, expected in cases:
        out = ask_model(model_id, q)          # your provider call
        ok += expected.lower() in out.lower()
    return f"smoke-eval: {ok}/{len(cases)}"

# inside handler, for each fresh model:
line = f"• {m} — {quick_bench(m)}"

A full comparison against the current production model is already a task from eval regression; here a smoke test is enough to tell whether the model is “alive” and whether it is worth an evening.

What to put in the alert text

• the list of new ids (only interesting families, no service ones);
• if possible — the price per 1M tokens and the context window of the new model;
• the smoke-eval result, if you run it;
• a link to the provider’s release notes.