Nightly eval run finished — scorecard

The difference from eval regression: that one stays silent while all is well and shouts only on a drop. Here it is a routine daily report: the run finished, here is the scorecard. This is the push you want to see every morning, even when everything is green — it is your “git status for model quality” before you start the day.

One push with the aggregate: pass rate, delta vs yesterday’s run, and an explicit list of what regressed.

Runner: run the suite and assemble the scorecard

A state file keeps the previous run’s results — from them we compute deltas and catch cases that passed yesterday and fail today (regressions).

import os, json, time, requests

EVAL_FILE = "evals/suite.jsonl"        # {"id":..., "input":..., "expected":...}
STATE     = "/tmp/eval-suite-last.json"

def notify(title, msg, prio):
    requests.post(f"{os.environ['NOTIFLY_URL']}/message",
                  params={"token": os.environ["NOTIFLY_TOKEN"]},
                  json={"title": title, "message": msg, "priority": prio},
                  timeout=5)

def run_case(case) -> bool:
    out = my_app.answer(case["input"])          # your real pipeline
    return grade(out, case["expected"])          # 0/1, LLM judge or exact match

def main():
    cases   = [json.loads(l) for l in open(EVAL_FILE)]
    results = {c["id"]: run_case(c) for c in cases}

    passed = sum(results.values())
    total  = len(results)
    rate   = passed / total

    prev = json.load(open(STATE)) if os.path.exists(STATE) else {}
    prev_results = prev.get("results", {})
    prev_rate    = prev.get("rate", rate)

    # cases that were ok and became fail = regressions
    regressed = sorted(cid for cid, ok in results.items()
                       if not ok and prev_results.get(cid) is True)
    fixed     = sorted(cid for cid, ok in results.items()
                       if ok and prev_results.get(cid) is False)

    delta = rate - prev_rate
    arrow = "▲" if delta > 0 else "▼" if delta < 0 else "="
    lines = [
        f"Pass: {passed}/{total} ({rate:.0%})",
        f"Δ к прошлому: {arrow} {delta:+.1%}",
    ]
    if regressed:
        lines.append("Регрессии: " + ", ".join(regressed[:8]))
    if fixed:
        lines.append("Починилось: " + ", ".join(fixed[:8]))

    # priority differs: regressions present — more visible, all green — quiet
    prio = 6 if regressed else 3
    notify(f"🧾 Eval-скоркарта {rate:.0%} {arrow}", "\n".join(lines), prio)

    json.dump({"rate": rate, "results": results, "ts": time.time()},
              open(STATE, "w"))

if __name__ == "__main__":
    main()

priority=3 (silent) for the routine “all green” — you read it over breakfast without a call in the middle of the night. If regressions appeared, priority=6, more visible, but it is still a report, not a fire.

Scheduled run

At night via cron/systemd-timer on your own server:

0 4 * * *  /usr/bin/python3 /opt/evals/run_suite.py

Or with the same scheduled YC pattern if the pipeline is reachable from the cloud. The morning push becomes part of the digest.

Breakdown by category

If the suite is tagged by topic (RAG / format / safety / math), it helps to give the pass rate for each — you see at once where it dropped:

from collections import defaultdict
by_cat = defaultdict(lambda: [0, 0])
for c in cases:
    ok = results[c["id"]]
    by_cat[c.get("category", "other")][0] += ok
    by_cat[c.get("category", "other")][1] += 1

cat_lines = [f"  {cat}: {p}/{t}" for cat, (p, t) in sorted(by_cat.items())]
# add cat_lines to the scorecard message

What to put in the alert text

• the pass rate and the passed/total fraction;
• the delta vs the previous run (with an arrow);
• the list of regressed case ids (top 8) — the most important part;
• the breakdown by category and a link to the full report in S3/gist.