Skip to content
Notifly Notifly

Canary deployment errors

Canary deployment is the cheapest way to catch regressions. The condition “5xx for canary > 1% with a stable baseline” fits nicely as a push:

import os, time, statistics, requests


W = "/tmp/canary.json"


def observe(version: str, status: int):
    import json
    s = (json.load(open(W)) if os.path.exists(W) else {})
    g = s.setdefault(version, {"ok":0, "err":0})
    if status >= 500: g["err"] += 1
    else:             g["ok"]  += 1
    json.dump(s, open(W, "w"))


    if (g["ok"] + g["err"]) > 200:
        rate = g["err"] / (g["ok"] + g["err"])
        baseline = 0.001  # assume 0.1%
        if rate > max(baseline * 5, 0.01):
            push("🐤 Canary error-rate",
                 f"{version}: {int(rate*100)}% (baseline {baseline*100:.2f}%)\n"
                 f"Окно: {g['ok']+g['err']} req\n\n"
                 "Откатить деплой?",
                 priority=10)


def push(t, m, p):
    requests.post(f"{os.environ['NOTIFLY_URL']}/message",
                  params={"token": os.environ["NOTIFLY_TOKEN"]},
                  json={"title": t, "message": m, "priority": p}, timeout=5)

You can put a link into the push via WebScript, which will call your rollback endpoint — you’ll get a “rollback from your phone”.

Section titled “Related recipes”