GPU utilization

The GPU is the most expensive line in a solo developer’s bill when running local models. Two problems look equally silent:

• idle while paying — the instance is up, the model is unloaded, but $X/hour keeps ticking (you forgot to shut it down after an experiment);
• sustained saturation — the card is pegged at 100% for hours, the inference queue grows, and you find out about it from users.

Standard dashboards show this only if you look at them. A push comes on its own.

Option 1: poll nvidia-smi on a timer

nvidia-smi can output metrics as CSV — we parse them and decide what it is: idle or saturation. A state file prevents a duplicate alert.

import os, time, subprocess, requests

NOTIFLY_URL   = os.environ["NOTIFLY_URL"]
NOTIFLY_TOKEN = os.environ["NOTIFLY_TOKEN"]

IDLE_UTIL   = 5      # % load below which we consider the card idle
BUSY_UTIL   = 95     # % load above which — saturation
WINDOW      = 10     # how many consecutive samples confirm the state
COST_PER_HR = 1.20   # $/hour for the instance — to estimate "burned" money

def notify(title, msg, prio):
    requests.post(f"{NOTIFLY_URL}/message",
                  params={"token": NOTIFLY_TOKEN},
                  json={"title": title, "message": msg, "priority": prio},
                  timeout=5)

def sample():
    # utilization.gpu (%), memory.used (MiB), power.draw (W) for each card
    out = subprocess.check_output([
        "nvidia-smi",
        "--query-gpu=utilization.gpu,memory.used,power.draw",
        "--format=csv,noheader,nounits",
    ], text=True)
    rows = []
    for line in out.strip().splitlines():
        util, mem, power = (x.strip() for x in line.split(","))
        rows.append((float(util), float(mem), float(power)))
    return rows

STATE = "/tmp/gpu-util.txt"

def push_window(value):
    arr = []
    if os.path.exists(STATE):
        arr = [float(x) for x in open(STATE).read().split() if x]
    arr.append(value)
    arr = arr[-WINDOW:]
    open(STATE, "w").write(" ".join(map(str, arr)))
    return arr

def flag(name):
    return f"/tmp/gpu-{name}.flag"

def alert_once(name, title, msg, prio):
    if not os.path.exists(flag(name)):
        notify(title, msg, prio)
        open(flag(name), "w").close()

def clear(name):
    if os.path.exists(flag(name)):
        os.remove(flag(name))

def check():
    rows = sample()
    util = max(r[0] for r in rows)          # take the busiest card
    arr = push_window(util)
    if len(arr) < WINDOW:
        return

    if all(u <= IDLE_UTIL for u in arr):
        waste = COST_PER_HR * (WINDOW / 60)  # rough estimate over the window
        alert_once("idle",
                   "🟡 GPU простаивает, но оплачивается",
                   f"Загрузка ≤{IDLE_UTIL}% уже {WINDOW} замеров подряд. "
                   f"Сожжено ≈${waste:.2f}. Выключить инстанс?",
                   priority=6)
    else:
        clear("idle")

    if all(u >= BUSY_UTIL for u in arr):
        alert_once("busy",
                   "🔴 GPU перегружена",
                   f"Загрузка ≥{BUSY_UTIL}% уже {WINDOW} замеров. "
                   f"Очередь инференса растёт — добавьте карту или троттлите.",
                   priority=8)
    else:
        clear("busy")

check()

Idle (priority=6) is about money, so it can stay calm; saturation (priority=8) hits users, so it’s louder. power.draw is useful as a second signal: a card where utilization twitches but power stays steadily low is most likely just holding the model in memory, computing nothing.

Option 2: cron on the instance itself

If the GPU is on a rented server, put the script into cron right there:

*/1 * * * *  /usr/bin/python3 /opt/notifly/gpu_util.py

State files in /tmp survive between runs, so the window and flags work without an external DB.

Option 3: heartbeat “instance is alive and busy”

Invert the logic: let the script send a heartbeat only while the card is actually computing something. A missed heartbeat = “inference stopped” (process crashed, CUDA OOM), and Notifly itself will send an alert on timeout — without a single line about “check whether the worker is alive”.

What to put in the alert text

• the card index (GPU 0/1/...) — if there are several;
• memory.used and power.draw — to tell “holds the model” from “actually computing”;
• an estimate of money burned over the idle window;
• the instance shutdown command (yc compute instance stop ...).