The queue on your inference server is growing

Your own inference server (vLLM, TGI, Ollama) is not elastic like a cloud API: it has a fixed number of GPUs and finite throughput. When more requests arrive than the card can process, they pile up in the queue — and the user sees a spinner for 30 seconds instead of an answer. A dashboard shows this only if you watch it; a push arrives on its own.

vLLM and TGI expose Prometheus metrics on /metrics. We poll them with a scheduled function and alert on a threshold on queue depth.

Option 1: vLLM — parse /metrics

vLLM has a ready gauge vllm:num_requests_waiting (queued requests) and vllm:num_requests_running (in flight). A threshold + a state file against spam:

import os, re, requests

NOTIFLY_URL   = os.environ["NOTIFLY_URL"]
NOTIFLY_TOKEN = os.environ["NOTIFLY_TOKEN"]
METRICS_URL   = os.environ.get("METRICS_URL", "http://localhost:8000/metrics")

WAITING_LIMIT = 20            # queued requests above which we alert
FLAG = "/tmp/infq-alerted.flag"

def notify(title, msg, prio):
    requests.post(f"{NOTIFLY_URL}/message",
                  params={"token": NOTIFLY_TOKEN},
                  json={"title": title, "message": msg, "priority": prio},
                  timeout=5)

def scrape(metric: str) -> float:
    text = requests.get(METRICS_URL, timeout=5).text
    # a line like: vllm:num_requests_waiting{...} 42.0
    for line in text.splitlines():
        if line.startswith(metric):
            m = re.search(r"\s([0-9.]+)\s*$", line)
            if m:
                return float(m.group(1))
    return 0.0

def handler(event, context):
    waiting = scrape("vllm:num_requests_waiting")
    running = scrape("vllm:num_requests_running")

    if waiting > WAITING_LIMIT:
        if not os.path.exists(FLAG):
            notify("🐌 Очередь инференса растёт",
                   f"В очереди {waiting:.0f} запросов (порог {WAITING_LIMIT}), "
                   f"на GPU считается {running:.0f}.\n"
                   f"Пользователи ждут — добавьте реплику или включите троттлинг.",
                   prio=8)
            open(FLAG, "w").close()
    elif os.path.exists(FLAG):
        os.remove(FLAG)          # the queue drained — clear the flag, ready for next time

    return {"statusCode": 200, "body": f"waiting={waiting} running={running}"}

priority=8 — a queue hits users directly (they are waiting), so it is loud but not a “ten”: the server is alive, just overloaded.

Option 2: TGI — its own metric names

Text Generation Inference exposes tgi_queue_size and a histogram tgi_request_queue_duration. Only the metric name changes:

waiting = scrape("tgi_queue_size")
# queue latency: take the histogram sum/count for the average

If the wait time matters more than the queue length, compute the average from the histogram _sum / _count and alert when the mean wait in the queue exceeds, say, 2 seconds.

Option 3: Ollama — no metrics, measure latency from outside

Ollama has no Prometheus endpoint, so we measure the queue indirectly — with an active monitor or a synthetic request that times itself. A rising latency of the canary request = a growing queue:

import time
t0 = time.time()
requests.post("http://localhost:11434/api/generate",
              json={"model": "llama3", "prompt": "ping", "stream": False},
              timeout=30)
dt = time.time() - t0
if dt > 5:
    notify("🐌 Ollama медленно отвечает",
           f"Канареечный запрос занял {dt:.1f}с — вероятно, очередь/перегрузка.",
           prio=7)

Option 4: heartbeat “the worker is draining the queue”

An inverted signal: let the worker send a heartbeat after each processed request. If the pings stop, the worker hung or died under load, and Notifly sends an alert on timeout even if /metrics is already unreachable.

What to put in the alert text

• the server/model name and the endpoint;
• num_requests_waiting / queue_size and the threshold;
• the mean wait time in the queue, if you compute it;
• a hint: add a replica, raise max_num_seqs, enable throttling at the entrance.