Empty and truncated model responses

The model returned an empty string, or the response was cut off mid-sentence (finish_reason=length) — individually that’s normal, but a spike almost always means trouble: someone set max_tokens=16, the provider degraded, or a new model version went silent on your prompt. Meanwhile the user sees a blank screen and your error metrics stay quiet, because the HTTP status is 200.

1. Count empties and truncations in a sliding window

Wrap every model call and keep a counter over the last N responses. Keep the state in Redis to survive restarts and span all instances.

import os, json, requests, redis

R   = redis.from_url(os.environ["REDIS_URL"])
KEY = "llm-completions"          # list of "ok" / "empty" / "trunc"
WIN = 200                        # window size
MIN = 50                         # don't alert until we have enough data

def observe(resp):
    """resp — a response object from an OpenAI-compatible API."""
    choice = resp.choices[0]
    content = (choice.message.content or "").strip()
    reason  = choice.finish_reason

    if reason == "length":
        tag = "trunc"
    elif not content:
        tag = "empty"
    else:
        tag = "ok"

    R.lpush(KEY, tag)
    R.ltrim(KEY, 0, WIN - 1)

    window = R.lrange(KEY, 0, WIN - 1)
    if len(window) < MIN:
        return

    empty = sum(1 for x in window if x == b"empty")
    trunc = sum(1 for x in window if x == b"trunc")
    n     = len(window)

    if empty / n > 0.10:
        notify("🕳️ Пустые ответы модели",
               f"Пустой content: {empty}/{n} ({int(empty/n*100)}%).\n"
               "Проверьте промпт, stop-последовательности и статус провайдера.",
               priority=8)
    elif trunc / n > 0.20:
        notify("✂️ Ответы обрезаются",
               f"finish_reason=length: {trunc}/{n} ({int(trunc/n*100)}%).\n"
               "Скорее всего, слишком маленький max_tokens.",
               priority=7)

def notify(title, message, priority):
    requests.post(f"{os.environ['NOTIFLY_URL']}/message",
                  params={"token": os.environ["NOTIFLY_TOKEN"]},
                  json={"title": title, "message": message, "priority": priority},
                  timeout=5)

2. Alert deduplication

A spike lasts minutes — you don’t want a push on every single call. Set a flag with a TTL so you alert at most once per 10 minutes:

def notify_once(title, message, priority, cooldown=600):
    flag = "alert:" + title
    if R.set(flag, "1", nx=True, ex=cooldown):
        notify(title, message, priority)

3. Synthetic canary

Don’t wait for real traffic — once a minute a cloud function with a timer-trigger calls the model with a fixed prompt that must produce a non-empty answer:

def handler(event, context):
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Ответь одним словом: тест"}],
        max_tokens=20,
    )
    c = (r.choices[0].message.content or "").strip()
    if not c or r.choices[0].finish_reason == "length":
        notify("🕳️ Канарейка: пустой ответ",
               f"finish_reason={r.choices[0].finish_reason}, len={len(c)}",
               priority=8)
    return {"statusCode": 200}

What to put in the alert text

• share of empty / truncated in the window and its size;
• the max_tokens value requests are using;
• model and provider;
• the last non-empty response for contrast.

Related recipes

• LLM provider availability — a common root cause.
• Repetition and looping — the opposite generation failure.