Repetition and looping in generation

Degenerate generation — the model repeats one line dozens of times or gets stuck in a ”… and so on, and so on …” loop — is a classic symptom of a misconfiguration (temperature=0 with no repetition_penalty set), a broken fine-tune, or degradation of the model itself. The response is technically “successful” but useless, and without a detector only the user will notice.

1. An n-gram repetition detector

Measure the share of repeated n-grams in the text. A high share = degeneration.

import os, requests

def repetition_ratio(text: str, n: int = 3) -> float:
    """Share of non-unique n-grams. 0.0 = all unique, ~1.0 = degeneration."""
    tokens = text.split()
    if len(tokens) < n * 2:
        return 0.0
    grams = [tuple(tokens[i:i + n]) for i in range(len(tokens) - n + 1)]
    unique = len(set(grams))
    return 1.0 - unique / len(grams)

def longest_repeat_run(text: str) -> int:
    """Maximum number of consecutive identical lines."""
    lines = [l.strip() for l in text.splitlines() if l.strip()]
    best = cur = 1
    for a, b in zip(lines, lines[1:]):
        cur = cur + 1 if a == b else 1
        best = max(best, cur)
    return best

def check_output(answer: str):
    ratio = repetition_ratio(answer)
    run   = longest_repeat_run(answer)
    if ratio > 0.5 or run >= 4:
        notify("🔁 Зацикленная генерация",
               f"Повтор n-грамм: {int(ratio*100)}%, "
               f"подряд одинаковых строк: {run}.\n"
               f"Фрагмент: {answer[:200]}…\n\n"
               "Проверьте temperature / repetition_penalty и версию модели.",
               priority=7)
        return True
    return False

def notify(title, message, priority):
    requests.post(f"{os.environ['NOTIFLY_URL']}/message",
                  params={"token": os.environ["NOTIFLY_TOKEN"]},
                  json={"title": title, "message": message, "priority": priority},
                  timeout=5)

The 0.5 threshold is tuned for natural language; raise it for code — code legitimately repeats (imports, brackets).

2. A spike, not a one-off

A single degeneration is noise. Count the share of “bad” responses in a window, like in the hallucination spike:

import redis
R = redis.from_url(os.environ["REDIS_URL"])

def observe(answer: str):
    bad = 1 if (repetition_ratio(answer) > 0.5 or longest_repeat_run(answer) >= 4) else 0
    R.lpush("llm-degenerate", bad)
    R.ltrim("llm-degenerate", 0, 199)
    win = [int(x) for x in R.lrange("llm-degenerate", 0, 199)]
    if len(win) >= 50 and sum(win) / len(win) > 0.15:
        if R.set("alert:degenerate", "1", nx=True, ex=600):
            notify("🔁 Всплеск зацикливаний",
                   f"Вырожденных ответов: {int(sum(win)/len(win)*100)}% за {len(win)}",
                   priority=8)

What to put in the alert text

• the metrics: share of repeated n-grams and run length;
• a fragment of the degenerate output (first 200 chars);
• generation parameters (temperature, top_p, repetition_penalty);
• model and version.

Related recipes

• Empty and truncated responses — the other extreme.
• Hallucination spike — the general “share of bad in a window” approach.