Skip to content
Notifly Notifly

RAG retrieval returns nothing

Retrieval returned zero chunks above the similarity threshold — and the model answers “there’s no information in the provided context” or, worse, hallucinates. A single empty retrieval is normal (a query outside the knowledge base), but a spike in the empty rate almost always means something broke: reindexed into the wrong collection, embeddings changed, or the score_threshold was set too high. All HTTP statuses stay green.

1. Count the empty-retrieval rate in a window

Section titled “1. Count the empty-retrieval rate in a window”

Wrap the real retrieval and tag every query as empty or not.

import os, requests, redis


R         = redis.from_url(os.environ["REDIS_URL"])
KEY       = "rag-retrieval"      # "hit" / "miss"
WIN       = 300
MIN       = 40
THRESHOLD = 0.75                 # similarity threshold


def retrieve(query_vec, top_k=8):
    hits = vector_search(query_vec, top_k=top_k)        # your client
    good = [h for h in hits if h.score >= THRESHOLD]


    R.lpush(KEY, b"hit" if good else b"miss")
    R.ltrim(KEY, 0, WIN - 1)
    _maybe_alert()
    return good


def _maybe_alert():
    win = R.lrange(KEY, 0, WIN - 1)
    if len(win) < MIN:
        return
    miss = sum(1 for x in win if x == b"miss")
    rate = miss / len(win)
    # normal baseline of empty retrievals is 5–15%; alarm on a clear excess
    if rate > 0.35 and R.set("alert:retrieval-empty", "1", nx=True, ex=900):
        notify("🫥 RAG: пустой ретривал",
               f"Запросов без контекста: {miss}/{len(win)} ({int(rate*100)}%).\n"
               f"Порог похожести: {THRESHOLD}.\n\n"
               "Проверьте: та ли коллекция, совпадает ли модель embeddings, "
               "не задран ли cutoff.",
               priority=8)


def notify(title, message, priority):
    requests.post(f"{os.environ['NOTIFLY_URL']}/message",
                  params={"token": os.environ["NOTIFLY_TOKEN"]},
                  json={"title": title, "message": message, "priority": priority},
                  timeout=5)

2. A synthetic canary on known queries

Section titled “2. A synthetic canary on known queries”

Don’t wait for prod to degrade. Every 10 minutes a cloud function runs queries for which we know for sure the context is in the base:

CANARIES = [
    "как сбросить пароль",
    "какие способы оплаты вы поддерживаете",
    "политика возврата средств",
]


def handler(event, context):
    empty = []
    for q in CANARIES:
        vec  = embed(q)
        good = [h for h in vector_search(vec, top_k=8) if h.score >= THRESHOLD]
        if not good:
            empty.append(q)
    if empty:
        notify("🫥 Канарейка ретривала пуста",
               "Нет контекста для заведомо покрытых запросов:\n"
               + "\n".join(f"• {q}" for q in empty),
               priority=9)
    return {"statusCode": 200}

If baseline queries stopped finding context, the index is almost certainly empty or incompatible, and that needs fixing immediately.

What to put in the alert text

Section titled “What to put in the alert text”
  • share of empty retrievals and the window size;
  • the current score_threshold;
  • collection name and embeddings model (cross-check with the drift check);
  • a couple of example queries left without context.
Section titled “Related recipes”