Stale knowledge index

RAG degrades not with a bang but with silence: the reindex stalled a week ago, yet the system keeps answering — just from stale data. No errors, no 500s. Catch two independent freshness signals: when the index was last built and how old the newest document in it is. The second matters more — it’s about the SLA, not the volume.

1. Freshness by last-reindex time

Every successful reindex run writes a timestamp. Once an hour a cloud function compares it against the SLA:

import os, time, requests

STAMP    = "/var/lib/rag/last-reindex.ts"
SLA_HOURS = 24                    # the reindex should run once a day

def handler(event, context):
    if not os.path.exists(STAMP):
        notify("🕰️ Index was never built", "No reindex timestamp found.", 8)
        return {"statusCode": 200}

    age_h = (time.time() - float(open(STAMP).read())) / 3600
    if age_h > SLA_HOURS:
        notify("🕰️ Index is stale",
               f"Last reindex: {age_h:.1f} h ago "
               f"(SLA {SLA_HOURS} h).\n"
               "RAG is answering from old data — check the cron/job.",
               priority=8)
    return {"statusCode": 200}

def notify(title, message, priority):
    requests.post(f"{os.environ['NOTIFLY_URL']}/message",
                  params={"token": os.environ["NOTIFLY_TOKEN"]},
                  json={"title": title, "message": message, "priority": priority},
                  timeout=5)

And at the end of a successful run the reindex job simply does:

open("/var/lib/rag/last-reindex.ts", "w").write(str(time.time()))

2. Freshness by the newest document in the index

The index may have been built regularly — but from a dead source (the crawler broke, the folder stopped filling up). Look at the freshest document in the collection:

FRESHNESS_HOURS = 48              # the freshest document shouldn't be older than 2 days

def check_newest_doc():
    # take the max updated_at across chunk metadata
    newest = vector_db_max_metadata("updated_at")     # your client → epoch seconds
    if newest is None:
        notify("🕰️ No documents in the index", "The collection is empty.", 9)
        return
    age_h = (time.time() - newest) / 3600
    if age_h > FRESHNESS_HOURS:
        notify("🕰️ Sources are stale",
               f"Freshest document in the index: {age_h:.1f} h ago "
               f"(SLA {FRESHNESS_HOURS} h).\n"
               "The index is being built, but the source stopped updating.",
               priority=7)

The two signals catch different failures: the first is “the job doesn’t run”, the second is “the job runs but no data arrives”.

3. A heartbeat as the cheapest option

If you don’t want a separate cloud function, a heartbeat with intervalSec slightly larger than your cron period is enough. Ping it at the end of a successful reindex; a missed ping produces the push “index didn’t update” on its own:

python -m rag.reindex && curl -fsS "$REINDEX_PING_URL" -o /dev/null

The heartbeat covers signal #1 for free; signal #2 still needs its own metadata check.

What to put in the alert text

• the age of the last reindex and the SLA;
• the age of the freshest document;
• the collection name;
• the expected cron period — so you can instantly tell “one run missed” from “stalled entirely”.

Related recipes

• Vector index rebuild finished — the successful run that writes the stamp.
• Vector DB / RAG infrastructure — a heartbeat from the indexing pipeline.
• Document ingestion failures — why fresh documents don’t arrive.