Vector index rebuild finished

A full index rebuild (crawling → chunking → embeddings → upsert) can run from minutes to hours. Checking the terminal every five minutes isn’t an option, and a silent reindex failure means your RAG answers from stale data until morning. Ping yourself the moment it finishes — successfully or not.

1. A wrapper around the reindex job

import os, time, requests

def reindex_all(source_docs):
    t0 = time.time()
    total   = len(source_docs)
    indexed = 0
    skipped = []

    for doc in source_docs:
        try:
            chunks = chunk_and_embed(doc)      # your logic
            upsert_to_vector_db(chunks)
            indexed += 1
        except Exception as e:
            skipped.append((doc["id"], f"{type(e).__name__}: {e}"))

    dur = int(time.time() - t0)
    ok  = len(skipped) == 0

    lines = "\n".join(f"• {i}: {err}" for i, err in skipped[:10])
    notify(
        f"{'🟢' if ok else '🟡'} Переиндексация завершена",
        f"Документов: {indexed}/{total}\n"
        f"Пропущено: {len(skipped)}\n"
        f"Длительность: {dur // 60} мин {dur % 60} с\n"
        + (f"\nПервые ошибки:\n{lines}" if skipped else ""),
        priority=6 if ok else 8,
    )
    return {"indexed": indexed, "skipped": len(skipped), "dur": dur}

def notify(title, message, priority):
    requests.post(f"{os.environ['NOTIFLY_URL']}/message",
                  params={"token": os.environ["NOTIFLY_TOKEN"]},
                  json={"title": title, "message": message, "priority": priority},
                  timeout=5)

This is a special case of the general long-task completion pattern: we wait for the work to end and send one meaningful push instead of tailing logs.

2. From a shell script (cron)

If your reindex is reindex.sh, wrap it without a single line of Python:

#!/usr/bin/env bash
set -a; source ~/.notifly.env; set +a
START=$(date +%s)

if OUT=$(python -m rag.reindex 2>&1); then
  RC=0; ICON="🟢"
else
  RC=$?; ICON="🟡"
fi
DUR=$(( $(date +%s) - START ))

curl -fsS "$NOTIFLY_URL/message?token=$NOTIFLY_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg t "$ICON Переиндексация (rc=$RC)" \
              --arg m "Длительность: ${DUR}s

$(echo "$OUT" | tail -c 800)" \
              --argjson p $([ $RC -eq 0 ] && echo 6 || echo 8) \
              '{title:$t, message:$m, priority:$p}')" >/dev/null

3. Plus a heartbeat if the job is scheduled

The push arrives only when the job reaches the end. If the process was OOM-killed halfway, there’s no push. Back it up with a heartbeat: ping it at the very end of a successful run, and a missed ping will produce the alert on its own.

python -m rag.reindex && curl -fsS "$REINDEX_PING_URL" -o /dev/null

What to put in the push text

• documents indexed / total;
• how many were skipped and why (the first few errors);
• duration;
• collection size before/after (see chunk drift).

Related recipes

• Vector DB / RAG infrastructure — monitoring the DB itself.
• Document ingestion failures — detail on skipped documents.
• Stale knowledge index — when the reindex never ran at all.