Chunking and index config drift

The index was built with one chunking config (chunk_size=512, overlap=64, text-embedding-3-small), while queries go out with a different one — someone tweaked the config and redeployed the query service without reindexing. Nothing formally breaks: vectors have the same dimension, search works, but the neighbors in the results are now garbage. This is one of the least visible ways to wreck RAG quality.

1. Record the build config inside the index

During reindex, put the config fingerprint into the collection metadata (or into a service point). This is the single source of truth for “what it was built with”:

import os, json, hashlib, requests

def build_config():
    return {
        "chunk_size":   int(os.environ["CHUNK_SIZE"]),
        "chunk_overlap": int(os.environ["CHUNK_OVERLAP"]),
        "embed_model":  os.environ["EMBED_MODEL"],
        "embed_dim":    int(os.environ["EMBED_DIM"]),
        "splitter":     os.environ.get("SPLITTER", "recursive"),
    }

def config_fingerprint(cfg: dict) -> str:
    return hashlib.sha256(json.dumps(cfg, sort_keys=True).encode()).hexdigest()[:12]

def save_index_config():
    cfg = build_config()
    # your client: save into collection metadata
    vector_db_set_meta("index_config", json.dumps(cfg))
    vector_db_set_meta("index_config_fp", config_fingerprint(cfg))

2. Compare query config against build config on startup

On startup (and once an hour) the query service reads the fingerprint from the index and compares it with its own. A mismatch is an immediate push:

def check_config_drift():
    build_fp = vector_db_get_meta("index_config_fp")
    query_fp = config_fingerprint(build_config())

    if build_fp and build_fp != query_fp:
        build_cfg = json.loads(vector_db_get_meta("index_config") or "{}")
        query_cfg = build_config()
        diff = [f"{k}: индекс={build_cfg.get(k)} → запрос={query_cfg.get(k)}"
                for k in query_cfg if build_cfg.get(k) != query_cfg.get(k)]
        notify("🧩 Дрейф конфига RAG",
               "Конфиг сборки индекса ≠ конфиг на запросе:\n"
               + "\n".join(f"• {d}" for d in diff)
               + "\n\nЛибо переиндексируйте, либо откатите query-конфиг.",
               priority=9)
        return False
    return True

def notify(title, message, priority):
    requests.post(f"{os.environ['NOTIFLY_URL']}/message",
                  params={"token": os.environ["NOTIFLY_TOKEN"]},
                  json={"title": title, "message": message, "priority": priority},
                  timeout=5)

The startup check catches the mismatch right after a deploy — before the first user gets a garbage retrieval.

3. A special case: the embeddings model diverged

Changing embed_model is the most dangerous: with a matching dimension, search won’t even break — it just returns meaningless neighbors. Pull the model out into a separate explicit check at the highest priority:

if build_cfg.get("embed_model") != query_cfg.get("embed_model"):
    notify("🧩 Разошлась модель embeddings",
           f"Индекс: {build_cfg.get('embed_model')} → "
           f"запрос: {query_cfg.get('embed_model')}. Ретривал вернёт мусор.",
           priority=10)

What to put in the alert text

• the fields that diverged, with both sides (index → query);
• the config fingerprints;
• the collection name;
• when the index was last built (see stale index).

Related recipes

• Provider changed the embeddings version — when the provider changes the model, not you.
• RAG retrieval returns nothing — a common consequence of drift.
• Stale knowledge index — a neighboring index-quality signal.