LLM endpoint region failover

If you keep the same model deployment in several regions (Azure OpenAI East US

• West Europe, mirrored endpoints with other providers), on failure the client routes traffic to a live region itself. It works — but invisibly. And you need to know about the change of active region: another region has its own latency, its own quotas, sometimes a different model version. A silent failover today is a quota surprise tomorrow.

Push on active region change

The rotator remembers the current active region; when traffic goes to another one — we send a push once per change, not on every request.

import os, json, requests
import openai

NOTIFLY_URL   = os.environ["NOTIFLY_URL"]
NOTIFLY_TOKEN = os.environ["NOTIFLY_TOKEN"]

REGIONS = [   # order = priority
    {"name": "eastus",       "base": "https://my-eastus.openai.azure.com"},
    {"name": "westeurope",   "base": "https://my-westeu.openai.azure.com"},
]
STATE = os.path.expanduser("~/.active-region.json")

def notify(title, msg, prio):
    requests.post(f"{NOTIFLY_URL}/message",
                  params={"token": NOTIFLY_TOKEN},
                  json={"title": title, "message": msg, "priority": prio},
                  timeout=5)

def _active():
    if os.path.exists(STATE):
        return json.load(open(STATE)).get("region")
    return None

def _set_active(name, reason):
    prev = _active()
    if name != prev:                       # region changed — report once
        json.dump({"region": name}, open(STATE, "w"))
        if prev is not None:
            notify(
                f"🌍 Регион сменился: {prev} → {name}",
                f"Активный LLM-эндпоинт теперь {name}. Причина: {reason}. "
                f"Проверьте квоты и версию модели в новом регионе.",
                priority=7,
            )

def chat(**kwargs):
    """Try regions in order; on the first live one, record it as active."""
    last_err = None
    for reg in REGIONS:
        try:
            client = openai.AzureOpenAI(
                azure_endpoint=reg["base"],
                api_key=os.environ["AZURE_KEY"],
                api_version="2024-10-21")
            resp = client.chat.completions.create(**kwargs)
            _set_active(reg["name"], reason="primary ok")
            return resp
        except Exception as e:
            last_err = f"{reg['name']}: {type(e).__name__}"
            continue
    raise RuntimeError(f"all regions unavailable ({last_err})")

resp = chat(model="gpt-4o-mini",
            messages=[{"role": "user", "content": "ping"}])

The push goes out on a state change, not on every request from the new region — otherwise after a failover the phone would go off nonstop. priority=7: not a catastrophe (the service works), but it’s worth stepping in — to double-check the new region’s quotas.

Back it with active monitors

The in-code logic only sees the regions you actually hit. To know the health of all regions regardless of your traffic, put an active monitor on each one’s health URL. Then you’ll see the backup region go down before it’s needed for failover, and not at the moment the primary is already down and the backup is dead too.

Don’t forget the failback

An A→B change is noticeable, but a silent B→A return is often missed. The same _set_active already catches both directions — just make sure the alert text is neutral (“region changed”), and you’ll learn about both the departure and the return to the primary region equally.

What to put in the alert text

• from where and to where the traffic moved (eastus → westeurope);
• the reason for the switch (5xx, timeout, region quota exhaustion);
• a reminder to double-check the quota and model version in the new region;
• a timestamp — for deduplication if the failover “flaps” A↔B.