Realtime/voice websocket drop

Voice and realtime interfaces hold a persistent websocket (OpenAI Realtime API, streaming ASR/TTS). One drop is normal, the client will reconnect. But a reconnect storm means a real problem: the user’s conversation tears mid-sentence. HTTP monitoring doesn’t see this at all — the connection lives not by requests but by seconds and minutes, and drops with its own close codes.

We catch abnormal close codes and a spike in the reconnect rate.

Count reconnects and abnormal close codes

We wrap the websocket client: 1000 is a normal close, everything else we count in a rolling window; a spike = a push.

import os, time, collections, asyncio, requests
import websockets   # pip install websockets

NOTIFLY_URL   = os.environ["NOTIFLY_URL"]
NOTIFLY_TOKEN = os.environ["NOTIFLY_TOKEN"]

WINDOW_SEC    = 120     # reconnect counting window
RECONNECT_MAX = 5       # this many reconnects per window = storm
COOLDOWN      = 180
NORMAL_CLOSE  = {1000, 1001}   # normal codes: normal / going away

_reconnects = collections.deque()
_last_alert = 0.0

def notify(title, msg, prio):
    requests.post(f"{NOTIFLY_URL}/message",
                  params={"token": NOTIFLY_TOKEN},
                  json={"title": title, "message": msg, "priority": prio},
                  timeout=5)

def _record_reconnect(code):
    global _last_alert
    now = time.time()
    _reconnects.append(now)
    while _reconnects and _reconnects[0] < now - WINDOW_SEC:
        _reconnects.popleft()
    abnormal = code not in NORMAL_CLOSE
    if (len(_reconnects) >= RECONNECT_MAX or abnormal) and now - _last_alert > COOLDOWN:
        _last_alert = now
        notify(
            f"🔌 Realtime WS рвётся: {len(_reconnects)} реконнектов",
            f"Последний close-код {code} за {WINDOW_SEC}с. "
            f"У пользователей рвётся голос/стрим.",
            priority=8,
        )

async def realtime_session(url, headers):
    """Hold the connection; on a drop count a reconnect and reconnect with backoff."""
    backoff = 1
    while True:
        try:
            async with websockets.connect(url, additional_headers=headers) as ws:
                backoff = 1                     # connected successfully — reset
                async for message in ws:
                    handle(message)
        except websockets.ConnectionClosed as e:
            _record_reconnect(e.code)
        except Exception:
            _record_reconnect(-1)               # -1 = drop without a close code
        await asyncio.sleep(min(backoff, 30))
        backoff *= 2                            # exponential backoff

def handle(message):
    ...   # parse audio/text frames

Normal codes (1000/1001) don’t alarm — it’s not just the network that drops, but the user themselves hanging up. The alert fires either on an abnormal close code (1006 abnormal closure, 1011 server error) or on the rate of reconnects. Exponential backoff in the client itself keeps a reconnect storm from turning into a DDoS against the provider.

Abnormal codes worth watching

• 1006 — abnormal closure, the connection dropped without a close frame (network, proxy);
• 1011 — internal server error on the provider’s side;
• 1013 — try again later, the provider is overloaded;
• provider-specific app codes — often an expired session token.

Different codes mean different causes: 1011/1013 about the provider’s side, 1006 about your network, app codes about authorization.

Back it with a session heartbeat

While the connection lives, send a rare heartbeat “voice session active”. If the worker holding the websocket dies entirely (crashed process, OOM), there will be no reconnects at all — nothing left to count. A missed heartbeat catches exactly that silence, which the reconnect counter won’t see.

What to put in the alert text

• the last close code and its meaning;
• the number of reconnects over the window;
• how many sessions are affected (if you hold several);
• the connection uptime before the drop — a short uptime hints at auth/network.