Token streaming stall

For streaming interfaces what matters isn’t “did the answer arrive”, but “how did it trickle”. Two symptoms spoil UX while staying invisible to ordinary monitoring:

• TTFT spiked — time to the first token became 5 seconds instead of 300 ms, the user stares at an empty cursor;
• the stream stalled midway — tokens were flowing, then the provider froze for seconds, the answer comes out in torn chunks.

The final status is 200 OK all the while. We catch the timings inside the stream.

Wrap the async stream and measure timings

We measure TTFT and the maximum gap between tokens; on exceeding either threshold — a push. COOLDOWN keeps the phone from being flooded on mass degradation.

import os, time, asyncio, requests
from openai import AsyncOpenAI   # same principle for anthropic.AsyncAnthropic

NOTIFLY_URL   = os.environ["NOTIFLY_URL"]
NOTIFLY_TOKEN = os.environ["NOTIFLY_TOKEN"]

TTFT_MS   = 2000     # time-to-first-token threshold
GAP_MS    = 1500     # threshold for the gap between adjacent tokens
COOLDOWN  = 120      # no more than one alert per 2 minutes

client = AsyncOpenAI()
_last_alert = 0.0

def notify(title, msg, prio):
    requests.post(f"{NOTIFLY_URL}/message",
                  params={"token": NOTIFLY_TOKEN},
                  json={"title": title, "message": msg, "priority": prio},
                  timeout=5)

def _maybe_alert(title, msg):
    global _last_alert
    now = time.time()
    if now - _last_alert > COOLDOWN:
        _last_alert = now
        notify(title, msg, priority=6)

async def stream_watched(**kwargs):
    """Streams the answer and watches TTFT and inter-token gaps on the fly."""
    t0 = time.time()
    last = t0
    ttft = None
    max_gap = 0.0

    stream = await client.chat.completions.create(stream=True, **kwargs)
    async for chunk in stream:
        delta = chunk.choices[0].delta.content
        if not delta:
            continue
        now = time.time()
        if ttft is None:
            ttft = (now - t0) * 1000
            if ttft > TTFT_MS:
                _maybe_alert(
                    f"🐌 TTFT {int(ttft)} мс",
                    f"Время до первого токена > {TTFT_MS} мс — стрим тупит на старте.")
        else:
            max_gap = max(max_gap, (now - last) * 1000)
        last = now
        yield delta

    if max_gap > GAP_MS:
        _maybe_alert(
            f"⏸️ Стрим залипал: пауза {int(max_gap)} мс",
            f"Между токенами была пауза > {GAP_MS} мс — рвётся вывод у пользователя.")

# usage
async def main():
    async for token in stream_watched(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Напиши хайку про латентность"}]):
        print(token, end="", flush=True)

asyncio.run(main())

TTFT is checked on the very first token — we react without waiting for the end of the answer. The gap between tokens (max_gap) catches exactly the “stall midway”, which gets smeared out and lost in the total time. Both alerts are priority=6: it’s about UX, not a full outage.

The threshold should depend on the model

For reasoning models (the o-series, extended thinking) TTFT is normally higher — they “think” before the first visible token. Don’t set one threshold for everyone: keep TTFT_MS per-model, otherwise you’ll bury yourself in false alerts on slow but healthy models.

Server side: aggregate, don’t alert on each

In production, compute p95 TTFT over a window of requests and alert on median degradation — as in the recipe LLM latency degradation. The per-request alert from the example above is good for a dev environment and debugging, but in production it turns into noise.

What to put in the alert text

• TTFT and the maximum inter-token gap in ms;
• the model and provider (their thresholds differ);
• the endpoint/region, if there are several;
• p95 TTFT over the last window — to tell a one-off spike from a trend.