Skip to content
Notifly Notifly

The model answers in the wrong language

The user writes in Russian — and the model answers in English. Or on an Arabic question it replies in Farsi. For a single answer it’s annoying, for production it’s a quiet degradation: the user doesn’t complain, they just leave. This almost always surfaces as a jump after a deploy: someone edited the system prompt and dropped the “reply in the user’s language” line, or the model was swapped — and the new one has different default behavior.

We catch not a single slip but the rising share of mismatches — and send a push once per spike.

1. A lightweight language detector

Section titled “1. A lightweight language detector”

We detect the answer’s language and compare it to the expected one. The expected one comes from the request locale or the language of the user’s input. A full-blown option is langdetect / fasttext; but for the “Cyrillic vs Latin” pair a dependency-free script heuristic is enough.

import os, re, json, time, statistics, requests


def detect_lang(text):
    # cheap heuristic: the dominant alphabet.
    cyr = len(re.findall(r"[а-яёА-ЯЁ]", text))
    lat = len(re.findall(r"[a-zA-Z]", text))
    if cyr == 0 and lat == 0:
        return "unknown"
    return "ru" if cyr >= lat else "en"


# for more languages — a full detector:
#   from langdetect import detect
#   def detect_lang(text): return detect(text)   # 'ru', 'en', 'fa', ...


def is_mismatch(answer_text, expected_lang):
    got = detect_lang(answer_text)
    return got != "unknown" and got != expected_lang, got

expected_lang is the request’s language: either an explicit locale (request.locale) or the language detected from the user’s input with the same detect_lang.

2. A rolling window + a spike flag

Section titled “2. A rolling window + a spike flag”

We accumulate the share of mismatches in a window and send a push when it crosses a threshold. The flag file guarantees one alert per spike — otherwise every answer would fire a new one.

WIN = []                       # 1 = mismatch, 0 = ok
FLAG = "/tmp/lang_mismatch.flag"
THRESHOLD = 0.15               # >15% of answers in the wrong language
MIN_WINDOW = 40


def observe(answer_text, expected_lang, user_input=""):
    bad, got = is_mismatch(answer_text, expected_lang)
    WIN.append(1 if bad else 0)
    if len(WIN) > 200:
        WIN.pop(0)
    if len(WIN) < MIN_WINDOW:
        return
    rate = statistics.mean(WIN)
    if rate >= THRESHOLD and not os.path.exists(FLAG):
        open(FLAG, "w").write(str(time.time()))
        sample = redact(user_input)
        notify("🌐 Ответы не на том языке",
               f"{int(rate*100)}% ответов на неверном языке (окно {len(WIN)}).\n"
               f"Пример: ввод «{expected_lang}» → ответ «{got}»: {sample}\n"
               "Проверьте системный промпт и модель.",
               priority=8)
    elif rate < THRESHOLD * 0.5 and os.path.exists(FLAG):
        os.remove(FLAG)        # the spike has passed — clear the flag


def redact(text, n=60):
    # truncate and strip obvious PII
    text = re.sub(r"[\w.+-]+@[\w-]+\.[\w.]+", "[email]", text)
    text = re.sub(r"\+?\d[\d\s()-]{7,}\d", "[phone]", text)
    return (text[:n] + "…") if len(text) > n else text


def notify(title, message, priority):
    requests.post(f"{os.environ['NOTIFLY_URL']}/message",
                  params={"token": os.environ["NOTIFLY_TOKEN"]},
                  json={"title": title, "message": message, "priority": priority},
                  timeout=5)

In the push we include a redacted input → output sample — so the nature of the breakage is visible at once, without leaking personal data.

3. A baseline compare (vs yesterday)

Section titled “3. A baseline compare (vs yesterday)”

An absolute threshold won’t notice slow drift. Once a day we compare today’s mismatch share against yesterday’s — this catches “suddenly 20% answering in English where yesterday it was 2%”.

STATE = "/tmp/lang_mismatch_daily.json"


def daily_baseline_check(today_rate):
    prev = (json.load(open(STATE)) if os.path.exists(STATE) else {}).get("rate", today_rate)
    if today_rate - prev >= 0.10:      # +10 pp over 24 hours
        notify("📈 Скачок ответов не на том языке",
               f"Сегодня {int(today_rate*100)}% против {int(prev*100)}% вчера.\n"
               "Смотрите последний деплой: правка промпта или смена модели?",
               priority=9)
    json.dump({"rate": today_rate, "ts": time.time()}, open(STATE, "w"))

Run it via cron / systemd-timer or a scheduled cloud function on YC.

What to put in the push

Section titled “What to put in the push”
  • the mismatch share over the window + the expected → got pair;
  • a redacted input sample (no emails/phones);
  • the baseline compare (today vs yesterday);
  • a hint about the cause: system prompt edited?, model swapped?.

Keep the counter in shared storage (Redis) so the window survives restarts and covers all instances.

Section titled “Related recipes”