Skip to content
Notifly Notifly

Agent keeps hammering a failing tool

An agent calls tools, and some calls fail — that’s normal: the network blinked, a rate limit, a random 500. The model retries, and usually it heals itself. The trouble starts when the same tool fails over and over: auth expired, the provider is degraded, the endpoint returns 500 steadily. The agent doesn’t realize the problem isn’t in its arguments, and stubbornly hammers the wall — burning tokens (see budget runaway) and going nowhere.

You need a tracker that, within a single run, counts the error share per tool and catches the moment failures stop being random.

Approach 1: error rate + consecutive streak

Section titled “Approach 1: error rate + consecutive streak”

We wrap tool execution. We accumulate per tool name: how many calls total, how many errors, how many failures in a row. We push on two triggers — either N failures in a row (persistent), or the error share crosses a threshold on a large enough number of calls.

import os, requests, collections


class ToolErrorTracker:
    def __init__(self, run_id, streak=4, rate=0.5, min_calls=6):
        self.run_id    = run_id
        self.streak    = streak      # this many failures in a row = persistent
        self.rate      = rate        # error share we treat as an emergency
        self.min_calls = min_calls   # don't panic on a small sample
        self.total     = collections.Counter()
        self.errors    = collections.Counter()
        self.consec    = collections.Counter()
        self._fired    = set()       # so we don't spam per tool


    def record(self, tool, ok, detail=""):
        self.total[tool] += 1
        if ok:
            self.consec[tool] = 0
            return
        self.errors[tool] += 1
        self.consec[tool] += 1


        if tool in self._fired:
            return


        # persistent: the same thing fails in a row
        if self.consec[tool] >= self.streak:
            self._fired.add(tool)
            notify("🛠️❌ Tool падает подряд",
                   f"run {self.run_id}: {tool} упал {self.consec[tool]} раз "
                   f"подряд. {detail} Похоже на persistent-сбой.",
                   priority=9)
            return


        # rate: many errors on a noticeable sample
        n, e = self.total[tool], self.errors[tool]
        if n >= self.min_calls and e / n >= self.rate:
            self._fired.add(tool)
            notify("🛠️⚠️ Высокий error-rate у tool-а",
                   f"run {self.run_id}: {tool} — {e}/{n} ошибок "
                   f"({e/n:.0%}). Агент буксует.",
                   priority=7)




def notify(title, message, priority):
    requests.post(f"{os.environ['NOTIFLY_URL']}/message",
                  params={"token": os.environ["NOTIFLY_TOKEN"]},
                  json={"title": title, "message": message, "priority": priority},
                  timeout=5)

Embedding it — a wrapper around the tool call:

tracker = ToolErrorTracker(run_id="refactor-auth-2026-07-01")


def run_tool(tool, **kwargs):
    try:
        result = tool.execute(**kwargs)
        tracker.record(tool.name, ok=True)
        return result
    except Exception as e:
        tracker.record(tool.name, ok=False, detail=str(e)[:120])
        raise

The two thresholds aren’t arbitrary: streak catches a fast, obvious failure (auth expired — it errors out instantly), rate catches a “flaky” tool that fails every other time and slowly eats the run.

Approach 2: telling transient apart from persistent

Section titled “Approach 2: telling transient apart from persistent”

Not every error deserves a push. 429 (rate limit) and 503 are usually transient — they clear up after backoff. But 401/403 (auth), a steady 500, or ConnectionRefused are almost always persistent — retrying is pointless, you need a human.

TRANSIENT = {408, 429, 502, 503, 504}   # a retry is appropriate
PERSISTENT = {400, 401, 403, 404, 500}  # a retry won't help


def classify(status):
    if status in PERSISTENT:
        return "persistent"
    if status in TRANSIENT:
        return "transient"
    return "unknown"


def record_http(tracker, tool, status):
    kind = classify(status)
    tracker.record(tool, ok=(status < 400), detail=f"HTTP {status} ({kind})")
    # persistent code — don't wait for a streak, raise the alarm right away
    if kind == "persistent" and tool not in tracker._fired:
        tracker._fired.add(tool)
        loud = 9 if status in (401, 403) else 8
        notify("🛠️🔒 Tool отдаёт persistent-ошибку",
               f"run {tracker.run_id}: {tool} → HTTP {status}. "
               f"{'Протухла авторизация?' if status in (401,403) else 'Endpoint лежит.'}",
               priority=loud)

The point: on a 401 there’s no sense waiting for four failures in a row — the token won’t recover on its own, wake a human on the first one. And on a 429 stay quiet, let the backoff do its job.

Approach 3: the external angle — no changes to the agent’s code

Section titled “Approach 3: the external angle — no changes to the agent’s code”

If the agent is a black box (a third-party CLI, someone else’s SDK) and you can’t wrap the calls, catch errors from the outside. Many tools (API gateways, MCP servers) already emit webhooks on 5xx — route them into the Webhook Router and it turns a spike of errors into a push. If there are no webhooks but the tool has a health check, put a Monitor on it: the tool’s endpoint goes red → push, and you already know why the run is stalling, before it even finishes.

The precision is lower (it’s about the service itself, not a specific run), but it works for any agent without a single line of edits.

What to put in the push

Section titled “What to put in the push”
  • run_id and the name of the failing tool — to find it in the logs;
  • transient or persistent, the HTTP code / text of the last error;
  • how many failures in a row / the error share (5/8, 62%);
  • a hint about the action: refresh the token, provider is down — pause the run.
Section titled “Related recipes”