Agent tool-call timeout

When an AI agent “hangs”, the first suspect is the model. But more often the tool is to blame: a function-call went into your code (a DB query, HTTP to an external API, a shell command), and something froze there. The model waits for the tool result and looks completely stuck from the outside, though it’s fine itself — a specific tool call is stalled.

We instrument the tool executor: measure each call’s time and send a push when a tool repeatedly exceeds its budget.

Wrap the tool executor with a timeout

Each tool runs in a thread with a deadline; timeouts accumulate by tool name, and on repeated triggers a push goes out — “this exact tool is hanging”.

import os, time, collections, requests
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FTimeout

NOTIFLY_URL   = os.environ["NOTIFLY_URL"]
NOTIFLY_TOKEN = os.environ["NOTIFLY_TOKEN"]

TIMEOUTS   = {"db_query": 5, "web_search": 10, "run_shell": 30}  # budget, sec
DEFAULT    = 15
THRESHOLD  = 3        # this many timeouts of one tool = alert
WINDOW_SEC = 600

_pool = ThreadPoolExecutor(max_workers=8)
_timeouts = collections.defaultdict(collections.deque)   # tool -> timestamps

def notify(title, msg, prio):
    requests.post(f"{NOTIFLY_URL}/message",
                  params={"token": NOTIFLY_TOKEN},
                  json={"title": title, "message": msg, "priority": prio},
                  timeout=5)

def _record_timeout(tool, secs):
    now = time.time()
    dq = _timeouts[tool]
    dq.append(now)
    while dq and dq[0] < now - WINDOW_SEC:
        dq.popleft()
    if len(dq) >= THRESHOLD:
        notify(
            f"🧰 Тул '{tool}' виснет",
            f"{len(dq)} таймаутов за {WINDOW_SEC // 60} мин "
            f"(бюджет {secs}с). Агент выглядит зависшим, но дело в инструменте.",
            priority=8,
        )
        dq.clear()   # reset so we don't spam until it builds up again

def run_tool(name, fn, *args, **kwargs):
    """Run the tool with a deadline; on timeout — record it and re-raise."""
    budget = TIMEOUTS.get(name, DEFAULT)
    fut = _pool.submit(fn, *args, **kwargs)
    try:
        return fut.result(timeout=budget)
    except FTimeout:
        _record_timeout(name, budget)
        raise TimeoutError(f"tool '{name}' exceeded {budget}s")

# usage inside the agent loop, when the model asked to call a tool
def dispatch(tool_call):
    return run_tool(tool_call.name, TOOLS[tool_call.name], **tool_call.args)

The budget is per-tool (TIMEOUTS): a web search and a DB query have different normal times, a single threshold is useless. The alert fires on repeats (THRESHOLD), not on the first timeout: one slow call is normal, but three in a row is already “the tool is broken”.

Tell “the tool is hanging” from “the agent is looping”

A tool timeout is about an external tool. If the agent loop itself spins and burns tokens without progress, that’s a different case — stuck agent / loop via a heartbeat. Set up both: one catches “a tool hung”, the other “the model is going in circles”.

MCP tools are a frequent culprit

If the tools come from an MCP server, the hang is almost always in it (network, a third-party API behind it). Complement this with monitoring of the MCP itself — MCP server health — to tell “a specific call hung” from “the whole MCP server went down”.

What to put in the alert text

• the tool name and its time budget;
• the number of timeouts over the window;
• the arguments of the last hung call (careful with secrets);
• which agent step invoked it — to reproduce.