Skip to content
Notifly Notifly

Multi-agent orchestration deadlocked

A single hung agent is agent-stuck: a watchdog on the loop and you’re done. But when there are several agents (LangGraph / CrewAI / AutoGen), a class of failures appears that a solo agent doesn’t have: each agent individually is alive and doing something, yet the system as a whole isn’t moving.

  • the researcher waits for data from the fetcher, the fetcher is hung on a failing tool;
  • the writer waits for a review from the critic, the critic waits for the final text from the writer — a classic circular wait;
  • the message queue is empty, all agents are in “waiting for input”, and nobody will write to anybody anymore.

Each agent’s local heartbeats stay quiet here — formally, everyone is “alive”. You need a signal about global progress.

Approach 1: a dead-man-switch on the whole orchestrator

Section titled “Approach 1: a dead-man-switch on the whole orchestrator”

We keep one global counter of completed steps. Any agent, having carried a step to completion, bumps mark_progress(). A separate watchdog watches: if the counter hasn’t grown for longer than STALL_SEC — the orchestration has stalled, send a push.

import os, time, threading, requests


class ProgressWatchdog:
    def __init__(self, stall_sec=300):
        self.stall_sec = stall_sec
        self.counter   = 0
        self.last_bump = time.time()
        self._fired    = False
        self._lock     = threading.Lock()


    def mark_progress(self, agent, step):
        # any agent calls this after finishing a real step
        with self._lock:
            self.counter += 1
            self.last_bump = time.time()
            self._fired = False   # progress resumed — reset the alert
            self._last = f"{agent}:{step}"


    def watch(self):
        while True:
            time.sleep(15)
            idle = time.time() - self.last_bump
            if idle >= self.stall_sec and not self._fired:
                self._fired = True
                notify("🕸️🛑 Оркестрация встала",
                       f"Ни один шаг не завершился {int(idle)}с "
                       f"(всего шагов: {self.counter}, последний: {self._last}). "
                       f"Похоже на deadlock.",
                       priority=9)




def notify(title, message, priority):
    requests.post(f"{os.environ['NOTIFLY_URL']}/message",
                  params={"token": os.environ["NOTIFLY_TOKEN"]},
                  json={"title": title, "message": message, "priority": priority},
                  timeout=5)

Hook it into the orchestrator and run the watcher in the background:

wd = ProgressWatchdog(stall_sec=300)
threading.Thread(target=wd.watch, daemon=True).start()


# in a graph node / agent callback — on step completion:
def on_step_complete(agent, step):
    wd.mark_progress(agent, step)   # global progress moved forward


orchestrator.run(...)   # finishes on its own — the daemon thread dies with the process

The key difference from agent-stuck: there the heartbeat is on one loop, here it’s one counter for the whole system. An individual agent may honestly grind away (so its own heartbeat keeps pinging), but if that doesn’t move the shared counter — the orchestration is dead anyway, and the watchdog will see it.

Approach 2: a Notifly heartbeat instead of your own thread

Section titled “Approach 2: a Notifly heartbeat instead of your own thread”

If the orchestrator is a short-lived process (a scheduled Yandex Cloud function, a CI job), you don’t need your own while thread: let a heartbeat watch from the outside. Create a heartbeat in the admin dashboard with interval = your STALL_SEC, and ping it only on real global progress:

PROGRESS_PING = os.environ["ORCH_PROGRESS_PING"]   # https://.../heartbeat/ping/H...


def on_step_complete(agent, step):
    # ping NOT on every agent twitch, but on a completed orchestration step
    requests.get(PROGRESS_PING, timeout=3)

As long as steps complete — the ping goes out, the heartbeat is green. The orchestration stalls — the ping stops, and after intervalSec + graceSec Notifly sends the alert itself. Bonus — it fires even on a full process crash (OOM, kill), when no internal watchdog is alive anymore. Enable a recovery message — when the system unblocks, you’ll get confirmation.

Approach 3: a circular-wait detector

Section titled “Approach 3: a circular-wait detector”

The dead-man-switch will say “we’re stuck”, but not why. Often the cause is a circular wait: A waits on B, B waits on A. Keep a “who-waits-on-whom” graph and look for a cycle in it — this catches a deadlock instantly, without waiting for the timeout.

class WaitGraph:
    def __init__(self):
        self.waits = {}   # agent -> whom it's currently waiting on (or None)


    def set_waiting(self, agent, waiting_on):
        self.waits[agent] = waiting_on
        cycle = self._find_cycle(agent)
        if cycle:
            notify("🔁🛑 Циклическое ожидание агентов",
                   f"Deadlock: {' → '.join(cycle)} → {cycle[0]}. "
                   f"Никто не разблокируется сам.",
                   priority=10)


    def clear(self, agent):
        self.waits[agent] = None   # the agent got what it waited for, moves on


    def _find_cycle(self, start):
        seen, path, cur = set(), [], start
        while cur is not None and cur not in seen:
            seen.add(cur); path.append(cur)
            cur = self.waits.get(cur)
        if cur is not None:               # returned to an already-visited node
            return path[path.index(cur):]
        return None


wg = WaitGraph()
wg.set_waiting("writer", "critic")   # writer waits on the critic
wg.set_waiting("critic", "writer")   # critic waits on the writer → cycle, push!

Combine them: the cycle detector catches an explicit deadlock instantly, while the dead-man-switch stays as insurance against quiet hangs that never made it into the wait graph (an agent hung on a tool, not on another agent).

What to put in the push

Section titled “What to put in the push”
  • how many steps completed in total and when the last one was (5 min ago);
  • the name of the last active agent and step;
  • if it’s a cycle — the wait chain itself, writer → critic → writer;
  • the message queue depth (an empty queue + everyone “waiting” = a sure deadlock).
Section titled “Related recipes”