Background job failure notifications

A background job fails — and it’s easy not to find out: the user saw nothing, the worker log is quiet, and the email about the failed export never went out. The worst case is when a job exhausts all retries and simply “dies”. We catch exactly that moment — the final failure — and send a push with the job name and the error.

General rules

• Server-side only. Hook inside the worker; call notify() from there.
• Catch the final failure, not every retry. Otherwise one task with 5 retries will fire 5 pushes. You want the moment the attempts run out.
• Dedupe by job class. If “200 emails failed”, you want a single push “SendEmail × 200”, not 200 separate ones. Dedupe by the job class name.

Base wrapper

We take notify() from the overview page:

import os, requests

def notify(title, message, priority=5):
    requests.post(
        f"{os.environ['NOTIFLY_URL']}/message",
        params={"token": os.environ["NOTIFLY_TOKEN"]},
        json={"title": title, "message": message, "priority": priority},
        timeout=5,
    )

Celery (Python)

The task_failure signal fires when a task fails for good (retries raise Retry and don’t reach here). Dedupe by the task name:

import time, traceback
from collections import defaultdict
from celery.signals import task_failure
from notifly import notify

_last = defaultdict(float)   # task name -> time of last push
_COOLDOWN = 300              # 5 minutes per the same task class

@task_failure.connect
def on_task_failure(sender=None, task_id=None, exception=None, **kw):
    name = sender.name if sender else "unknown"
    if time.time() - _last[name] < _COOLDOWN:
        return                                   # dedupe by task class
    _last[name] = time.time()

    tb = "".join(traceback.format_exception(exception))[-1200:]
    notify(
        f"🔧 Джоба упала: {name}",
        f"task_id={task_id}\n{type(exception).__name__}: {exception}\n\n{tb}",
        8,
    )

BullMQ (Node.js)

We listen for the failed event but push only when all attempts are exhausted (attemptsMade >= opts.attempts):

const {Worker} = require('bullmq');
const {notify} = require('./notifly');

const lastSent = new Map();          // job name -> timestamp
const COOLDOWN_MS = 300_000;         // 5 minutes per class

const worker = new Worker('emails', async (job) => {
  await processJob(job);
}, {connection: {host: 'localhost', port: 6379}});

worker.on('failed', (job, err) => {
  if (!job) return;
  // send only on the final failure, not on every retry
  if (job.attemptsMade < (job.opts.attempts || 1)) return;

  const key = job.name;
  if (Date.now() - (lastSent.get(key) || 0) < COOLDOWN_MS) return;  // dedupe by class
  lastSent.set(key, Date.now());

  notify(
    `🔧 Джоба упала: ${job.name}`,
    `id=${job.id}, попыток=${job.attemptsMade}\n${err.message}\n\n${err.stack?.slice(0, 1200)}`,
    8
  );
});

Sidekiq (Ruby)

Sidekiq has death_handlers — called when a job exhausts all retries and moves to the Dead set. That’s exactly the moment we need:

require 'net/http'
require 'json'

$notifly_last = {}          # job class -> time of last push

def notifly(title, message, priority = 5)
  uri = URI("#{ENV['NOTIFLY_URL']}/message?token=#{ENV['NOTIFLY_TOKEN']}")
  Net::HTTP.post(uri, {title: title, message: message, priority: priority}.to_json,
                 'Content-Type' => 'application/json')
rescue => e
  warn "notifly failed: #{e.message}"
end

Sidekiq.configure_server do |config|
  config.death_handlers << ->(job, ex) {
    klass = job['class']
    # dedupe by job class: no more than once per 5 minutes
    next if $notifly_last[klass] && Time.now - $notifly_last[klass] < 300
    $notifly_last[klass] = Time.now

    notifly(
      "🔧 Джоба умерла: #{klass}",
      "jid=#{job['jid']}\n#{ex.class}: #{ex.message}",
      8
    )
  }
end

What else to send

• Growing queue depth. If unprocessed tasks > N, the worker isn’t keeping up; a quiet alert (priority 5).
• A long task got stuck. A job running many times longer than usual is worth a look; a heartbeat fits well here too.
• Retry handling. Separately you can report “task went to its 3rd retry” — quietly (priority 3), to catch instability before the final failure.

Benefits

• Silent failures become visible. A background job doesn’t shout at the user — now it shouts at you.
• Context in the message. Task name, id, and error — usually enough to know where to look.
• No storm. Dedupe by class: one broken deploy doesn’t turn into hundreds of pushes.