Backup result notifications

“A backup nobody checks is not a backup.” The simplest way to check is to get a short success message every time and a loud one on error. If the daily “green” notification doesn’t arrive, something is broken.

Universal wrapper

Works for restic, borg, pg_dump, rsync, tar — anything. Save as /usr/local/bin/notifly-wrap:

#!/usr/bin/env bash
# Usage:
#   notifly-wrap "Job name" -- command to run
set -eu
set -a; source /etc/notifly.env; set +a

NAME="${1:?name required}"; shift
[ "${1:-}" = "--" ] && shift

START=$(date +%s)
HOST=$(hostname -s)
LOG=$(mktemp)

if "$@" >"$LOG" 2>&1; then
    DUR=$(( $(date +%s) - START ))
    SIZE=$(wc -l <"$LOG")
    /usr/local/bin/notifly-send \
        "✅ $NAME — OK ($HOST)" \
        "Длительность: ${DUR}s
Строк лога: $SIZE
Хост: $HOST" 3
    RC=0
else
    RC=$?
    DUR=$(( $(date +%s) - START ))
    TAIL=$(tail -c 1500 "$LOG")
    /usr/local/bin/notifly-send \
        "❌ $NAME — FAIL (rc=$RC) — $HOST" \
        "Длительность: ${DUR}s
Код: $RC

Последний вывод:
$TAIL" 9
fi

rm -f "$LOG"
exit "$RC"

sudo chmod +x /usr/local/bin/notifly-wrap

Example: daily PostgreSQL backup

/etc/cron.d/notifly-pg-backup:

SHELL=/bin/bash
0 3 * * * postgres /usr/local/bin/notifly-wrap "pg_dump app" -- \
    bash -c 'pg_dump -Fc app | gzip > /var/backups/pg/app-$(date +\%F).sql.gz'

Every morning at 03:00 you will receive either ”✅ pg_dump app — OK” or ”❌ pg_dump app — FAIL” with the tail of stderr.

Example: restic to S3

0 4 * * * root /usr/local/bin/notifly-wrap "restic backup /etc /home /srv" -- \
    /usr/local/bin/restic -r s3:s3.example.com/bucket backup /etc /home /srv

Example: rsync to a remote server

30 2 * * * root /usr/local/bin/notifly-wrap "rsync /var/www → backup-host" -- \
    rsync -aHAX --delete /var/www/ backup@backup-host:/data/www/

”Canary”: a message if the backup never ran

If the backup script itself failed and the notification wasn’t sent, you won’t know. The solution is a separate “canary timer” that checks once a day whether there is a fresh backup file:

/usr/local/bin/notifly-backup-canary:

#!/usr/bin/env bash
set -eu
set -a; source /etc/notifly.env; set +a

DIR=/var/backups/pg
LATEST=$(ls -t "$DIR"/*.sql.gz 2>/dev/null | head -1 || true)
NOW=$(date +%s)

if [ -z "$LATEST" ]; then
    /usr/local/bin/notifly-send \
        "🚨 Нет ни одного бэкапа в $DIR" \
        "Канарейка не нашла файлов бэкапа на $(hostname -s)." 10
    exit 1
fi

AGE=$(( NOW - $(stat -c %Y "$LATEST") ))
if [ "$AGE" -gt 90000 ]; then  # > 25 hours
    /usr/local/bin/notifly-send \
        "🚨 Старый бэкап на $(hostname -s)" \
        "Последний файл: $(basename "$LATEST")
Возраст: $((AGE/3600))ч." 10
fi

Run it via cron at 11:00 — after the night window:

0 11 * * * root /usr/local/bin/notifly-backup-canary

Benefits

• Backup health is visible every day. A green notification means you know the system is alive.
• On error you immediately see the cause — the last 1500 bytes of stderr go into the message body.
• The canary catches “silent” failures: even if the main script didn’t run, a separate timer will still shout.

What to improve next

• Store the backup size in the notification and warn if it suddenly becomes smaller or larger than usual by >50%.
• Include pg_verifybackup or restic check — so that only a verified backup is considered “green”.