Notifications for failed cron jobs

By default cron sends errors to the local root mail, which nobody reads. Replace that channel with Notifly: every failed job → a push to your phone with the last lines of output.

Method 1: universal wrapper

Use the ready-made notifly-wrap from the backup recipe. Cron jobs will look like this:

*/10 * * * * www-data /usr/local/bin/notifly-wrap "Sync ratings" -- /usr/local/bin/sync-ratings.sh
0    *  * * * root     /usr/local/bin/notifly-wrap "Yum updates" -- yum -y update
30   3  * * * root     /usr/local/bin/notifly-wrap "Logrotate"   -- logrotate -f /etc/logrotate.conf

The green “✅” can be suppressed by changing the wrapper — send only on rc != 0. A minimal “failures-only” version:

#!/usr/bin/env bash
set -eu
set -a; source /etc/notifly.env; set +a
NAME="${1:?}"; shift; [ "${1:-}" = "--" ] && shift
LOG=$(mktemp)
if ! "$@" >"$LOG" 2>&1; then
    RC=$?
    /usr/local/bin/notifly-send \
        "❌ cron: $NAME (rc=$RC) — $(hostname -s)" \
        "$(tail -c 1500 "$LOG")" 8
    rm -f "$LOG"; exit "$RC"
fi
rm -f "$LOG"

Method 2: global MAILTO via a hook

If you don’t want to edit all cron jobs, you can intercept the mail cron sends to root and forward it to Notifly.

Add to /etc/aliases (or /etc/postfix/aliases):

root: |/usr/local/bin/notifly-mailpipe

/usr/local/bin/notifly-mailpipe:

#!/usr/bin/env bash
set -eu
set -a; source /etc/notifly.env; set +a

BODY=$(cat | head -c 2000)
[ -z "$BODY" ] && exit 0

/usr/local/bin/notifly-send \
    "📬 cron-вывод на $(hostname -s)" \
    "$BODY" 6

sudo chmod +x /usr/local/bin/notifly-mailpipe
sudo newaliases

Now any cron job that outputs anything to stdout/stderr turns into a push notification. This is a “dirty but universal” method — useful to quickly cover an inherited server.

Method 3: canary check

Sometimes the cron daemon itself may fail to run (for example, after editing the crontab with an error). Create a simple “canary” that sends a message if there has been no heartbeat:

*/5 * * * * root touch /tmp/cron-alive

And a separate systemd timer:

/etc/systemd/system/notifly-cron-canary.timer

[Unit]
Description=Cron canary check

[Timer]
OnCalendar=*:0/30
Persistent=true

[Install]
WantedBy=timers.target

/etc/systemd/system/notifly-cron-canary.service

[Service]
Type=oneshot
EnvironmentFile=/etc/notifly.env
ExecStart=/bin/bash -c '\
  AGE=$(( $(date +%%s) - $(stat -c %%Y /tmp/cron-alive 2>/dev/null || echo 0) )); \
  if [ "$AGE" -gt 900 ]; then \
    /usr/local/bin/notifly-send "⚠️ Cron мёртв на $(hostname -s)" \
      "Heartbeat-файл старше 15 минут (возраст ${AGE}s)." 10; \
  fi'

sudo systemctl daemon-reload
sudo systemctl enable --now notifly-cron-canary.timer

Windows: Task Scheduler + wrapper

The Windows equivalent of cron is Task Scheduler. It tracks exit codes by itself, so instead of wrappers it’s easiest to subscribe to “task failed” events in the Microsoft-Windows-TaskScheduler/Operational log (Event ID 203).

Method 1: global watcher for all tasks

C:\scripts\Notifly-TaskFailed.ps1

param([string]$TaskName, [string]$ResultCode)
. C:\scripts\Notifly.ps1

Send-Notifly `
    -Title "❌ Task Scheduler: $TaskName — $env:COMPUTERNAME" `
    -Message "Код возврата: $ResultCode" `
    -Priority 8

Subscribe to all Task Scheduler failures:

$xml = @"
<QueryList>
  <Query Id="0" Path="Microsoft-Windows-TaskScheduler/Operational">
    <Select Path="Microsoft-Windows-TaskScheduler/Operational">
      *[System[(EventID=203 or EventID=323)]]
    </Select>
  </Query>
</QueryList>
"@
$Trigger = New-ScheduledTaskTrigger -AtStartup
$Trigger.Subscription = $xml
$Action = New-ScheduledTaskAction -Execute "powershell.exe" `
    -Argument "-NoProfile -ExecutionPolicy Bypass -File C:\scripts\Notifly-TaskFailed.ps1 -TaskName 'Unknown' -ResultCode 'see Event Log'"
Register-ScheduledTask -TaskName "Notifly Watch Failed Tasks" -Trigger $Trigger -Action $Action `
    -Principal (New-ScheduledTaskPrincipal -UserId "SYSTEM" -LogonType ServiceAccount -RunLevel Highest)

Method 2: per-task wrapper

An analogue of notifly-wrap — runs the command, captures the exit code, and on error sends the tail of the log:

C:\scripts\Notifly-Wrap.ps1

param([Parameter(Mandatory)] [string]$Name,
      [Parameter(Mandatory, ValueFromRemainingArguments)] [string[]]$Cmd)
. C:\scripts\Notifly.ps1

$log = New-TemporaryFile
$start = Get-Date
& $Cmd[0] $Cmd[1..($Cmd.Count-1)] *> $log
$rc = $LASTEXITCODE
$dur = ((Get-Date) - $start).TotalSeconds

if ($rc -ne 0) {
    $tail = Get-Content $log -Tail 30 | Out-String
    Send-Notifly -Title "❌ $Name (rc=$rc) — $env:COMPUTERNAME" `
                 -Message "Длительность: $([int]$dur)s`n`n$tail" `
                 -Priority 8
}
Remove-Item $log -Force
exit $rc

To use in a task: in the “Action” field specify powershell.exe -NoProfile -File C:\scripts\Notifly-Wrap.ps1 -Name "Ночной бэкап" robocopy ....

Benefits

• No more “lost” errors in local mail. Every failed job = a push notification.
• Context in your pocket — the last ~1500 characters of stderr, usually enough for diagnosis.
• Canary catches the worst case — when cron itself is not running.

Further improvements

• Separate Notifly channels for different types of jobs (ETL, backup, monitoring) — so they can be filtered in the admin and client apps.
• Store “time of last success” in Prometheus textfile collector and alert if it is not updated for N hours.