Nginx 5xx error spike notification

A spike of 5xx means users are seeing errors right now: an upstream went down, the PHP-FPM worker pool ran out, the database dropped, or a deploy went sideways. You can see it in Grafana, but nobody looks at Grafana at three in the morning. Let nginx call you the moment the rate of 502/503/504 crosses a threshold.

What we’ll do

Once a minute the script reads the fresh lines of /var/log/nginx/access.log, counts the responses with a 5xx status in them (field $9 of the combined format) and:

• if > 10 5xx per minute — sends a normal warning (priority 5);
• if > 50 5xx per minute — sends a loud alert (priority 9).

To avoid re-reading the whole log every time (and re-counting old errors), we store a byte offset in a state file — the position up to which the log has already been read. On each run we read only the “tail” from that offset to the end of the file. This works even on gigabyte-sized logs: we never parse more than what accumulated over a minute.

We put the top offending URLs and a breakdown by status code into the message — so it’s immediately clear whether the whole site is down or a single endpoint failed.

Script

Save as /usr/local/bin/notifly-nginx-5xx:

#!/usr/bin/env bash
set -eu
set -a; source /etc/notifly.env; set +a

LOG=/var/log/nginx/access.log
WARN=10          # 5xx per minute -> warning
CRIT=50          # 5xx per minute -> critical alert
STATE_DIR=/var/lib/notifly-nginx
OFFSET_FILE="$STATE_DIR/offset"
FLAG=/tmp/notifly-nginx-5xx.flag
DEBOUNCE=600     # don't repeat an alert more often than once every 10 minutes
mkdir -p "$STATE_DIR"

HOST=$(hostname -s)

# Current log size and the position we already read up to last time
SIZE=$(stat -c %s "$LOG" 2>/dev/null || echo 0)
OFFSET=$(cat "$OFFSET_FILE" 2>/dev/null || echo 0)

# The log may have been rotated by logrotate: if it got shorter — read from the start
[ "$OFFSET" -gt "$SIZE" ] && OFFSET=0

# Read only the new bytes, updating the offset in one go
CHUNK=$(mktemp)
trap 'rm -f "$CHUNK"' EXIT
dd if="$LOG" bs=1 skip="$OFFSET" count=$((SIZE - OFFSET)) 2>/dev/null > "$CHUNK" || true
echo "$SIZE" > "$OFFSET_FILE"

# Count 5xx: $9 in the combined format is the HTTP response status
COUNT=$(awk '$9 ~ /^5[0-9][0-9]$/' "$CHUNK" | wc -l)

LEVEL=ok
[ "$COUNT" -gt "$WARN" ] && LEVEL=warn
[ "$COUNT" -gt "$CRIT" ] && LEVEL=crit
[ "$LEVEL" = ok ] && exit 0

# Debounce: while the flag is younger than DEBOUNCE seconds — stay quiet
NOW=$(date +%s)
LAST=$(stat -c %Y "$FLAG" 2>/dev/null || echo 0)
[ $((NOW - LAST)) -lt "$DEBOUNCE" ] && exit 0
touch "$FLAG"

# Top status codes and top URLs among the 5xx requests
BY_CODE=$(awk '$9 ~ /^5[0-9][0-9]$/{print $9}' "$CHUNK" | sort | uniq -c | sort -rn | head -5)
BY_URL=$(awk '$9 ~ /^5[0-9][0-9]$/{print $7}' "$CHUNK" | sort | uniq -c | sort -rn | head -5)

if [ "$LEVEL" = crit ]; then
    /usr/local/bin/notifly-send \
        "🔥 КРИТИЧНО: $COUNT×5xx/мин на $HOST" \
        "nginx отдал $COUNT ошибок 5xx за последнюю минуту.

По кодам:
$BY_CODE

Топ URL:
$BY_URL" 9
else
    /usr/local/bin/notifly-send \
        "⚠️ Всплеск 5xx на $HOST" \
        "nginx отдал $COUNT ошибок 5xx за последнюю минуту (порог $WARN).

По кодам:
$BY_CODE

Топ URL:
$BY_URL" 5
fi

Make it executable:

sudo chmod +x /usr/local/bin/notifly-nginx-5xx

Run on schedule

Using cron

sudo crontab -e

Add the line — runs every minute:

* * * * * /usr/local/bin/notifly-nginx-5xx >/dev/null 2>&1

Using systemd timer (recommended)

/etc/systemd/system/notifly-nginx-5xx.service:

[Unit]
Description=Notifly nginx 5xx rate check

[Service]
Type=oneshot
ExecStart=/usr/local/bin/notifly-nginx-5xx

/etc/systemd/system/notifly-nginx-5xx.timer:

[Unit]
Description=Run notifly-nginx-5xx every minute

[Timer]
OnBootSec=2min
OnUnitActiveSec=1min

[Install]
WantedBy=timers.target

Activate it:

sudo systemctl daemon-reload
sudo systemctl enable --now notifly-nginx-5xx.timer
sudo systemctl list-timers notifly-nginx-5xx.timer

Windows: PowerShell + Task Scheduler

On Windows servers the role of the nginx access log is played by IIS logs: they live in C:\inetpub\logs\LogFiles\W3SVC*\ in W3C format, where the response status is stored in the sc-status field. The field order is defined by the #Fields: line at the start of the file, but in a typical configuration sc-status comes after the method and URI. Below we read the “tail” of the fresh log file, find the index of the sc-status field from the header, and count the 5xx.

The script uses the common function Send-Notifly from the sysadmin template/index.

C:\scripts\Notifly-IIS-5xx.ps1

. C:\scripts\Notifly.ps1

$Warn = 10
$Crit = 50
$FlagFile = "$env:TEMP\notifly-iis-5xx.flag"
$Host = $env:COMPUTERNAME

# The most recent IIS log file across all sites
$log = Get-ChildItem "C:\inetpub\logs\LogFiles\W3SVC*\*.log" -ErrorAction SilentlyContinue |
       Sort-Object LastWriteTime -Descending | Select-Object -First 1
if (-not $log) { exit 0 }

# Last ~2000 lines — with a margin for a minute of traffic
$lines = Get-Content $log.FullName -Tail 2000

# Take the index of the sc-status field from the "#Fields: ..." header
$header = $lines | Where-Object { $_ -like "#Fields:*" } | Select-Object -Last 1
if (-not $header) { exit 0 }
$fields = ($header -replace "^#Fields:\s*", "") -split "\s+"
$statusIdx = [Array]::IndexOf($fields, "sc-status")
$uriIdx    = [Array]::IndexOf($fields, "cs-uri-stem")
if ($statusIdx -lt 0) { exit 0 }

# Only lines from the current minute (the date/time fields come first in a W3C log)
$minute = (Get-Date).ToUniversalTime().ToString("yyyy-MM-dd HH:mm")
$rows = $lines |
    Where-Object { $_ -and $_[0] -ne "#" -and $_.StartsWith($minute) } |
    ForEach-Object { , ($_ -split "\s+") } |
    Where-Object { $_[$statusIdx] -match "^5\d\d$" }

$count = @($rows).Count
$level = "ok"
if ($count -gt $Warn) { $level = "warn" }
if ($count -gt $Crit) { $level = "crit" }
if ($level -eq "ok") { exit 0 }

# Debounce for 10 minutes
if (Test-Path $FlagFile) {
    $age = (Get-Date) - (Get-Item $FlagFile).LastWriteTime
    if ($age.TotalMinutes -lt 10) { exit 0 }
}
New-Item -ItemType File -Path $FlagFile -Force | Out-Null

$byCode = $rows | Group-Object { $_[$statusIdx] } | Sort-Object Count -Descending |
          Select-Object -First 5 | ForEach-Object { "  $($_.Count)× $($_.Name)" }
$byUrl = if ($uriIdx -ge 0) {
    $rows | Group-Object { $_[$uriIdx] } | Sort-Object Count -Descending |
    Select-Object -First 5 | ForEach-Object { "  $($_.Count)× $($_.Name)" }
} else { @() }

$msg = "IIS отдал $count ошибок 5xx за последнюю минуту.`n`nПо кодам:`n" +
       ($byCode -join "`n") + "`n`nТоп URL:`n" + ($byUrl -join "`n")

if ($level -eq "crit") {
    Send-Notifly -Title "🔥 КРИТИЧНО: $count×5xx/мин — $Host" -Message $msg -Priority 9
} else {
    Send-Notifly -Title "⚠️ Всплеск 5xx на $Host" -Message $msg -Priority 5
}

Register in Task Scheduler (as administrator, every minute as SYSTEM):

$Action  = New-ScheduledTaskAction `
    -Execute "powershell.exe" `
    -Argument "-NoProfile -ExecutionPolicy Bypass -File C:\scripts\Notifly-IIS-5xx.ps1"
$Trigger = New-ScheduledTaskTrigger -Once -At (Get-Date) `
    -RepetitionInterval (New-TimeSpan -Minutes 1)
$Princ   = New-ScheduledTaskPrincipal -UserId "SYSTEM" -LogonType ServiceAccount -RunLevel Highest
Register-ScheduledTask -TaskName "Notifly IIS 5xx" `
    -Action $Action -Trigger $Trigger -Principal $Princ -Description "Notifly: всплеск 5xx в IIS"

Benefits

• You learn about the problem before your users do — the phone rings the moment of the spike, not when a support ticket arrives.
• The breakdown by code and URL right in the push — you immediately see whether the whole site is down (502 across all URLs) or a single endpoint broke (504 on /api/report).
• A byte offset = cheap at any scale — each minute only the new “tail” of the log is read, so the script is equally fast on a 10 MB and a 10 GB access log.
• Debounce prevents spam: during a prolonged outage you’ll get one message every 10 minutes, not 60 in a row.

What to improve next

• Send ”✅ 5xx are back to normal” once the rate drops below the threshold again after an alert — by analogy with the ok → warn → crit logic from the disk-space recipe.
• Attach the full list of failing requests via the extras field, so you don’t bloat the push text.
• Count not the absolute number but the share of 5xx among all requests — this is more accurate on sites with very different traffic during the day and at night.
• Link it with the service-down notification: a 5xx spike often follows a downed upstream or PHP-FPM.
• For priorities and message fields, see the sending pushes and monitoring sections.