Skip to content
Notifly Notifly

Low disk space notification

A full disk is one of the most common causes of service failures: PostgreSQL stops accepting queries, nginx — stops writing logs, containers cannot start. It’s better to learn about the problem days in advance than to read a post-mortem.

What we’ll do

Section titled “What we’ll do”

The script checks the usage of all mounted disks every 15 minutes and:

  • if usage is ≥ 80 % — sends a normal warning;
  • if usage is ≥ 95 % — sends a high-priority message.

To avoid receiving the same message every 15 minutes, we’ll store the “already sent” state in /var/lib/notifly-disk/.

Script

Section titled “Script”

Save as /usr/local/bin/notifly-disk-check:

#!/usr/bin/env bash
set -eu
set -a; source /etc/notifly.env; set +a


WARN=80
CRIT=95
STATE_DIR=/var/lib/notifly-disk
mkdir -p "$STATE_DIR"


HOST=$(hostname -s)


# Only consider real filesystems, excluding tmpfs/devtmpfs
df -PH -x tmpfs -x devtmpfs -x overlay | tail -n +2 | while read -r line; do
    USE=$(echo "$line" | awk '{print $5}' | tr -d '%')
    MNT=$(echo "$line" | awk '{print $6}')
    SIZE=$(echo "$line" | awk '{print $2}')
    AVAIL=$(echo "$line" | awk '{print $4}')


    KEY=$(echo "$MNT" | tr '/' '_')
    STATE_FILE="$STATE_DIR/$KEY"
    LAST=$(cat "$STATE_FILE" 2>/dev/null || echo 0)


    LEVEL=ok
    [ "$USE" -ge "$WARN" ] && LEVEL=warn
    [ "$USE" -ge "$CRIT" ] && LEVEL=crit


    if [ "$LEVEL" != "$LAST" ]; then
        case "$LEVEL" in
            warn)
                /usr/local/bin/notifly-send \
                    "💾 Диск заполнен на $USE% — $HOST" \
                    "Раздел $MNT: занято $USE%, свободно $AVAIL из $SIZE." 5
                ;;
            crit)
                /usr/local/bin/notifly-send \
                    "🔥 КРИТИЧНО: диск $USE% — $HOST" \
                    "Раздел $MNT: занято $USE%, свободно $AVAIL из $SIZE. Срочно освободите место!" 9
                ;;
            ok)
                /usr/local/bin/notifly-send \
                    "✅ Диск в норме — $HOST" \
                    "Раздел $MNT снова в пределах нормы ($USE%)." 3
                ;;
        esac
        echo "$LEVEL" > "$STATE_FILE"
    fi
done

Make it executable:

Окно терминала
sudo chmod +x /usr/local/bin/notifly-disk-check

Run on schedule

Section titled “Run on schedule”

Using cron

Section titled “Using cron”
Окно терминала
sudo crontab -e

Add the line:

*/15 * * * * /usr/local/bin/notifly-disk-check >/dev/null 2>&1
Section titled “Using systemd timer (recommended)”

/etc/systemd/system/notifly-disk.service:

[Unit]
Description=Notifly disk space check


[Service]
Type=oneshot
ExecStart=/usr/local/bin/notifly-disk-check

/etc/systemd/system/notifly-disk.timer:

[Unit]
Description=Run notifly-disk every 15 minutes


[Timer]
OnBootSec=5min
OnUnitActiveSec=15min


[Install]
WantedBy=timers.target

Activate it:

Окно терминала
sudo systemctl daemon-reload
sudo systemctl enable --now notifly-disk.timer
sudo systemctl list-timers notifly-disk.timer

Windows: PowerShell + Task Scheduler

Section titled “Windows: PowerShell + Task Scheduler”

Equivalent script for Windows servers. Uses the common function Send-Notifly from the sysadmin template/index.

C:\scripts\Notifly-Disk-Check.ps1
. C:\scripts\Notifly.ps1


$Warn = 80
$Crit = 95
$StateDir = "C:\ProgramData\Notifly\disk-state"
New-Item -ItemType Directory -Path $StateDir -Force | Out-Null
$Host = $env:COMPUTERNAME


Get-CimInstance Win32_LogicalDisk -Filter "DriveType=3" | ForEach-Object {
    if (-not $_.Size) { return }
    $usedPct = [int]((($_.Size - $_.FreeSpace) / $_.Size) * 100)
    $freeGB  = [math]::Round($_.FreeSpace / 1GB, 1)
    $sizeGB  = [math]::Round($_.Size     / 1GB, 1)
    $letter  = $_.DeviceID.TrimEnd(":")


    $stateFile = Join-Path $StateDir "$letter.txt"
    $last = if (Test-Path $stateFile) { Get-Content $stateFile -Raw } else { "ok" }


    $level = "ok"
    if ($usedPct -ge $Warn) { $level = "warn" }
    if ($usedPct -ge $Crit) { $level = "crit" }


    if ($level -ne $last.Trim()) {
        switch ($level) {
            "warn" { Send-Notifly -Title "💾 Диск $letter`: $usedPct% — $Host" `
                                  -Message "Свободно $freeGB ГБ из $sizeGB ГБ." -Priority 5 }
            "crit" { Send-Notifly -Title "🔥 КРИТИЧНО: диск $letter`: $usedPct% — $Host" `
                                  -Message "Свободно всего $freeGB ГБ. Срочно освободите место!" -Priority 9 }
            "ok"   { Send-Notifly -Title "✅ Диск $letter в норме — $Host" `
                                  -Message "Раздел $letter​: занято $usedPct%." -Priority 3 }
        }
        $level | Set-Content $stateFile
    }
}

Register in Task Scheduler (as administrator, every 15 minutes as SYSTEM):

Окно терминала
$Action  = New-ScheduledTaskAction `
    -Execute "powershell.exe" `
    -Argument "-NoProfile -ExecutionPolicy Bypass -File C:\scripts\Notifly-Disk-Check.ps1"
$Trigger = New-ScheduledTaskTrigger -Once -At (Get-Date) `
    -RepetitionInterval (New-TimeSpan -Minutes 15)
$Princ   = New-ScheduledTaskPrincipal -UserId "SYSTEM" -LogonType ServiceAccount -RunLevel Highest
Register-ScheduledTask -TaskName "Notifly Disk Check" `
    -Action $Action -Trigger $Trigger -Principal $Princ -Description "Notifly: проверка дисков"

Benefits

Section titled “Benefits”
  • No database outages at night due to full disks — you’ll learn about it days before a disaster.
  • The same channel for all servers — the title contains the hostname; in Notifly’s admin it’s easy to create one channel for a fleet of machines or one per server.
  • Three-level logic (ok → warn → crit) prevents spam: as long as nothing changes, there are no notifications.

What to improve next

Section titled “What to improve next”
  • Add a link to the dashboard/wiki in the message.
  • Attach the output of du -sh /var/log/* | sort -h | tail -10 to immediately see what ate the space — pass it via the extras field.
  • Configure separate Notifly channels for prod / staging — they’ll have different sounds.