Skip to content
Notifly Notifly

RAID array degraded notification

RAID exists to survive a disk death without downtime — but only if you replace the failed disk in time. The catch is that a degraded array runs almost unnoticed: the service is alive, the application responds, yet you already have no redundancy, and the death of a second disk means data loss. You need to learn about a dropped disk within the hour, not a week later in a post-mortem.

What we’ll do

Section titled “What we’ll do”

The script checks storage health every 15 minutes and sends a notification when:

  • an array is degraded or rebuilding (/proc/mdstat + mdadm --detail) — high priority;
  • SMART reports poor disk health (smartctl -H) — high priority;
  • everything is clean again — a recovery message.

Just like in the disk space recipe, we store a three-level state (ok/warn/crit) in /var/lib/notifly-raid/ so we don’t receive the same message every 15 minutes.

Script

Section titled “Script”

The script needs the mdadm and smartmontools (smartctl) packages. Save as /usr/local/bin/notifly-raid-check:

#!/usr/bin/env bash
set -eu
set -a; source /etc/notifly.env; set +a


STATE_DIR=/var/lib/notifly-raid
mkdir -p "$STATE_DIR"
HOST=$(hostname -s)


# --- Check every software RAID array via /proc/mdstat ---
for MD in /sys/block/md*; do
    [ -e "$MD" ] || continue
    DEV=$(basename "$MD")
    NODE="/dev/$DEV"


    DETAIL=$(mdadm --detail "$NODE" 2>/dev/null || true)
    STATE=$(echo "$DETAIL" | awk -F: '/State :/ {sub(/^[ \t]+/, "", $2); print $2; exit}')
    LEVEL_RAID=$(echo "$DETAIL" | awk -F: '/Raid Level :/ {gsub(/[ \t]/, "", $2); print $2; exit}')


    STATE_FILE="$STATE_DIR/$DEV"
    LAST=$(cat "$STATE_FILE" 2>/dev/null || echo ok)


    LEVEL=ok
    # active/clean with no flags is normal; treat everything else as a problem
    case "$STATE" in
        *degraded*|*FAILED*|*failed*) LEVEL=crit ;;
        *recovering*|*resyncing*|*rebuild*|*resync*) LEVEL=warn ;;
    esac


    if [ "$LEVEL" != "$LAST" ]; then
        case "$LEVEL" in
            warn)
                /usr/local/bin/notifly-send \
                    "⚠️ RAID $DEV пересобирается — $HOST" \
                    "$LEVEL_RAID $DEV: состояние «$STATE». Идёт восстановление, дождитесь окончания и проверьте диск." 5
                ;;
            crit)
                /usr/local/bin/notifly-send \
                    "🔥 RAID $DEV ДЕГРАДИРОВАЛ — $HOST" \
                    "$LEVEL_RAID $DEV: состояние «$STATE». Массив без избыточности! Срочно замените выпавший диск." 10
                ;;
            ok)
                /usr/local/bin/notifly-send \
                    "✅ RAID $DEV снова в норме — $HOST" \
                    "$LEVEL_RAID $DEV: состояние «$STATE» (clean)." 4
                ;;
        esac
        echo "$LEVEL" > "$STATE_FILE"
    fi
done


# --- Check SMART health of the physical disks ---
for DISK in /dev/sd?; do
    [ -e "$DISK" ] || continue
    NAME=$(basename "$DISK")


    HEALTH=$(smartctl -H "$DISK" 2>/dev/null \
        | awk -F: '/(overall-health|SMART Health Status)/ {gsub(/[ \t]/, "", $2); print $2; exit}')
    [ -n "$HEALTH" ] || continue


    STATE_FILE="$STATE_DIR/smart-$NAME"
    LAST=$(cat "$STATE_FILE" 2>/dev/null || echo ok)


    # PASSED/OK is normal, everything else (FAILED!) is critical
    LEVEL=ok
    case "$HEALTH" in
        PASSED|OK) LEVEL=ok ;;
        *) LEVEL=crit ;;
    esac


    if [ "$LEVEL" != "$LAST" ]; then
        case "$LEVEL" in
            crit)
                /usr/local/bin/notifly-send \
                    "🔥 SMART: диск $NAME умирает — $HOST" \
                    "smartctl -H $DISK вернул «$HEALTH». Замените диск, пока массив ещё цел." 9
                ;;
            ok)
                /usr/local/bin/notifly-send \
                    "✅ SMART диска $NAME в норме — $HOST" \
                    "smartctl -H $DISK снова «$HEALTH»." 4
                ;;
        esac
        echo "$LEVEL" > "$STATE_FILE"
    fi
done

Make it executable:

Окно терминала
sudo chmod +x /usr/local/bin/notifly-raid-check

Run on schedule

Section titled “Run on schedule”

Using cron

Section titled “Using cron”
Окно терминала
sudo crontab -e

Add the line:

*/15 * * * * /usr/local/bin/notifly-raid-check >/dev/null 2>&1
Section titled “Using systemd timer (recommended)”

/etc/systemd/system/notifly-raid.service:

[Unit]
Description=Notifly RAID health check


[Service]
Type=oneshot
ExecStart=/usr/local/bin/notifly-raid-check

/etc/systemd/system/notifly-raid.timer:

[Unit]
Description=Run notifly-raid every 15 minutes


[Timer]
OnBootSec=5min
OnUnitActiveSec=15min


[Install]
WantedBy=timers.target

Activate it:

Окно терминала
sudo systemctl daemon-reload
sudo systemctl enable --now notifly-raid.timer
sudo systemctl list-timers notifly-raid.timer

Windows: PowerShell + Task Scheduler

Section titled “Windows: PowerShell + Task Scheduler”

On Windows the equivalent of /proc/mdstat is Get-PhysicalDisk (the HealthStatus of each disk) and Storage Spaces: virtual disks are checked via Get-VirtualDisk. The script raises an alarm if any disk or virtual volume is not Healthy. Uses the common function Send-Notifly from the sysadmin template/index.

C:\scripts\Notifly-Raid-Check.ps1
. C:\scripts\Notifly.ps1


$StateDir = "C:\ProgramData\Notifly\raid-state"
New-Item -ItemType Directory -Path $StateDir -Force | Out-Null
$Host = $env:COMPUTERNAME


function Get-Level([string]$health) {
    switch ($health) {
        "Healthy" { "ok" }
        "Warning" { "warn" }
        default   { "crit" }   # Unhealthy and anything unknown
    }
}


# --- Physical disk health ---
Get-PhysicalDisk | ForEach-Object {
    $name  = "phys-$($_.DeviceId)"
    $level = Get-Level $_.HealthStatus


    $stateFile = Join-Path $StateDir "$name.txt"
    $last = if (Test-Path $stateFile) { (Get-Content $stateFile -Raw).Trim() } else { "ok" }
    if ($level -eq $last) { return }


    switch ($level) {
        "warn" { Send-Notifly -Title "⚠️ Диск $($_.FriendlyName) — предупреждение — $Host" `
                              -Message "HealthStatus: $($_.HealthStatus). Проверьте SMART и запланируйте замену." -Priority 5 }
        "crit" { Send-Notifly -Title "🔥 Диск $($_.FriendlyName) УМИРАЕТ — $Host" `
                              -Message "HealthStatus: $($_.HealthStatus). Срочно замените диск, пока пул цел!" -Priority 9 }
        "ok"   { Send-Notifly -Title "✅ Диск $($_.FriendlyName) в норме — $Host" `
                              -Message "HealthStatus: $($_.HealthStatus)." -Priority 4 }
    }
    $level | Set-Content $stateFile
}


# --- Storage Spaces virtual disk health ---
Get-VirtualDisk -ErrorAction SilentlyContinue | ForEach-Object {
    $name  = "vdisk-$($_.FriendlyName)"
    $level = Get-Level $_.HealthStatus


    $stateFile = Join-Path $StateDir "$($name -replace '[\\/:]','_').txt"
    $last = if (Test-Path $stateFile) { (Get-Content $stateFile -Raw).Trim() } else { "ok" }
    if ($level -eq $last) { return }


    switch ($level) {
        "warn" { Send-Notifly -Title "⚠️ Том $($_.FriendlyName) деградирует — $Host" `
                              -Message "HealthStatus: $($_.HealthStatus), OperationalStatus: $($_.OperationalStatus)." -Priority 5 }
        "crit" { Send-Notifly -Title "🔥 Том $($_.FriendlyName) ДЕГРАДИРОВАЛ — $Host" `
                              -Message "HealthStatus: $($_.HealthStatus). Пул без избыточности — срочно замените диск!" -Priority 10 }
        "ok"   { Send-Notifly -Title "✅ Том $($_.FriendlyName) в норме — $Host" `
                              -Message "HealthStatus: $($_.HealthStatus) (Healthy)." -Priority 4 }
    }
    $level | Set-Content $stateFile
}

Register in Task Scheduler (as administrator, every 15 minutes as SYSTEM):

Окно терминала
$Action  = New-ScheduledTaskAction `
    -Execute "powershell.exe" `
    -Argument "-NoProfile -ExecutionPolicy Bypass -File C:\scripts\Notifly-Raid-Check.ps1"
$Trigger = New-ScheduledTaskTrigger -Once -At (Get-Date) `
    -RepetitionInterval (New-TimeSpan -Minutes 15)
$Princ   = New-ScheduledTaskPrincipal -UserId "SYSTEM" -LogonType ServiceAccount -RunLevel Highest
Register-ScheduledTask -TaskName "Notifly RAID Check" `
    -Action $Action -Trigger $Trigger -Principal $Princ -Description "Notifly: проверка RAID и SMART"

Benefits

Section titled “Benefits”
  • A dropped disk doesn’t go unnoticed. A degraded array runs “as usual,” and without a notification you only learn about the problem when the second disk dies — that is, when it’s already too late.
  • SMART warns you in advance. Many disks report poor health well before a full failure — that lead time is enough to calmly order a replacement.
  • Three-level logic (ok → warn → crit) prevents spam: as long as the state doesn’t change there are no notifications, while rebuild and recovery are shown separately.

What to improve next

Section titled “What to improve next”
  • Attach the full output of cat /proc/mdstat and mdadm --detail /dev/mdX via the extras field to see exactly which disk dropped, right in the app.
  • Add a heartbeat check: if the script itself stops running (for example, smartctl went missing), the silence becomes noticeable too.
  • Monitor the rebuild speed (speed/finish from /proc/mdstat) and send a separate message when it finishes — see also service monitoring.