Notifications about unhealthy Docker containers

A container with a healthcheck can quietly go unhealthy and keep “running”, accepting traffic but returning errors. And a container with restart: always can spin in a restart loop for hours — Docker dutifully restarts it, while users see timeouts. Both cases are easy to miss: docker ps shows the container as running. Better to get a push than to read complaints.

What we’ll do

The script goes through all running containers every 2 minutes and:

if a container is in the unhealthy state — sends a high-priority message;
if the restart count grew since the last check (the container is looping) — also sends a high-priority message;
when the container is healthy again and restarts have stopped — sends a “green” recovery message.

To avoid receiving the same message every 2 minutes, we store the last health and restart count per container in /var/lib/notifly-docker/ — in spirit these are the same state files as in the disk recipe.

Script

Save as /usr/local/bin/notifly-docker-check:

#!/usr/bin/env bash
set -eu
set -a; source /etc/notifly.env; set +a

STATE_DIR=/var/lib/notifly-docker
mkdir -p "$STATE_DIR"

HOST=$(hostname -s)

# Names of all running containers
docker ps --format '{{.Names}}' | while read -r NAME; do
    [ -n "$NAME" ] || continue

    # Health: healthy / unhealthy / starting / none (if no healthcheck is defined)
    HEALTH=$(docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$NAME" 2>/dev/null || echo none)
    # How many times Docker has restarted the container
    RESTARTS=$(docker inspect --format '{{.RestartCount}}' "$NAME" 2>/dev/null || echo 0)

    KEY=$(echo "$NAME" | tr '/' '_')
    HEALTH_FILE="$STATE_DIR/$KEY.health"
    RESTART_FILE="$STATE_DIR/$KEY.restarts"

    LAST_HEALTH=$(cat "$HEALTH_FILE" 2>/dev/null || echo none)
    LAST_RESTARTS=$(cat "$RESTART_FILE" 2>/dev/null || echo "$RESTARTS")

    # 1. Container just became unhealthy (was something else)
    if [ "$HEALTH" = "unhealthy" ] && [ "$LAST_HEALTH" != "unhealthy" ]; then
        /usr/local/bin/notifly-send \
            "⚠️ Container $NAME: unhealthy — $HOST" \
            "Healthcheck of container $NAME failed. Total restarts: $RESTARTS." 8
    fi

    # 2. Restart count grew — the container is looping
    if [ "$RESTARTS" -gt "$LAST_RESTARTS" ]; then
        DELTA=$((RESTARTS - LAST_RESTARTS))
        /usr/local/bin/notifly-send \
            "🔥 Container $NAME keeps restarting — $HOST" \
            "New restarts: $DELTA (total $RESTARTS). Looks like a restart loop, check docker logs $NAME." 9
    fi

    # 3. Recovery: healthy again and restarts not growing
    if [ "$HEALTH" = "healthy" ] && [ "$LAST_HEALTH" = "unhealthy" ]; then
        /usr/local/bin/notifly-send \
            "✅ Container $NAME is healthy again — $HOST" \
            "Healthcheck of container $NAME is passing again." 4
    fi

    echo "$HEALTH"   > "$HEALTH_FILE"
    echo "$RESTARTS" > "$RESTART_FILE"
done

Make it executable:

sudo chmod +x /usr/local/bin/notifly-docker-check

You can inspect container state by eye with the same format the script uses:

docker ps --format '{{.Names}} {{.Status}}'

In the Status column, containers with a healthcheck show (healthy) or (unhealthy) in parentheses.

Run on schedule

Using cron

sudo crontab -e

Add the line:

*/2 * * * * /usr/local/bin/notifly-docker-check >/dev/null 2>&1

Using systemd timer (recommended)

/etc/systemd/system/notifly-docker.service:

[Unit]
Description=Notifly docker health check

[Service]
Type=oneshot
ExecStart=/usr/local/bin/notifly-docker-check

/etc/systemd/system/notifly-docker.timer:

[Unit]
Description=Run notifly-docker every 2 minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=2min

[Install]
WantedBy=timers.target

Activate it:

sudo systemctl daemon-reload
sudo systemctl enable --now notifly-docker.timer
sudo systemctl list-timers notifly-docker.timer

Windows: PowerShell + Task Scheduler

Docker Desktop on Windows keeps containers inside WSL2, but the CLI works exactly the same way — we poll docker ps and docker inspect from PowerShell. The script uses the common function Send-Notifly from the sysadmin template/index.

. C:\scripts\Notifly.ps1

$StateDir = "C:\ProgramData\Notifly\docker-state"
New-Item -ItemType Directory -Path $StateDir -Force | Out-Null
$Host = $env:COMPUTERNAME

# Names of all running containers
docker ps --format '{{.Names}}' | Where-Object { $_ } | ForEach-Object {
    $name = $_

    # Health and restart count via docker inspect
    $health   = docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' $name 2>$null
    if (-not $health) { $health = "none" }
    $restarts = [int](docker inspect --format '{{.RestartCount}}' $name 2>$null)

    $key         = $name -replace '/', '_'
    $healthFile  = Join-Path $StateDir "$key.health"
    $restartFile = Join-Path $StateDir "$key.restarts"

    $lastHealth   = if (Test-Path $healthFile)  { (Get-Content $healthFile -Raw).Trim() }  else { "none" }
    $lastRestarts = if (Test-Path $restartFile) { [int](Get-Content $restartFile -Raw) } else { $restarts }

    # 1. Container just became unhealthy
    if ($health -eq "unhealthy" -and $lastHealth -ne "unhealthy") {
        Send-Notifly -Title "⚠️ Container $name`: unhealthy — $Host" `
                     -Message "Healthcheck failed. Total restarts: $restarts." -Priority 8
    }

    # 2. Restart count grew — restart loop
    if ($restarts -gt $lastRestarts) {
        $delta = $restarts - $lastRestarts
        Send-Notifly -Title "🔥 Container $name keeps restarting — $Host" `
                     -Message "New restarts: $delta (total $restarts). Check docker logs $name." -Priority 9
    }

    # 3. Recovery
    if ($health -eq "healthy" -and $lastHealth -eq "unhealthy") {
        Send-Notifly -Title "✅ Container $name is healthy again — $Host" `
                     -Message "Healthcheck is passing again." -Priority 4
    }

    $health   | Set-Content $healthFile
    $restarts | Set-Content $restartFile
}

$Action  = New-ScheduledTaskAction `
    -Execute "powershell.exe" `
    -Argument "-NoProfile -ExecutionPolicy Bypass -File C:\scripts\Notifly-Docker-Check.ps1"
$Trigger = New-ScheduledTaskTrigger -Once -At (Get-Date) `
    -RepetitionInterval (New-TimeSpan -Minutes 2)
$Princ   = New-ScheduledTaskPrincipal -UserId "SYSTEM" -LogonType ServiceAccount -RunLevel Highest
Register-ScheduledTask -TaskName "Notifly Docker Check" `
    -Action $Action -Trigger $Trigger -Principal $Princ -Description "Notifly: container check"

Benefits

Catch silent failures: an unhealthy container still shows up in docker ps as “Up”, but actually stops serving requests — without healthcheck monitoring this is easy to overlook.
Restart loops are visible immediately: a growing RestartCount catches containers that crash and restart in a loop long before users notice.
No spam: state is stored per container, so as long as unhealthy doesn’t change there are no new notifications, and recovery arrives exactly once.

What to improve next

Enrich the message with a log tail: docker logs --tail 20 $NAME — pass it via the extras field to see the cause right away.
Separately notify about containers that disappeared from docker ps entirely (dropped to exited) — similar to the systemd services recipe.
Split prod / staging across separate Notifly channels so they get different sounds.