Skip to content
Notifly Notifly

Auditing sudo and root commands

A compromised server doesn’t give itself away immediately. One of the most reliable markers is who runs which commands via sudo. If sudo is suddenly run by a user who isn’t on the admin list, there’s a direct root login, or authentication failures start piling up — you need to know that very second, not from a post-mortem.

This scenario is about privilege escalation and differs from SSH login notifications: there we catch the login itself, here — what a person does under sudo/root once already inside.

What we’ll do

Section titled “What we’ll do”

We run a long-lived watcher (a systemd service with Restart=always) that reads the log in real time and reacts to sudo: lines:

  • sudo from an unexpected user (not in the allowlist alice, bob) — 🚨 high priority (9);
  • sudo authentication failure (authentication failure, NOT in sudoers) — ⚠️ high priority (8);
  • a direct root login/session — 🚨 high priority (9);
  • an expected admin ran sudo — 🔑 quiet/low priority (3) for a digest (can be disabled entirely).

Into the message we put the invoking user, the TTY, and the command itself (COMMAND=).

Script

Section titled “Script”

On Debian/Ubuntu sudo entries go to /var/log/auth.log, on systemd/RHEL — to the journal (journalctl) and to /var/log/secure. The watcher is universal: if journalctl is available — we read it, otherwise tail -F on auth.log.

Save as /usr/local/bin/notifly-sudo-audit:

#!/usr/bin/env bash
set -eu
set -a; source /etc/notifly.env; set +a


HOST=$(hostname -s)


# Which users are allowed to escalate privileges.
# Anything outside this list is a reason for a loud alert.
ALLOWLIST="alice bob"


# Read the sudo log in real time. journalctl (systemd) or auth.log (Debian).
tail_source() {
    if command -v journalctl >/dev/null 2>&1; then
        # SYSLOG_IDENTIFIER=sudo covers both systemd and RHEL (/var/log/secure)
        journalctl SYSLOG_IDENTIFIER=sudo -f -o cat --since now
    elif [ -f /var/log/auth.log ]; then
        tail -F -n0 /var/log/auth.log
    else
        tail -F -n0 /var/log/secure
    fi
}


in_allowlist() {
    for u in $ALLOWLIST; do
        [ "$1" = "$u" ] && return 0
    done
    return 1
}


tail_source | while read -r line; do
    # We only care about sudo session lines
    echo "$line" | grep -q "sudo:" || continue


    # Authentication failure / not in sudoers — the most suspicious
    if echo "$line" | grep -qiE "authentication failure|NOT in sudoers|incorrect password"; then
        USER=$(echo "$line" | grep -oP 'sudo:\s*\K\S+' | head -1)
        TTY=$(echo "$line"  | grep -oP 'TTY=\K\S+'     | head -1)
        /usr/local/bin/notifly-send \
            "⚠️ Ошибка sudo: ${USER:-?}@$HOST" \
            "Пользователь: ${USER:-?}
TTY: ${TTY:-?}
Хост: $HOST
Строка: $line" 8 || true
        continue
    fi


    # Only process lines with an actually executed command
    echo "$line" | grep -q "COMMAND=" || continue


    USER=$(echo "$line"    | grep -oP 'sudo:\s*\K\S+'  | head -1)
    TTY=$(echo "$line"     | grep -oP 'TTY=\K\S+'      | head -1)
    RUNAS=$(echo "$line"   | grep -oP 'USER=\K\S+'     | head -1)
    COMMAND=$(echo "$line" | grep -oP 'COMMAND=\K.*'   | head -1)


    # Direct escalation to root — always loud
    if [ "${RUNAS:-root}" = "root" ] && ! in_allowlist "${USER:-?}"; then
        /usr/local/bin/notifly-send \
            "🚨 Неизвестный sudo→root: ${USER:-?}@$HOST" \
            "Пользователь: ${USER:-?} (нет в allowlist)
TTY: ${TTY:-?}
Команда: ${COMMAND:-?}
Хост: $HOST" 9 || true
    elif ! in_allowlist "${USER:-?}"; then
        # sudo from someone outside the admin list
        /usr/local/bin/notifly-send \
            "🚨 Чужой sudo: ${USER:-?}@$HOST" \
            "Пользователь: ${USER:-?} (нет в allowlist)
Как: ${RUNAS:-?}
TTY: ${TTY:-?}
Команда: ${COMMAND:-?}" 9 || true
    else
        # Expected admin — quiet digest (comment out if you like)
        /usr/local/bin/notifly-send \
            "🔑 sudo: ${USER:-?}@$HOST" \
            "Команда: ${COMMAND:-?}
TTY: ${TTY:-?}" 3 || true
    fi
done

Make it executable:

Окно терминала
sudo chmod +x /usr/local/bin/notifly-sudo-audit

Run as a systemd service

Section titled “Run as a systemd service”

So the watcher lives permanently and comes back after reboots and crashes, we wrap it in a service with Restart=always.

/etc/systemd/system/notifly-sudo-audit.service:

[Unit]
Description=Notify about suspicious sudo/root activity
After=network-online.target


[Service]
ExecStart=/usr/local/bin/notifly-sudo-audit
Restart=always
RestartSec=10s


[Install]
WantedBy=multi-user.target

Activate it:

Окно терминала
sudo systemctl daemon-reload
sudo systemctl enable --now notifly-sudo-audit

Test:

Окно терминала
sudo -k
sudo id            # expected admin — a quiet 🔑 arrives
sudo -u nobody id  # foreign user — a 🚨 flies in

Direct root login

Section titled “Direct root login”

A direct root login (for example, via the console or su -) is visible through PAM sessions. Add a second subscription to the watcher, or a separate pam_exec in the SSH scenario, where USER=root is already flagged with priority 9. The ideal policy is PermitRootLogin no in sshd plus an alert on any session opened for user root line in the log.

Windows: PowerShell + Event Log

Section titled “Windows: PowerShell + Event Log”

An equivalent for Windows servers: we watch the Security Event Log via a subscription in Task Scheduler. The relevant codes:

  • 4672 — special privileges assigned to a new logon (i.e. an admin logged in);
  • 4720 — a user account was created.

Both events are classic markers of privilege escalation and of a “quiet” backdoor-account creation. We use the shared function Send-Notifly.

Notifier script

Section titled “Notifier script”
C:\scripts\Notifly-Privilege.ps1
param([int]$EventId = 4672)
. C:\scripts\Notifly.ps1


# Who is expected to have admin rights. Anything outside the list is a reason to alert.
$AllowList = @("alice", "bob", "Administrator")


# Take the latest event from the Security Log
$ev = Get-WinEvent -FilterHashtable @{LogName='Security'; Id=$EventId} -MaxEvents 1
if (-not $ev) { return }


# Parse EventData
$xml  = [xml]$ev.ToXml()
$data = @{}
$xml.Event.EventData.Data | ForEach-Object { $data[$_.Name] = $_.'#text' }


if ($EventId -eq 4672) {
    $user = $data.SubjectUserName
    # Ignore system accounts and computer accounts
    if ($user -in @("SYSTEM", "LOCAL SERVICE", "NETWORK SERVICE")) { return }
    if ($user -like "*$") { return }


    $prio = if ($user -in $AllowList) { 3 } else { 9 }
    $icon = if ($user -in $AllowList) { "🔑 Админ-вход" } else { "🚨 Чужой админ-вход" }


    Send-Notifly `
        -Title "$icon: $user@$env:COMPUTERNAME" `
        -Message "Пользователь: $user`nДомен: $($data.SubjectDomainName)`nСервер: $env:COMPUTERNAME`nВремя: $($ev.TimeCreated.ToString('yyyy-MM-dd HH:mm:ss'))" `
        -Priority $prio
}
else {
    # 4720 — a new user was created (always loud)
    $newUser = $data.TargetUserName
    $by      = $data.SubjectUserName
    Send-Notifly `
        -Title "🚨 Создан пользователь: $newUser на $env:COMPUTERNAME" `
        -Message "Новый аккаунт: $newUser`nКем создан: $by`nСервер: $env:COMPUTERNAME`nВремя: $($ev.TimeCreated.ToString('yyyy-MM-dd HH:mm:ss'))" `
        -Priority 9
}

Subscribing to events

Section titled “Subscribing to events”
Окно терминала
# 4672 — special privileges assigned (admin logon)
$xml = @"
<QueryList>
  <Query Id="0" Path="Security">
    <Select Path="Security">
      *[System[EventID=4672]]
    </Select>
  </Query>
</QueryList>
"@
$Trigger = New-ScheduledTaskTrigger -AtStartup
$Trigger.Subscription = $xml
$Action = New-ScheduledTaskAction -Execute "powershell.exe" `
    -Argument "-NoProfile -ExecutionPolicy Bypass -File C:\scripts\Notifly-Privilege.ps1 -EventId 4672"
Register-ScheduledTask -TaskName "Notifly Privilege Watch" -Trigger $Trigger -Action $Action `
    -Principal (New-ScheduledTaskPrincipal -UserId "SYSTEM" -LogonType ServiceAccount -RunLevel Highest) `
    -Description "Notifly: аудит админ-входов"


# 4720 — a user account was created
$xml2 = $xml -replace '4672', '4720'
$T2 = New-ScheduledTaskTrigger -AtStartup
$T2.Subscription = $xml2
$A2 = New-ScheduledTaskAction -Execute "powershell.exe" `
    -Argument "-NoProfile -ExecutionPolicy Bypass -File C:\scripts\Notifly-Privilege.ps1 -EventId 4720"
Register-ScheduledTask -TaskName "Notifly New Account Watch" -Trigger $T2 -Action $A2 `
    -Principal (New-ScheduledTaskPrincipal -UserId "SYSTEM" -LogonType ServiceAccount -RunLevel Highest) `
    -Description "Notifly: аудит новых учёток"

Benefits

Section titled “Benefits”
  • Any privilege escalation is visible. A foreign sudo or a direct root is exactly the moment an attack turns into a takeover of the machine.
  • Authentication failures are an early signal. NOT in sudoers often means someone is already sitting under a stolen account and trying to widen their rights.
  • The allowlist quells the noise. As long as only alice and bob run commands, you get quiet 🔑 (or nothing), and loud 🚨 sound only on anomalies.

What to improve next

Section titled “What to improve next”
  • Send the full context (last, who, environment) via the extras field to triage the incident straight from your phone.
  • Priorities via priority: an expected admin — 3, a foreign sudo and failures — 8–9, creating a root/wheel user — 10.
  • Link to the Fail2ban notification — external bruteforce and suspicious sudo inside often go hand in hand.