Planned work without false alerts

Every release with a restart means a dozen false alerts: the HTTP monitor catches a 502, the heartbeat didn’t get its ping, Alertmanager sends InstanceDown, and escalation is already texting the person on duty. After a couple of nights like that people start ignoring alerts — and miss the real outage.

A maintenance window solves this out of the box: for the given interval Notifly keeps checking but doesn’t wake anyone.

What happens inside the window

• Monitors and heartbeats are checked and written to the history, but the failure alert is held back. If the problem is still there after the window, the very first check sends the alert. If something went down and came back up inside the window, you get nothing.
• Channel messages (including those from Alertmanager, Grafana and Zabbix) are created as usual but arrive silently and are not escalated to email, SMS, phone calls or messengers.
• The status page shows “Scheduled maintenance”, and check failures inside the window don’t hurt the 30/90-day uptime.

Option 1. Recurring work — from the admin panel

Monitoring → Maintenance windows → Create window:

Name:      Backup and VACUUM
Channels:  prod-db   (☑ including nested channels)
Type:      Weekly
Days:      Tue
Time:      02:00–04:00, Europe/Moscow

A window across midnight works too (23:00–01:00) — days are counted by the start day. Instead of a whole channel you can pick individual monitors and heartbeats.

Option 2. A window around a deploy — from CI

A one-time window needs an API token (prefix M) with write access and no channel restriction, plus the channel id (you can see it in the channel page address or in the GET /application response):

#!/usr/bin/env bash
# deploy-with-window.sh — a 30-minute window around the rollout
set -euo pipefail
API="https://api.notifly.ru"
AUTH="Authorization: Bearer $NOTIFLY_API_TOKEN"

start=$(date -u +%Y-%m-%dT%H:%M:%SZ)
end=$(date -u -d '+30 min' +%Y-%m-%dT%H:%M:%SZ)

id=$(curl -sf -X POST "$API/maintenance-window" -H "$AUTH" \
  -H "Content-Type: application/json" \
  -d "{\"name\":\"Deploy $(git rev-parse --short HEAD)\",
       \"channelIds\":[$NOTIFLY_CHANNEL_ID],\"includeSubtree\":true,\"monitors\":[],
       \"schedule\":\"once\",\"startsAt\":\"$start\",\"endsAt\":\"$end\",
       \"enabled\":true}" | jq -r .id)

sleep 30          # Notifly services pick up a new window within half a minute
./deploy.sh

# the rollout finished early — close the window, monitoring is back on duty
curl -sf -X DELETE "$API/maintenance-window/$id" -H "$AUTH"

The sleep 30 isn’t superfluous: monitors and message ingestion re-read windows about every half minute, so it’s better to create the window in advance rather than at the moment the work starts.

Window, quiet hours or pause?

Tool	Whose silence	When to use it
Maintenance window	a service / channel	planned work: deploy, upgrade, routine maintenance
Quiet hours	a person	nights and weekends: wake only for urgent, the rest as a digest
Pausing a monitor / heartbeat	a single check	a one-off repair of unknown duration

Benefits

• Alerts are trusted again: the channel isn’t flooded with hundreds of “down/up” messages on every release.
• Honest uptime: planned work doesn’t spoil the status page numbers, and customers see “Scheduled maintenance” rather than an outage.
• Nothing gets lost: if the service didn’t come back after the window, the alert arrives with the very first check.

What to improve next

• Also keep a window in Alertmanager itself (a silence for the duration of the deploy) — then the alerts won’t arrive even silently.
• Attach the window not to the whole channel but only to the monitors of the service being restarted — so that an outage of a neighbouring system during the release still wakes you up.