Prometheus and Grafana alerts on your phone

Prometheus already knows what’s on fire — what’s missing is delivery: email gets lost, and running a separate bot gateway just to get a push on your phone is no fun. Notifly has a native receiver for the Alertmanager format: one receiver in alertmanager.yml — and alerts arrive as push notifications with the right priority, Runbook and Silence links and a separate [RESOLVED].

The same receiver understands the webhook contact point of Grafana Alerting.

Step 1. Channel and token

Create a channel in the admin panel, e.g. prometheus, and copy its channel token (prefix A). In the channel settings, on the Sending tab, the Receive from monitoring systems block already has ready-made configuration snippets with this token.

Step 2. A receiver in Alertmanager

route:
  receiver: notifly
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: notifly
    webhook_configs:
      - url: https://api.notifly.ru/integrations/alertmanager
        send_resolved: true
        max_alerts: 50
        http_config:
          authorization:
            type: Bearer
            credentials: <channel token>

Reload the configuration (SIGHUP or POST /-/reload). Keep the token in the header rather than in the URL: webhook addresses end up in Alertmanager’s logs.

One Alertmanager notification = one Notifly message: the whole group_by group arrives as a single [FIRING:3] HighCPUUsage push, not a hundred separate ones. Recovery comes as a separate quiet [RESOLVED] … message.

Step 3. Priorities from severity

The priority comes from the severity label: critical/page → 9, error/high → 8, warning → 6, info → 3. If your scale is different, set the notifly_priority label in the rule:

groups:
  - name: node
    rules:
      - alert: DiskAlmostFull
        expr: node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.05
        for: 10m
        labels:
          severity: warning
          notifly_priority: '8' # louder than a regular warning
        annotations:
          summary: 'Less than 5% disk left on {{ $labels.instance }}'
          runbook_url: https://wiki.example.com/runbooks/disk

The full mapping table is in the receiver documentation.

Grafana Alerting

Alerting → Contact points → Add contact point, Webhook integration:

• URL — https://api.notifly.ru/integrations/alertmanager;
• Optional Webhook settings → Authorization Header: scheme Bearer, credentials — the channel token.

Tapping the notification opens the Grafana panel the alert came from.

Testing without Prometheus

curl -X POST "https://api.notifly.ru/integrations/alertmanager" \
     -H "Authorization: Bearer $NOTIFLY_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"version":"4","status":"firing","alerts":[{"status":"firing",
          "labels":{"alertname":"TestAlert","severity":"warning","instance":"host-1"},
          "annotations":{"summary":"Receiver test"}}]}'
# {"created":1,"ids":[...]}

So that only critical wakes you at night

• Escalation on the channel: if a critical push isn’t read in 5 minutes — SMS or a phone call, in 15 minutes — PagerDuty or Opsgenie. Marking it “Read” (in the app or straight from the Android notification) stops the re-sending.
• Quiet hours: at night only messages at or above the threshold (8 by default) are escalated — critical breaks through the silence, warning waits for the morning digest.
• Maintenance windows: alerts during planned work arrive silently and aren’t escalated — see “Planned work without false alerts”.

Benefits

• No gateway of your own: no alertmanager-telegram bots or proxies — just a receiver in the config.
• Noise under control: grouping is defined by Alertmanager itself, and Notifly doesn’t split a group into separate pushes.
• Context in your pocket: summary, labels and Source / Runbook / Silence links right in the notification.

What to improve next

• Route alerts to different channels (prod, staging, db) — a separate receiver with its own channel token for each branch of the route.
• Add a heartbeat on the monitoring itself: send the always-firing Watchdog rule (expr: vector(1)) through a separate route with a repeat_interval shorter than the heartbeat interval to a receiver with the URL https://api.notifly.ru/heartbeat/ping/H…. If Prometheus or Alertmanager goes silent, the pings stop and Notifly sends an alert.