Prometheus Alertmanager Phone Call Alerts: Wake Up for Critical Only

Alertmanager has no voice receiver. Route Prometheus alerts to a phone call for critical severity only: webhook config, conditions, and the Watchdog trap.

Updated

Table of Contents

Alertmanager has no voice receiver. To get a phone call when a Prometheus alert fires, add a webhook_configs receiver pointing at an Echobell channel whose subscription type is Calling. This guide covers the exact YAML, the condition that stops resolved alerts from calling you, severity routing, and the Watchdog alert that will otherwise ring your phone every four hours forever.

Prometheus is the default metrics stack for most infrastructure built in the last decade, and Alertmanager is genuinely good at the hard parts: deduplicating alerts, grouping them, silencing them during maintenance, and inhibiting the downstream noise when an upstream dependency fails.

What it will not do is wake anybody up.

Why Alertmanager cannot ring your phone on its own

Alertmanager ships receivers for email, Slack, PagerDuty, OpsGenie, Discord, Telegram, Pushover, Webex, MS Teams, and a dozen more. Every one of them delivers a message, and messages are subject to the ringer switch, Do Not Disturb, and iOS Focus Modes. At 03:00 that means the alert arrives and nothing happens.

There is no voice_configs. The options people usually land on are:

  • PagerDuty / OpsGenie / Splunk On-Call — these do place calls, and they are full incident-management platforms with per-seat pricing to match. The right answer if you need rotations and escalation trees; heavy if you need a phone to ring. (OpsGenie in particular is winding down, which is why so many teams are re-evaluating this layer right now.)
  • SMS bridges like Sachet — you run another service, you pay a gateway per message, and an SMS still lands as a message. On iOS a text does not break Focus Mode unless the sender is on your allow list.
  • Twilio-based glue — write a small webhook receiver, buy a number, pay per call, and now you own a piece of production infrastructure whose only job is to make a phone ring.

The generic webhook receiver is the escape hatch. It posts a documented JSON payload to any URL, which is all you need.

What you need

  • A running Prometheus + Alertmanager setup, and access to edit alertmanager.yml
  • Echobell installed (App Store / Google Play)
  • Ten minutes

This guide was written against Alertmanager 0.31. The webhook payload has been at version: "4" for years, so 0.2x releases behave identically.

Your Alertmanager needs outbound HTTPS to hook.echobell.one. It does not need to be reachable from the internet, so an Alertmanager inside a cluster, a VPC, or a homelab works fine.

Step 1 — Create a channel that calls you

In Echobell, create a channel named something like Prometheus Critical. Set its subscription notification type to Calling. This is the setting that matters: Calling alerts arrive as an incoming call screen and ring through iOS Focus Mode and Do Not Disturb, which a push notification does not.

Set the templates to read the Alertmanager payload directly:

Title: 🔴 {{commonLabels.alertname}} on {{commonLabels.instance}}
Body: {{commonAnnotations.summary}}
{{commonAnnotations.description}}

And, under Advanced Settings, a Link Template so the notification record jumps straight to the graph:

{{alerts[0].generatorURL}}

Then copy the channel's Webhook URL:

https://hook.echobell.one/t/<channel-token>

Treat that URL as a secret — anyone holding it can make your phone ring.

Step 2 — Add a webhook receiver

In alertmanager.yml:

route:
  group_by: ["alertname", "cluster", "service"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: echobell-critical

receivers:
  - name: echobell-critical
    webhook_configs:
      - url: "https://hook.echobell.one/t/<channel-token>"
        send_resolved: false

Reload with curl -X POST http://localhost:9093/-/reload or SIGHUP.

Note send_resolved: false. The default for a webhook receiver is true, unlike most other Alertmanager receivers, so leaving it out means your phone rings when the service breaks and rings again when it fixes itself. The second call is the one that teaches people to ignore the first. Step 4 shows how to get the recovery notice back without the ring.

Step 3 — Understand what actually arrives

Alertmanager groups alerts, then POSTs one payload per group:

{
  "version": "4",
  "groupKey": "{}:{alertname=\"HighErrorRate\"}",
  "truncatedAlerts": 0,
  "status": "firing",
  "receiver": "echobell-critical",
  "groupLabels": { "alertname": "HighErrorRate" },
  "commonLabels": { "alertname": "HighErrorRate", "severity": "critical" },
  "commonAnnotations": { "summary": "Error rate above 5% for 10m" },
  "externalURL": "http://alertmanager.internal:9093",
  "alerts": [
    {
      "status": "firing",
      "labels": { "alertname": "HighErrorRate", "instance": "api-7d9f:8080" },
      "annotations": { "summary": "Error rate above 5% for 10m" },
      "startsAt": "2026-09-04T02:41:07.351Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://prometheus:9090/graph?g0.expr=...",
      "fingerprint": "a1b2c3d4e5f60718"
    }
  ]
}

Echobell reads the JSON body as-is, so every one of those fields is available in templates and conditions. Nested access works with either syntax — {{commonLabels.severity}} or {{alerts[0].labels["instance"]}}.

Two properties of this payload drive everything below:

The top-level status is firing if any alert in the group is firing. It only becomes resolved once every alert in the group has resolved. That makes it a clean thing to filter on.

commonLabels holds only the labels shared by every alert in the group. This is the single most common surprise. If group_by is broad enough that one webhook carries HighErrorRate on three different instances, then commonLabels.instance is absent and {{commonLabels.instance}} renders as an empty string. There is more on how to handle that below.

Step 4 — Send recoveries as a quiet push

You still want to know when something recovers — you just do not want to be called about it. Add a second Echobell channel named Prometheus Recovered, set its notification type to Normal, give it these templates:

Title: ✅ {{commonLabels.alertname}} resolved
Body: {{commonAnnotations.summary}}

and, under Advanced Settings, this condition:

status == "resolved"

Conditions are expressions evaluated before anything is delivered. If the expression is false, Echobell accepts the request and sends nothing.

Then point the same receiver at both channels — one receiver can hold several webhook_configs:

receivers:
  - name: echobell-critical
    webhook_configs:
      # Rings the phone. Firing only.
      - url: "https://hook.echobell.one/t/<calling-channel-token>"
        send_resolved: false
      # Quiet push. The channel condition drops the firing half.
      - url: "https://hook.echobell.one/t/<recovery-channel-token>"
        send_resolved: true

The recovery channel receives both firing and resolved payloads and discards the firing ones. The result: downtime rings, recovery arrives as a push you read in the morning.

Step 5 — Route by severity, not by everything

A catch-all route that sends every alert to a calling channel is a machine for producing ignored phone calls. Split by severity in Alertmanager, where the routing tree belongs:

route:
  group_by: ["alertname", "cluster", "service"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: echobell-warning

  routes:
    # Watchdog never reaches a human. See Step 6.
    - matchers:
        - alertname = "Watchdog"
      receiver: "null"

    - matchers:
        - severity = "critical"
      receiver: echobell-critical
      group_wait: 10s
      repeat_interval: 1h

receivers:
  - name: "null"

  - name: echobell-critical
    webhook_configs:
      - url: "https://hook.echobell.one/t/<calling-channel-token>"
        send_resolved: false

  - name: echobell-warning
    webhook_configs:
      - url: "https://hook.echobell.one/t/<normal-channel-token>"
        send_resolved: true

Routes are evaluated top to bottom and the first match wins — continue defaults to false. So order matters: the Watchdog route has to sit above anything that would otherwise swallow it.

If you would rather keep one channel and filter on the Echobell side, the equivalent condition is:

status == "firing" && commonLabels.severity == "critical"

Doing it in Alertmanager is usually better, because severity then also drives group_wait and repeat_interval. Doing it in Echobell is better when you cannot get a config change merged today.

Step 6 — The Watchdog trap

If you run kube-prometheus-stack, you have an alert called Watchdog whose expression is vector(1). It is designed to fire forever — it exists so that an external system can notice when Prometheus itself has stopped. The default config routes it to a null receiver.

Point a catch-all route at a calling channel without excluding it and Watchdog will call your phone every repeat_interval, forever, starting immediately. This is the number-one way people conclude that phone alerts "do not work."

Keep the null route from Step 5. Then, optionally, do the useful thing with it: turn Watchdog into a real dead man's switch.

    - matchers:
        - alertname = "Watchdog"
      receiver: deadmansswitch
      group_wait: 0s
      group_interval: 1m
      repeat_interval: 50s

receivers:
  - name: deadmansswitch
    webhook_configs:
      - url: "https://hc-ping.com/<your-check-uuid>"
        send_resolved: false

Echobell cannot be the dead man's switch itself — it alerts when a request arrives, not when one stops arriving. So send the Watchdog ping to a service built for silence detection (Healthchecks.io, Cronitor, Dead Man's Snitch), and then point that service's "check went down" webhook at your Echobell calling channel. Now a phone call means "monitoring itself is dead," which is the one alert you most want to be woken by and the one nobody configures.

Tuning so it does not cry wolf

Three Alertmanager settings do most of the work, and one Prometheus one:

SettingWhereWhat it does
for:Alert ruleHow long the condition must hold before it fires at all. Your first line of defence against a two-second blip.
group_waitRouteHow long to wait for more alerts before the first notification. 30s default; drop to 10s for critical.
group_intervalRouteMinimum gap before a notification about new alerts in an existing group. Default 5m.
repeat_intervalRouteHow often an unresolved alert re-notifies. Default 4h — so a night-time outage calls you at 03:00 and again at 07:00.

repeat_interval is the one worth thinking about. Four hours is a long time to leave something broken; twenty minutes is a machine for making you disable the channel. One hour on critical is a reasonable starting point.

If you want an unanswered call to retry immediately rather than waiting for the next repeat_interval, turn on Retry Failed Call in Echobell's app settings.

Only ring outside working hours

During the workday you are probably already looking at a dashboard. Echobell's system time variables (all UTC) let a channel behave differently by hour, without a second Alertmanager route:

status == "firing" && (hour >= 17 || hour < 9)

That calls you only outside 09:00–17:00 UTC. Point a second, Normal-type channel at the inverse for daytime pushes:

status == "firing" && hour >= 9 && hour < 17

Add dayOfWeek >= 1 && dayOfWeek <= 5 to treat weekends as out-of-hours too. Remember these are always computed in UTC — offset for your own timezone. There is a fuller treatment in time window notifications using UTC conditions.

Handling the empty commonLabels problem

When a group contains alerts from several instances, commonLabels.instance disappears and your notification title reads 🔴 HighErrorRate on .

Three ways out, in order of preference:

  1. Put the label in group_by. If group_by includes instance, then every alert in a group shares it and commonLabels.instance is always present. The cost is more notifications — one per instance instead of one per alert name.
  2. Read the first alert instead. {{alerts[0].labels.instance}} is always populated. It is only one of possibly many, so pair it with a count: {{alerts[0].labels.instance}} (+{{alerts.length}} alerts).
  3. Design the label so empty still reads. Echobell has no default-value operator — {{a || "unknown"}} renders the literal text true, not a fallback — so write Instance: {{commonLabels.instance}} on its own line, where a blank value is obviously blank rather than a broken sentence.

Keeping the payload small

A group covering a hundred pods produces a large JSON body, and Echobell rejects trigger bodies over 1 MiB with HTTP 413. Cap it in Alertmanager:

      - url: "https://hook.echobell.one/t/<channel-token>"
        send_resolved: false
        max_alerts: 20

Alertmanager then sends at most twenty alerts and sets truncatedAlerts to the number it dropped, which you can surface in the body:

Body: {{commonAnnotations.summary}}
Alerts: {{alerts.length}} (+{{truncatedAlerts}} truncated)

Sharing the alert with your team

An Echobell channel can be shared via a subscription link, and every subscriber picks their own notification type. The same route can therefore ring the on-call engineer while landing as a normal push for everyone else — no per-seat pricing, and no extra routing rules in Alertmanager.

It also fits the reason many teams self-host Prometheus in the first place: your metrics and alert rules stay on your infrastructure, and Echobell keeps notification content and history on the device rather than on its servers.

What this setup does not give you

Being honest about the boundary saves you a bad migration later. Echobell is a delivery layer, not an incident-management platform. It has no:

  • On-call rotation schedules or follow-the-sun handoffs
  • Escalation trees that page a second person when the first does not answer
  • Incident timelines, acknowledgement tracking, or postmortem tooling

If your team needs those, you need PagerDuty, Grafana Cloud IRM, or similar. What this covers is the specific gap Alertmanager leaves open: converting a firing alert into a phone that actually rings. For solo operators, small teams, and homelabs, that is usually the entire requirement.

Troubleshooting

Nothing arrives at all. Check Alertmanager's own logs first (level=error component=dispatcher), then confirm the route actually resolves to your receiver — amtool config routes test severity=critical alertname=HighErrorRate tells you which receiver an alert would land in without waiting for one to fire.

Echobell returns HTTP 404. The channel token is wrong or the channel was deleted. An unknown token is a 404, not a silent success.

Echobell returns 200 with "notificationTriggered": false. Your condition evaluated to false. The response body also carries "conditionsMet": false, which is the fastest way to tell "my condition is wrong" apart from "my webhook never arrived." Check status == "firing" against what Alertmanager actually sent — the top-level status, not alerts[0].status.

HTTP 413. The payload exceeded 1 MiB. Set max_alerts as above.

HTTP 405. The channel has POST Only enabled and something sent a GET. Alertmanager posts, so this usually means you tested the URL in a browser.

The title has an empty gap in it. commonLabels did not contain that label for this group. See the section above.

Nothing rings, but the notification arrives. The subscription's notification type is Normal or Time Sensitive, not Calling. Notification type is chosen per subscriber, so check it on the device that is not ringing.

Testing without breaking production. Add a rule with expr: vector(1), a distinct alertname, and severity: critical, let it fire once, then delete it. Or fire one by hand:

curl -X POST http://localhost:9093/api/v2/alerts -H 'Content-Type: application/json' -d '[
  {"labels":{"alertname":"EchobellTest","severity":"critical"},
   "annotations":{"summary":"Testing the phone call path"}}
]'

Frequently asked questions

Can Prometheus Alertmanager make a phone call natively?

No. Alertmanager has receivers for email, Slack, PagerDuty, OpsGenie and many others, but there is no voice or SMS receiver. Phone calls require routing the generic webhook receiver to a service that can place one, such as Echobell, or paying for an incident-management platform.

Will the phone call bypass Do Not Disturb?

Yes. Echobell's Calling notification type presents as an incoming call, which rings through iOS Focus Mode and Do Not Disturb. See bypassing iOS Focus Mode for critical alerts for the details and the settings involved.

Does this work with Alertmanager behind a firewall or inside Kubernetes?

Yes. The webhook is an outbound HTTPS request from Alertmanager, so it only needs to reach hook.echobell.one. Your Alertmanager does not need a public address or an ingress.

How do I stop getting called when an alert resolves?

Set send_resolved: false on the webhook config that points at your calling channel. The webhook receiver defaults to true, unlike most other Alertmanager receivers, so this is opt-out rather than opt-in. To still receive recoveries quietly, add a second channel with the condition status == "resolved".

Why does my phone ring every four hours for the same alert?

That is repeat_interval, which defaults to 4h. Alertmanager re-notifies about a still-firing alert on that cadence. Set it per route — 1h on critical is a common choice. If the calls started immediately after you added a catch-all route, the culprit is more likely the always-firing Watchdog alert; see Step 6.

Can several people be called for the same alert?

Yes. Share the channel with your teammates and each subscriber chooses their own notification type. Everyone subscribed to the calling channel gets rung, at no per-seat cost.

Should I filter severity in Alertmanager or in Echobell conditions?

Prefer Alertmanager: routing there also lets you set group_wait and repeat_interval per severity, and the routing tree stays in version control with the rest of your config. Use Echobell conditions when you cannot change the Alertmanager config, or for filters Alertmanager has no concept of — such as time-of-day.

Wrap-up

The setup is one receiver, one send_resolved: false, and a routing tree that keeps everything except severity: critical away from the calling channel. It leaves your alert rules, grouping, silences, and inhibition exactly as they are, and it closes the gap between "Prometheus noticed" and "a human noticed."

Download Echobell for iPhone or get it on Google Play, then fire the EchobellTest alert above before you rely on the path for anything real.

Related articles

Uptime Kuma Phone Call Alerts: Make Your Phone Ring

Uptime Kuma ships 90+ notification providers but none that rings your phone. Here is how to add phone call alerts for downtime using one webhook.

Read more

Your AI Agent Is Waiting for You: Turn Approval Gates Into Phone Calls

Autonomous agents pause and wait silently when they need a human. Nothing in the agent stack rings your phone. Here's how to wire agent approval gates and failed runs into a real call with Echobell.

Read more

Get TradingView Alerts as Phone Calls with Echobell

Connect TradingView to Echobell via webhook to receive price alerts as phone calls or time-sensitive notifications, with custom voice content.

Read more