Backup Failure Alerts: Know Before You Need to Restore

Backups fail silently for weeks. Catch restic and borg errors, stale snapshots and missed runs, and get a phone call when a backup breaks. Copy-paste scripts.

Updated

Table of Contents

Most people find out their backups were broken on the day they need them. The job exited with an error three weeks ago, nobody read the log, and the newest snapshot is older than the mistake you are trying to undo.

The fix is not a better backup tool. It is making sure a backup problem reaches a human the same hour it happens. This guide shows how to do that for restic and borg (and anything else that runs from a shell), covering the three ways backups actually fail, with scripts you can copy. The final step sends the alert to your phone through Echobell, so a broken backup rings instead of sitting in a mailbox.

The three ways a backup fails

Monitoring only the first one is the most common mistake.

  1. It runs and fails. Repository locked, disk full, wrong password, network drop. The tool exits non-zero.
  2. It runs and half-succeeds. restic exits with code 3 when a snapshot was created but some files could not be read. borg exits with 1 for warnings. Both are easy to wave away, and both can hide a permissions problem that silently skips your most important directory.
  3. It never runs. The server was off, the timer was disabled, the crontab was overwritten. There is no error because there is no process. This is the failure that kills people.

Each needs a different detector, so we will build all three.

Step 1: Create a channel that can get your attention

In Echobell, create a channel called backups and set its notification type to Calling for the jobs that matter. Copy its webhook URL:

https://hook.echobell.one/t/<channel-token>

Give it templates that make sense half-asleep, using the fields we will send below:

Title: Backup {{status}} on {{host}}
Body: {{job}}: {{detail}}

See the template reference for the syntax.

Step 2: Catch failures with a wrapper script

A small wrapper turns exit codes into alerts. Save this as /usr/local/bin/backup-run:

#!/usr/bin/env bash
set -uo pipefail

HOOK="https://hook.echobell.one/t/<channel-token>"
JOB="nightly-restic"

alert() {
  curl -fsS -m 10 -X POST "$HOOK" \
    -H "Content-Type: application/json" \
    -d "{\"job\":\"$JOB\",\"host\":\"$(hostname)\",\"status\":\"$1\",\"detail\":\"$2\"}" || true
}

restic backup /srv /etc --exclude-caches
code=$?

case $code in
  0)  ;;                                   # success: stay quiet
  3)  alert "incomplete" "Snapshot saved, but some files were unreadable" ;;
  11) alert "locked" "Repository is locked by another process" ;;
  12) alert "auth-failed" "Wrong repository password" ;;
  *)  alert "failed" "restic exited with code $code" ;;
esac

exit $code

Why the details matter:

  • set -e is deliberately not used, so the script still reaches the alert after restic fails.
  • || true on the curl means an unreachable webhook never masks the real exit code.
  • -m 10 stops a hung network call from blocking the next scheduled run.

The restic exit codes used above are documented in restic's manual: 0 success, 1 fatal error, 3 snapshot created but some source files unreadable, 10 repository does not exist, 11 failed to lock, 12 wrong password. Confirm them against restic version on your own machine, as codes were added over time.

borg and borgmatic

borg returns 0 for success, 1 for warnings and 2 or higher for errors. A file changing mid-backup produces 1, which is routine for log directories, so a common approach is to page only on 2 and above and send a quiet notification for 1:

borg create --stats ::'{hostname}-{now}' /srv /etc
code=$?
if [ "$code" -ge 2 ]; then
  alert "failed" "borg exited with code $code"
elif [ "$code" -eq 1 ]; then
  alert "warning" "borg finished with warnings"
fi

If you use borgmatic, its on_error hook runs a command when borgmatic itself hits an error, which is the natural place for the same curl. Note the limit the borgmatic docs call out themselves: if borgmatic never runs, on_error never fires. That is failure number three, and it needs Step 3.

Step 3: Catch the backup that never ran

A wrapper can only report on a run that happened. To catch silence, flip the logic: the backup pings a monitor when it succeeds, and the monitor alerts you when the ping is late. This is called a dead man's switch.

Healthchecks.io (hosted or self-hosted) does exactly this:

  1. Create a check with the schedule 0 3 * * * and a grace time of an hour or two.
  2. Add its ping to the end of your job so it only fires on success.
  3. In the check's integrations, add a Webhook pointing at your Echobell channel URL.

The ping goes at the end of the wrapper:

# in backup-run, just before the final `exit $code`; pings only on success
if [ $code -eq 0 ]; then
  curl -fsS -m 10 --retry 3 -o /dev/null https://hc-ping.com/<your-uuid>
fi

Healthchecks also supports /start and /fail ping variants and can accept the exit code in the URL, so a single check can also catch jobs that hang. The result: a dead server, a disabled timer and a deleted crontab line all produce the same outcome, a ringing phone. For the general pattern and other monitors, see cron job failure alerts.

Step 4: Check that the backup is fresh and restorable

A backup can run on schedule, exit 0, and still be useless, for example when you are backing up an empty directory after a mount failed. Add two cheap sanity checks to a separate daily job:

Snapshot age. Alert if the newest snapshot is older than you expect:

latest=$(restic snapshots --latest 1 --json | jq -r '.[0].time')
age_h=$(( ( $(date +%s) - $(date -d "$latest" +%s) ) / 3600 ))
if [ "$age_h" -gt 30 ]; then
  alert "stale" "Newest snapshot is ${age_h}h old"
fi

Repository integrity. restic check verifies the repository structure. Run it weekly; for large repositories, restic check --read-data-subset=5% samples data without reading everything. Alert on a non-zero exit.

Neither replaces a real test restore. Once a quarter, restore a file you can verify to a scratch directory. It is the only test that proves the backup works.

Step 5: Ring only when it matters

If every warning phones you, you will mute the channel, and then it will not ring for the one that counts. Use Echobell conditions to separate tiers on one channel:

status == "failed" || status == "stale" || status == "auth-failed"

Send incomplete and warning to a second, Time Sensitive or normal channel, so they appear on your lock screen without a ring. Add a time window if you want daytime failures to be quiet, using the UTC variables described in the conditions reference.

Frequently asked questions

How do I get notified when a backup fails?

Wrap the backup command, read its exit code, and send a webhook on any non-zero code. Add a dead man's switch such as Healthchecks.io so you also hear about runs that never started.

What restic exit codes should I alert on?

Alert loudly on 1, 10, 11 and 12, which mean a real failure. Treat 3 (snapshot created but some files unreadable) as a warning to investigate, since it can hide a permissions problem.

How do I know if a backup silently stopped running?

Use a heartbeat: the job pings a monitor on success, and the monitor alerts when the ping is late. An error hook cannot catch this because nothing runs to raise the error.

Is a successful exit code enough to trust a backup?

No. It means the tool finished, not that the data is complete or restorable. Check snapshot age, run periodic integrity checks, and test a restore regularly.

Can I get a phone call instead of an email?

Yes. Email is easy to miss. A calling channel in Echobell behaves like an incoming call and gets through iOS Focus Mode, so a failed backup at 03:00 is noticed.

Wrap-up

Three detectors cover the three failure modes: an exit-code wrapper for runs that fail, a freshness check for runs that succeed uselessly, and a heartbeat for runs that never happen. Route them to a calling channel, keep warnings on a quieter one, and a broken backup becomes a five-minute fix instead of a lost weekend.

Download Echobell for iPhone or get it on Google Play, create the backups channel, and run the wrapper once with a deliberately wrong password to hear it ring.

Related articles

CloudWatch Alarm Phone Call Alerts Without Lambda or Twilio

SNS can't dial a phone, and its payload is hard to parse. Route CloudWatch alarms through EventBridge to a calling webhook: setup, filters, pitfalls.

Read more

Sentry Phone Call Alerts for Production-Breaking Errors

Sentry has no phone call action. Route issue alerts through a webhook so only production-breaking errors ring your phone: setup, payload, and filters.

Read more

Cron Job Failure Alerts: Get a Phone Call When a Job Dies

Cron jobs fail silently. Learn three reliable patterns to detect failed or missed scheduled jobs and get a push notification or phone call within seconds.

Read more