Backups, Monitoring and Recovery

A backup you have never restored is a hope, not a backup.

The 3-2-1 rule

Keep 3 copies of important data, on 2 different kinds of storage, with 1 copy off-site (another provider or region). Ransomware and account mistakes can delete everything in one place.

What to back up

• Databases (as dumps, not by copying live files)
• User uploads
• Configuration: /etc, unit files, .env (encrypted)
• Your runbook

The application code is already in Git, so it does not need backing up separately.

A backup script

/usr/local/bin/pg-backup.sh:

#!/usr/bin/env bash
set -euo pipefail
STAMP=$(date -u +%Y%m%dT%H%MZ)
DIR=/var/backups/postgres
mkdir -p "$DIR"
pg_dump -Fc pandadev > "$DIR/pandadev-$STAMP.dump"
find "$DIR" -name '*.dump' -mtime +14 -delete
# copy off-site (rclone works with S3, R2, B2, Google Drive...)
rclone copy "$DIR/pandadev-$STAMP.dump" offsite:pandadev-backups/

Schedule it with the systemd timer from the services module. Encrypt off-site copies (rclone crypt or age).

Restore drill

Once a month, restore to a scratch database and check it:

createdb pandadev_restore_test
pg_restore -d pandadev_restore_test /var/backups/postgres/pandadev-LATEST.dump
psql pandadev_restore_test -c "select count(*) from users;"
dropdb pandadev_restore_test

Time it. "How long would we be down?" now has a real answer.

Monitoring basics

Watch four things, and alert before they become outages:

Simple, effective tools: an external uptime checker (UptimeRobot, Better Stack) hitting /api/health, node_exporter with Prometheus and Grafana for metrics, and Sentry for application errors.

Final project

On a practice server: create an admin user, harden SSH, enable the firewall and automatic updates, run an app under systemd with hardening, rotate its logs, schedule an off-site backup with a timer, then destroy the database and restore it from backup. Write every step in your runbook. That runbook is what a professional system administrator delivers.