Documentation / Administer
Administer · v0.0.1 Backup & restore Protect the relational state, evidence artifacts, secrets protection key, and release identity as one recoverable system.
Audience Database administrators, SRE, disaster-recovery ownersReading time 17 minCommands in context 7 guided stepsReviewed source fcc5871 · 30 July 2026
RC Exact release scope. This page documents the reviewed v0.0.1 source at fcc5871. Check release status before enabling a gated capability.
Backup set Component Why it matters Protection PostgreSQL Users, configuration, runs, plans, audit, metadata, projections. Consistent custom-format dump or approved physical/PITR process. Artifact storage Logs, command evidence, reports, bundles, integrity material. Snapshot/copy with object metadata and checksums. Secrets protection key Decrypts protected stored material. Separate encrypted escrow with controlled recovery access. Release and configuration evidence Identifies exact runtime, compatibility, gates, and topology. Immutable copy with the recovery package. External dependencies Registry, object store, DNS, TLS, external database. Their own tested recovery plans and credentials.
Commands for this step Back up the filesystem artifact set Capture the artifact root with ownership and release metadata aligned to the same backup point as PostgreSQL.
Run in order Complete each step successfully before continuing.
01
Read-only Linux
Back up filesystem artifacts Archive run logs, command evidence, reports and integrity metadata from filesystem-backed storage.
Before you run it Quiesced or storage-consistent artifact snapshot Sufficient destination capacity Linux · backup-artifacts Copy
tar -czf clusterpilot-artifacts-$(date -u +%Y%m%dT%H%M%SZ).tar.gz \
-C /var/lib/clusterpilot/artifacts .
sha256sum clusterpilot-artifacts-*.tar.gz > clusterpilot-artifacts.sha256✓ Expected result The archive and checksum are retained with the database backup and release identity.
Reviewed source docs/operations/backup-restore-runbook.mdPostgreSQL logical backup The custom format is compressed and can be selectively inspected/restored with pg_restore. Keep credentials out of shell history and process output; the variable above represents an injected secret in the backup job, not a value to paste into a terminal.
PostgreSQL pg_dump reference↗ Commands for this step Create the PostgreSQL backup Create the logical backup and verify that the file exists before encrypting or transferring it.
Run in order Complete each step successfully before continuing.
01
Read-only Linux
Back up PostgreSQL Create a compressed logical database backup without restoring ownership or privilege statements.
Before you run it PostgreSQL client matching the supported server family Backup role and protected password delivery Linux · backup-postgres Copy
pg_dump \
--format=custom \
--compress=9 \
--no-owner \
--no-privileges \
--host "$PGHOST" \
--port "$PGPORT" \
--username "$PGUSER" \
--dbname "$PGDATABASE" \
--file clusterpilot-$(date -u +%Y%m%dT%H%M%SZ).dump✓ Expected result pg_dump exits 0 and creates a non-empty custom-format dump.
Reviewed source docs/operations/backup-restore-runbook.mdCommands for this step Choose exactly one approved backup encryption method Encrypt the verified dump before it leaves the backup host. Do not apply both formats to the same operational copy.
Choose one path Run only the command that matches your situation.
01
Changes state Linux
Encrypt a database backup with age Use this when Preferred when the organization uses age recipients.
Encrypt the database dump for an approved offline or external backup target.
Before you run it age Approved recipient identity Linux · backup-encrypt-age Copy
age --encrypt --recipient "$CLUSTERPILOT_BACKUP_AGE_RECIPIENT" \
--output clusterpilot.dump.age \
clusterpilot.dump✓ Expected result An encrypted .age file exists and decrypts in the restore drill.
Reviewed source docs/operations/backup-restore-runbook.md02
Changes state Linux
Encrypt a database backup with GPG Use this when Use instead of age when the approved recovery process is based on GPG.
Use the approved GPG recipient when age is not the enterprise encryption standard.
Before you run it GnuPG Trusted recipient key Linux · backup-encrypt-gpg Copy
gpg --batch --yes --encrypt \
--recipient "$CLUSTERPILOT_BACKUP_GPG_RECIPIENT" \
--output clusterpilot.dump.gpg \
clusterpilot.dump✓ Expected result An encrypted .gpg file exists and decrypts in the restore drill.
Reviewed source docs/operations/backup-restore-runbook.mdConsistency and verification 01 Quiesce or record the consistency boundary Coordinate active OperationRuns and artifact writes so database metadata and blobs can be reconciled.
02 Back up PostgreSQL and artifacts Capture timestamps, checksums, version, object count, and any excluded/held data.
03 Escrow key and release metadata Store separately under dual control where required.
04 Run structural verification List the dump, sample artifact blobs, verify checksums, and ensure the exact release is available.
05 Restore into an isolated environment Prove health, readiness, decryption, run/evidence reads, agent re-registration, and one bounded operation.
Restore procedure ✓ Deploy the same or an explicitly compatible ClusterPilot release before attaching restored state.
✓ Restore artifacts to the configured provider and verify metadata/checksum parity.
✓ Inject the matching secrets protection key and key ID before testing protected data.
✓ Run database migration/compatibility checks; never force a downgrade over forward-only migrations.
✓ Rotate bootstrap/session/provider credentials when incident or environment transfer policy requires it.
PostgreSQL pg_restore reference↗ Commands for this step Restore into a clean target and prove readiness Restore is a controlled recovery workflow, never an in-place experiment. Verify the recovered API and data before traffic is switched.
Run in order Complete each step successfully before continuing.
01
Destructive Linux
Restore PostgreSQL into a clean target Restore a verified database dump during an isolated recovery test or approved incident.
Before you run it Approved clean target Verified dump Matching separately protected Secrets__Protection__MasterKey Artifact backup from the same recovery point Linux · restore-postgres Copy
createdb --host "$PGHOST" --port "$PGPORT" --username "$PGUSER" "$PGDATABASE"
pg_restore \
--clean \
--if-exists \
--no-owner \
--no-privileges \
--host "$PGHOST" \
--port "$PGPORT" \
--username "$PGUSER" \
--dbname "$PGDATABASE" \
clusterpilot.dump✓ Expected result Restore completes; migrations, health, authentication, run history and artifact integrity pass application-level verification.
Reviewed source docs/operations/backup-restore-runbook.md02
Read-only HTTP API
Verify the restored control plane Prove that the restored application can read protected state and durable evidence.
Before you run it Restored API started in the isolated target Restored master key and artifact set HTTP API · restore-verify Copy
curl --fail-with-body -sS $CP_URL/health | jq .
curl --fail-with-body -sS $CP_URL/ready | jq .
curl --fail-with-body -sS -H "Authorization: Bearer $CLUSTERPILOT_API_KEY" \
$CP_URL/api/v1/admin/migrations/status | jq .
curl --fail-with-body -sS -H "Authorization: Bearer $CLUSTERPILOT_API_KEY" \
$CP_URL/api/v1/admin/hosting-diagnostics | jq .✓ Expected result The service is ready, migrations are current and artifact-source/persistence diagnostics are valid.
Reviewed source docs/operations/backup-restore-runbook.mdDefine and test objectives ClusterPilot does not invent an RPO or RTO for the customer. Define both from accepted data loss, evidence retention, run recovery, external dependency recovery, and staffing. Test on the real deployment profile and retain measured timings and failure observations.
A successful backup command is not a recovery test. Production approval requires a clean isolated restore and application-level verification.
Commands for this step Run an isolated disaster-recovery drill Use the release-owned drill to test the documented RPO/RTO path without touching the production environment.
Use when needed Do not run these commands unless the described condition applies.
01
Destructive PowerShell
Run the isolated disaster-recovery drill Exercise database, artifact and protected-key recovery against an isolated API target without touching production.
Before you run it Disposable isolated PostgreSQL target Restorable artifact set Separately protected recovery key Explicit DR exercise approval PowerShell · dr-isolated-drill Copy
$env:CLUSTERPILOT_DR_CONNECTION_STRING = "<isolated-postgresql-connection-string>"
$env:CLUSTERPILOT_DR_MASTER_KEY = "<isolated-base64-master-key>"
& ./scripts/dr-drill.ps1 `
-ApiProjectPath ./backend/src/ClusterPilot.Api/ClusterPilot.Api.csproj `
-ApiBaseUrl http://127.0.0.1:5085 `
-ConnectionString $env:CLUSTERPILOT_DR_CONNECTION_STRING `
-ArtifactRootPath ./artifacts/dr-drill `
-MasterKeyBase64 $env:CLUSTERPILOT_DR_MASTER_KEY `
-PostgresHost <isolated-postgres-host> `
-PostgresPort 5432 `
-PostgresDatabase clusterpilot_dr `
-PostgresUser clusterpilot_dr `
-PostgresPassword $env:CLUSTERPILOT_DR_POSTGRES_PASSWORD `
-StartupTimeoutSeconds 300✓ Expected result The drill proves application startup and restored database/artifact/key behavior, then records failures without modifying production.
Stop conditions and operator notes Never point this command at a production database, artifact root or API URL. Remove the temporary environment variables after the drill. Reviewed source scripts/dr-drill.ps1