Administer · v0.0.1

Upgrade & rollback

Stage a signed compatible release, protect data, preserve agent compatibility, and restore instead of forcing unsafe schema rollback.

AudienceRelease engineers, SRE, platform administratorsReading time18 minCommands in context3 guided stepsReviewed sourcefcc5871 · 30 July 2026

Exact release scope. This page documents the reviewed v0.0.1 source at fcc5871. Check release status before enabling a gated capability.

Pre-upgrade gates

✓
Target bundle, metadata, compatibility manifest, image signatures, SBOM, and provenance are verified.
✓
Current environment is healthy and no conflicting high-risk OperationRun is active.
✓
A current backup passed verification and a recent restore rehearsal meets the recovery objective.
✓
Database migration status, storage capacity, hosting diagnostics, clock, and dependencies are healthy.
✓
Target control plane accepts the active N/N-1 agent fleet and the agent rollout order is documented.
✓
Feature-gate changes and known release limitations have explicit owners and rollback decisions.

Commands for this step

Run the update preflight

Do not start a control-plane update until the API reports a compatible, unblocked plan.

Run in orderComplete each step successfully before continuing.

01
Read-onlyHTTP API
Run update preflight
Evaluate signed candidate, topology, compatibility, backup and rollout policy before update mutation.
Before you run it
- In-product update feature explicitly enabled
- Signed digest-pinned candidate
- Configured supported transport
- Fresh backup evidence
HTTP API · api-update-preflight
```
curl --fail-with-body -sS -X POST \
  -H "Authorization: Bearer $CLUSTERPILOT_API_KEY" \
  -H "Idempotency-Key: <uuid>" \
  -H "Content-Type: application/json" \
  --data @update-preflight.json \
  $CP_URL/api/v1/update/preflight | jq .
curl --fail-with-body -sS -H "Authorization: Bearer $CLUSTERPILOT_API_KEY" \
  $CP_URL/api/v1/update/status | jq .
```
Expected result
Every blocking check is resolved before an update run is created.
Reviewed sourcedocs/operations/update-operator-runbook.md

Upgrade sequence

01
Verify and mirror target artifacts
Use exact digests and preserve signed metadata in the customer registry/artifact store.
02
Run target preflight and dry-run
Resolve configuration, migration, compatibility, registry, storage, or governance conflicts.
03
Prepare compatible agents
Roll agents that advertise the next contract before the control plane emits new-version work.
04
Apply the control-plane release
Use the chart or release-owned installer; observe migration and leader behavior.
05
Verify
Check health, readiness, clock, build identity, workers, agents, run reads, artifacts, audit, and one bounded operation.
06
Advance staged gates
Enable only approved capabilities after the base platform is stable.

Database migration boundary

Database migrations are forward-oriented. A previous application image may not understand the upgraded schema. Do not run an older control plane against a migrated production database unless the compatibility contract explicitly permits it.

Commands for this step

Apply forward-only database migrations by approved exception

The release entrypoint normally owns migrations. Use this command only when the documented recovery decision explicitly requires manual forward migration.

Use when neededDo not run these commands unless the described condition applies.

01
Changes stateCross-platform
Apply forward-only EF Core migrations
Apply reviewed forward migrations during a controlled maintenance window when startup auto-migrate is disabled.
Before you run it
- Verified PostgreSQL and artifact backups
- Target release source or migration bundle
- Database__AutoMigrateOnStartup=false
- One approved migration actor
Cross-platform · db-manual-migrate
```
dotnet ef database update \
  --project backend/src/ClusterPilot.Infrastructure \
  --startup-project backend/src/ClusterPilot.Api \
  --context ClusterPilotDbContext
```
Expected result
GET /api/v1/admin/migrations/status reports pending: [] and /health becomes healthy.
Stop conditions and operator notes
- Never run dotnet ef database update <previous> in production.
Reviewed sourcedocs/operations/db-migration-runbook.md

Rollback decision

Condition	Action
Application rollout failed before migration	Roll back deployment to the verified previous digest and verify.
Migration completed and backward compatibility is documented	Use only the approved rollback plan and verify schema state.
Migration completed and compatibility is unknown/false	Restore the complete pre-upgrade backup set.
Agent contract mismatch	Stop issuing the new run contract, complete safe work, restore compatible agent/control-plane order.
Feature-gate regression	Disable the specific gate if the runbook and active operations permit it; retain evidence.

Commands for this step

Roll back the Helm release only when compatibility permits

A Helm rollback does not reverse a forward-only database migration. Follow the compatibility decision and use restore when schema rollback is not supported.

Use when neededDo not run these commands unless the described condition applies.

01
DestructiveKubernetes
Rollback a Helm release when compatibility permits
Return to a previous chart revision only after the compatibility and database rollback decision says allowed.
Before you run it
- Approved rollback decision
- Compatibility check does not report blocked
- Database schema has not advanced beyond the rollback floor
- Verified pre-upgrade backup
Kubernetes · helm-rollback
```
helm history clusterpilot --namespace clusterpilot
helm rollback clusterpilot <revision> --namespace clusterpilot --wait --timeout 600s
kubectl -n clusterpilot rollout status deployment/clusterpilot --timeout=300s
```
Expected result
The selected revision is deployed and health, readiness and smoke checks pass.
Stop conditions and operator notes
- Never use a binary-only rollback to cross an incompatible forward-only database migration.
Reviewed sourcedocs/operations/self-hosted-upgrade-rollback-runbook.md

Post-upgrade acceptance

✓
All replicas converge on the intended digest and report Ready.
✓
Only one leader owns singleton work where required.
✓
Agents are accepted, time is healthy, and queued work has no compatibility mismatch.
✓
Existing users, roles, sessions, secrets, runs, logs, artifacts, and audit records remain readable.
✓
Backup verification and a new recovery point complete on the upgraded release.
✓
Change evidence records release identity, migration, tests, open limits, and approver.

Upgrade & rollback

Pre-upgrade gates

Upgrade sequence

Verify and mirror target artifacts

Run target preflight and dry-run

Prepare compatible agents

Apply the control-plane release

Verify

Advance staged gates

Database migration boundary

Rollback decision

Post-upgrade acceptance