Administer · v0.0.1

Observability & support

Use health, readiness, clock, build, agent, run, log, artifact, and correlation evidence without leaking secrets.

AudienceSRE, NOC, support engineersReading time13 minCommands in context5 guided stepsReviewed sourcefcc5871 · 30 July 2026

Exact release scope. This page documents the reviewed v0.0.1 source at fcc5871. Check release status before enabling a gated capability.

Platform endpoints

Endpoint	Use	Alert interpretation
`GET /health`	Process/liveness health.	Repeated failure means restart/crash/dependency investigation.
`GET /ready`	Traffic readiness and required dependencies.	Remove from service and investigate the reported dependency.
`GET /health/clock`	Server clock/reference health and thresholds.	Correct time source before leases, signatures, or agent work are trusted.
`GET /api/v1/admin/build-info`	Release digest and supply-chain metadata.	Mismatch indicates incomplete rollout or unverified runtime.
`GET /api/v1/admin/agents/versions`	Agent versions, rejections, clock and fleet posture.	Mismatch/rejection can leave runs queued.

Commands for this step

Check the control plane and compatibility posture

Start with liveness, readiness, and clock. Continue to build, migration, and agent compatibility only when the basic endpoint boundary is reachable.

Run in orderComplete each step successfully before continuing.

01
Read-onlyHTTP API
Check liveness, readiness and clock
Determine whether the API process, dependencies and trusted time posture are healthy.
Before you run it
- CP_URL points to the approved ClusterPilot endpoint
- jq
HTTP API · health-basic
```
curl --fail-with-body -sS $CP_URL/health | jq .
curl --fail-with-body -sS $CP_URL/ready | jq .
curl --fail-with-body -sS $CP_URL/health/clock | jq .
```
Expected result
Health and readiness are successful; clock drift remains inside the approved thresholds.
Reviewed sourcedocs/agent/self-hosted-install.md
02
Read-onlyHTTP API
Inspect build and hosting diagnostics
Verify exact runtime identity, topology, persistence and artifact-source posture.
Before you run it
- Scoped administrative API key
- jq
HTTP API · health-build-hosting
```
curl --fail-with-body -sS -H "Authorization: Bearer $CLUSTERPILOT_API_KEY" \
  $CP_URL/api/v1/admin/build-info | jq .
curl --fail-with-body -sS -H "Authorization: Bearer $CLUSTERPILOT_API_KEY" \
  $CP_URL/api/v1/admin/hosting-diagnostics | jq .
curl --fail-with-body -sS -H "Authorization: Bearer $CLUSTERPILOT_API_KEY" \
  $CP_URL/api/v1/admin/distribution-diagnostics | jq .
```
Expected result
Commit and image identity match the release; startup is ready and no blocking distribution signal exists.
Reviewed sourcedocs/operations/self-hosted-upgrade-rollback-runbook.md
03
Read-onlyHTTP API
Check compatibility and database migrations
Make an upgrade or rollback decision from machine-readable release and schema evidence.
Before you run it
- Scoped administrative API key
HTTP API · health-compat-migrations
```
curl --fail-with-body -sS -H "Authorization: Bearer $CLUSTERPILOT_API_KEY" \
  $CP_URL/api/v1/admin/compatibility/check | jq .
curl --fail-with-body -sS -H "Authorization: Bearer $CLUSTERPILOT_API_KEY" \
  $CP_URL/api/v1/admin/migrations/status | jq .
```
Expected result
Compatibility is not unsupported, pending migrations are understood and lifecycle decisions are not blocked.
Reviewed sourcedocs/operations/self-hosted-upgrade-rollback-runbook.md
04
Read-onlyHTTP API
Inspect agent fleet compatibility
Review N/N-1 versions, contracts, rejection causes and clock health before issuing work.
Before you run it
- Scoped agent-read or administrative API key
HTTP API · health-agent-fleet
```
curl --fail-with-body -sS -H "Authorization: Bearer $CLUSTERPILOT_API_KEY" \
  $CP_URL/api/v1/admin/agents/versions | jq .
curl --fail-with-body -sS -H "Authorization: Bearer $CLUSTERPILOT_API_KEY" \
  $CP_URL/api/v1/agent/active | jq .
```
Expected result
Required agents are active, compatible and advertise the contract needed by queued runs.
Reviewed sourcedocs/agent/troubleshooting-and-migration.md

Operation telemetry

Use run state and step duration for operator impact; events for transitions; logs and command results for bounded detail; artifacts for durable evidence. Cursor, retention, integrity, and expiry errors must remain visible to the client.

Signal	Primary question
Run queue and assignment	Can compatible agents accept work?
Step latency and retries	Which lifecycle boundary is slow or unstable?
Failure codes and nextAction	What safe recovery class applies?
Artifact upload/integrity	Is evidence complete and trustworthy?
Provider rate/circuit state	Is external throttling or failure preventing progress?
Cleanup and stale leases	Can interrupted work recover without manual mutation?

Logging rules

✓
Use structured event names, severity, timestamp, component, environment, run/step/command/correlation IDs.
✓
Log secret references and credential IDs only when safe; never secret values, cookies, tokens, private keys, kubeconfigs, or Docker config.
✓
Bound command output, preserve truncation markers, and store large evidence as protected artifacts.
✓
Keep customer data retention and access controls aligned across application logs, run artifacts, audit, and the external log platform.
✓
Treat redaction as defense in depth; prevent sensitive values from entering inputs in the first place.

Support bundle contents

Include	Exclude
Release/build identity, configuration posture, feature-gate names	Environment files and raw Secret values
Health/readiness/clock summaries	Session cookies, CSRF tokens, API keys
Run/step state, error codes, correlation IDs	Provider tokens, SSH keys, registry passwords
Redacted logs and artifact metadata/checksums	Kubeconfigs, ServiceAccount tokens, private endpoints with session material
Replica/leader/agent compatibility and dependency status	Raw database dumps unless separately encrypted and approved

Minimum alert set

✓
API not healthy or not ready; restart loop; replica divergence.
✓
Database unavailable, migration conflict, storage capacity or artifact-integrity failure.
✓
No active compatible agent; excessive heartbeat clock skew; rejected handshake.
✓
Queued/running run beyond agreed threshold; stale step; failed cleanup; repeated retry.
✓
Provider circuit open or sustained rate limiting.
✓
Backup verification failure, missing recovery point, or retention hold conflict.

Commands for this step

Inspect the Prometheus metrics used by alerting

Use read-only metrics access to verify that alert rules reference real, populated series.

Use when neededDo not run these commands unless the described condition applies.

01
Read-onlyHTTP API
Read Prometheus metrics
Inspect queue, storage, retries, provider and distribution telemetry from the approved monitoring path.
Before you run it
- Metrics authorization and network policy permit the caller
HTTP API · health-metrics
```
curl --fail-with-body -sS -H "Authorization: Bearer $CLUSTERPILOT_API_KEY" \
  $CP_URL/metrics
```
Expected result
Prometheus exposition data is returned without secret-bearing labels.
Reviewed sourcedocs/operations/cloud-provider-rollout-compatibility-runbook.md