Troubleshooting
Start from the symptom and stable error code, preserve evidence, and choose one safe recovery path.
First response
- 01
Stop compounding changes
Do not start a conflicting run, repeatedly retry, delete evidence, or patch target infrastructure.
- 02
Capture the diagnostic envelope
Record UTC time, environment, user, release, error_code, nextAction, run/step/command/cluster/agent/correlation IDs.
- 03
Check platform dependencies
Verify health, readiness, clock, database, artifacts, replicas/leader, registry, and active agents.
- 04
Find the first causal failure
Use ordered run evidence. Cleanup failures may follow an earlier provider or host error.
- 05
Apply one documented recovery
Correct configuration/input, restore dependency, recover agent/lease, retry eligible work, or restore/cleanup.
API does not become ready
| Code | Likely cause | Action |
|---|---|---|
PRODUCTION_STARTUP_GUARD_FAILED | Mandatory production safety setting is missing/unsafe. | Correct database, storage, secrets, replicas, leader, or environment; never switch to Development. |
STARTUP_INVALID_CONFIGURATION | A value is malformed or an invalid combination. | Compare with the exact release example/chart and inspect the named setting. |
STARTUP_DEPENDENCY_BLOCKED | Database, artifact store, clock, or another required dependency is unavailable. | Restore dependency and verify network/TLS/auth. |
DB_MIGRATION_STATUS_UNAVAILABLE | Schema state cannot be assessed. | Stop rollout, restore database access, run migration diagnostics. |
Agent absent or work remains queued
| Code | Meaning | Action |
|---|---|---|
AGENT_VERSION_INCOMPATIBLE | Agent is outside the N/N-1 window. | Deploy a compatible agent before issuing new contract work. |
AGENT_CLOCK_SKEW_EXCESSIVE | Agent-reported UTC drift exceeds the rejection threshold. | Correct API/agent time source; verify /health/clock. |
AGENT_HANDSHAKE_POLICY_REJECTED | Placement, OS/runtime, trust, or reachability failed. | Inspect rejection_code and align the active hosting profile. |
CONTRACT_VERSION_MISMATCH | No active agent advertises the run contract. | Upgrade agent or issue work on a supported contract. |
Run, stream, and artifact errors
| Code family | Action |
|---|---|
| RUNS_RETRY_* | Read the specific replay, active-run, role, rate, failed-step, or kill-switch reason before retrying. |
| RUNS_STREAM_* / RUNS_LOGS_* | Preserve the cursor error; distinguish invalid/stale cursor, retention gap, and query failure. |
| RUNS_ARTIFACTS_PENDING | Wait for upload/projection; do not treat absence as final. |
| RUNS_ARTIFACTS_EXPIRED | Use retained metadata/audit and rerun only if policy and lifecycle allow. |
| ARTIFACT_BLOB_MISSING / ARTIFACT_INTEGRITY_FAILED | Quarantine the evidence result; investigate storage and restore from a verified copy. |
| RUNS_ARTIFACTS_QUOTA_EXCEEDED | Stop generating evidence, preserve active run safety, increase/clean storage under retention policy. |
Cluster and add-on symptoms
| Symptom | Check first | Do not |
|---|---|---|
| Node preparation fails | OS/kernel, swap/runtime, ports/routes/DNS/NTP, SSH, duplicate identity. | Skip hard probes or hand-edit a half-prepared node. |
| Lifecycle step is stale | Agent heartbeat, lease, checkpoint, recovery worker, first failed command. | Start another scale/upgrade/delete run. |
| Pod InvalidImageName | Rendered image for duplicate/malformed digest. | Increase Helm timeout. |
| ImagePullBackOff / no basic auth | Managed pull Secret metadata and workload/ServiceAccount reference. | Paste Docker config into logs or patch only the pod. |
| Add-on health gate failed | Check-specific redacted artifact and dependencies. | Open dashboard or mark installed based only on Running pods. |
| Delete leaves resources | Operation residual inventory and provider ownership tags/IDs. | Bulk-delete unrelated shared-project resources. |
Escalate safely
Attach a reviewed support bundle plus a short timeline, expected versus observed outcome, exact release identity, environment profile, run/correlation IDs, first failing code, and actions already attempted. State whether the environment is disposable and whether a destructive recovery is permitted.
Build a secret-safe support package →