Troubleshooting
Start from the symptom and stable error code, preserve evidence, and choose one safe recovery path.
Exact release scope. This page documents the reviewed v0.0.1 source at fcc5871. Check release status before enabling a gated capability.
First response
- 01
Stop compounding changes
Do not start a conflicting run, repeatedly retry, delete evidence, or patch target infrastructure.
- 02
Capture the diagnostic envelope
Record UTC time, environment, user, release, error_code, nextAction, run/step/command/cluster/agent/correlation IDs.
- 03
Check platform dependencies
Verify health, readiness, clock, database, artifacts, replicas/leader, registry, and active agents.
- 04
Find the first causal failure
Use ordered run evidence. Cleanup failures may follow an earlier provider or host error.
- 05
Apply one documented recovery
Correct configuration/input, restore dependency, recover agent/lease, retry eligible work, or restore/cleanup.
API does not become ready
| Code | Likely cause | Action |
|---|---|---|
PRODUCTION_STARTUP_GUARD_FAILED | Mandatory production safety setting is missing/unsafe. | Correct database, storage, secrets, replicas, leader, or environment; never switch to Development. |
STARTUP_INVALID_CONFIGURATION | A value is malformed or an invalid combination. | Compare with the exact release example/chart and inspect the named setting. |
STARTUP_DEPENDENCY_BLOCKED | Database, artifact store, clock, or another required dependency is unavailable. | Restore dependency and verify network/TLS/auth. |
DB_MIGRATION_STATUS_UNAVAILABLE | Schema state cannot be assessed. | Stop rollout, restore database access, run migration diagnostics. |
Agent absent or work remains queued
| Code | Meaning | Action |
|---|---|---|
AGENT_VERSION_INCOMPATIBLE | Agent is outside the N/N-1 window. | Deploy a compatible agent before issuing new contract work. |
AGENT_CLOCK_SKEW_EXCESSIVE | Agent-reported UTC drift exceeds the rejection threshold. | Correct API/agent time source; verify /health/clock. |
AGENT_HANDSHAKE_POLICY_REJECTED | Placement, OS/runtime, trust, or reachability failed. | Inspect rejection_code and align the active hosting profile. |
CONTRACT_VERSION_MISMATCH | No active agent advertises the run contract. | Upgrade agent or issue work on a supported contract. |
Run, stream, and artifact errors
| Code family | Action |
|---|---|
| RUNS_RETRY_* | Read the specific replay, active-run, role, rate, failed-step, or kill-switch reason before retrying. |
| RUNS_STREAM_* / RUNS_LOGS_* | Preserve the cursor error; distinguish invalid/stale cursor, retention gap, and query failure. |
| RUNS_ARTIFACTS_PENDING | Wait for upload/projection; do not treat absence as final. |
| RUNS_ARTIFACTS_EXPIRED | Use retained metadata/audit and rerun only if policy and lifecycle allow. |
| ARTIFACT_BLOB_MISSING / ARTIFACT_INTEGRITY_FAILED | Quarantine the evidence result; investigate storage and restore from a verified copy. |
| RUNS_ARTIFACTS_QUOTA_EXCEEDED | Stop generating evidence, preserve active run safety, increase/clean storage under retention policy. |
Cluster and add-on symptoms
| Symptom | Check first | Do not |
|---|---|---|
| Node preparation fails | OS/kernel, swap/runtime, ports/routes/DNS/NTP, SSH, duplicate identity. | Skip hard probes or hand-edit a half-prepared node. |
| Lifecycle step is stale | Agent heartbeat, lease, checkpoint, recovery worker, first failed command. | Start another scale/upgrade/delete run. |
| Pod InvalidImageName | Rendered image for duplicate/malformed digest. | Increase Helm timeout. |
| ImagePullBackOff / no basic auth | Managed pull Secret metadata and workload/ServiceAccount reference. | Paste Docker config into logs or patch only the pod. |
| Add-on health gate failed | Check-specific redacted artifact and dependencies. | Open dashboard or mark installed based only on Running pods. |
| Delete leaves resources | Operation residual inventory and provider ownership tags/IDs. | Bulk-delete unrelated shared-project resources. |
Escalate safely
Attach a reviewed support bundle plus a short timeline, expected versus observed outcome, exact release identity, environment profile, run/correlation IDs, first failing code, and actions already attempted. State whether the environment is disposable and whether a destructive recovery is permitted.
Build a secret-safe support package →