OperationRuns
Understand the durable execution primitive behind lifecycle, add-on, validation, and recovery work.
Exact release scope. This page documents the reviewed v0.0.1 source at fcc5871. Check release status before enabling a gated capability.
Run model
An OperationRun records the accepted request, contract version, plan, ordered steps, assignments, events, logs, commands, artifacts, terminal result, and recovery context. Use the run ID and correlation ID in change records and support cases.
| Layer | Examples |
|---|---|
| Admission | Caller, scope, role, idempotency key, validation and gate decision |
| Plan | Stable identifiers, intended resources, step order, compatibility and risk |
| Execution | Agent assignment, leases, command attempts, progress, cancellation |
| Evidence | Events, log chunks, artifacts, checksums, redacted command results |
| Outcome | Succeeded, failed, cancelled, next action, residual resources, cleanup state |
States and transitions
| State | Meaning | Do |
|---|---|---|
| Queued | Accepted but not yet assigned. | Inspect notAssignableReasonCode and active agent compatibility. |
| Running | An agent owns one or more active steps. | Observe; avoid conflicting work. |
| Cancel requested | Cancellation was accepted, cleanup may still be active. | Wait for terminal state; do not delete evidence. |
| Succeeded | All required steps and outcome projection completed. | Review acceptance evidence. |
| Failed | A required step or recovery policy terminated unsuccessfully. | Inspect the first causal failure and safe next action. |
| Cancelled | Cancellation and owned cleanup reached a terminal state. | Confirm residual resources before closing. |
Events, logs, commands, and artifacts
Event and log streams use ordered cursors. A stale cursor or retention gap is explicit; clients should not silently restart and present an incomplete stream as complete. Artifact downloads are integrity checked and can be pending, expired, missing, corrupt, too large, or unavailable.
- ✓Treat ProblemDetails error_code, nextAction, runId, stepId, commandId, and correlationId as the stable diagnostic envelope.
- ✓Keep secret values out of screenshots, copied logs, URLs, tickets, and run options.
- ✓Verify artifact checksum and release identity before using evidence for approval.
- ✓Export only the minimum evidence needed and retain it under the customer's data policy.
Cancel and retry
| Action | Allowed when | Important behavior |
|---|---|---|
| Cancel | The run is active and the contract exposes cancellation. | Returns a requested state first; terminal cancellation follows cleanup. |
| Retry failed steps | The run is failed, replay-safe, and failed steps are eligible. | Uses idempotency and may require preserved artifacts or a replay token. |
| Start a new run | Inputs or intended state changed materially. | Creates a new audit and evidence boundary; never overwrite history. |
| Manual repair | Only when an approved runbook explicitly requires it. | Record the change and reconcile ClusterPilot state afterward. |
Operator investigation order
- 01
Capture identifiers
Record run, cluster, step, command, agent, and correlation identifiers.
- 02
Read the admission and plan
Confirm the accepted intent, exact release/contract, and guardrail decisions.
- 03
Find the first causal failure
Later failures may be cleanup consequences. Start with the earliest failed required step.
- 04
Inspect redacted evidence
Use artifact references, bounded command output, provider response category, and nextAction.
- 05
Choose one recovery path
Correct input, restore dependency, recover agent, retry eligible work, cancel/cleanup, or escalate with a support bundle.