Operate · v0.0.1

Conformance validation

Run bounded Sonobuoy evidence workflows without confusing conformance with ClusterPilot health or production approval.

AudienceQA, release engineering, platform operatorsReading time16 minCommands in context14 guided stepsReviewed sourcefcc5871 · 30 July 2026

Exact release scope. This page documents the reviewed v0.0.1 source at fcc5871. Check release status before enabling a gated capability.

Purpose and boundary

ClusterPilot orchestrates Sonobuoy as an independent standardized evidence workflow. A Sonobuoy result does not change the ClusterPilot cluster-health model, and a passing result does not approve the provider, deployment, security posture, backup, or release by itself.

Sonobuoy v0.57.5 release

Validation modes

Mode	Purpose	Guardrail
Quick	Bounded reachability and smoke evidence.	Lowest operational impact; still uses an admitted run.
Non-disruptive conformance	Broader conformance evidence designed to avoid disruptive tests.	Requires compatibility, capacity, and artifact/cleanup checks.
Certified conformance	Certification-oriented suite.	Explicit confirmation and production-impact override required.

Commands for this step

Plan a live conformance environment

Release engineering uses this protected planner before any environment-scoped conformance validation.

Protected workflowRun only on an approved maintainer, security, or release system.

01
Read-onlyPowerShell
Plan a live conformance environment validation
Resolve the exact live quick, non-disruptive or certified conformance request without admitting the run.
Before you run it
- Approved disposable target cluster
- Scoped environment variables
- Exact candidate commit
- Evidence output root
PowerShell · release-conformance-environment-plan
```
& ./tests/conformance/Invoke-ClusterConformanceEnvironmentValidation.ps1 `
  -Mode quick `
  -DeploymentProfile harbor `
  -Architecture amd64 `
  -Scenario success `
  -ExpectedResultState passed `
  -BaseUrl $env:CLUSTERPILOT_BASE_URL `
  -ApiKey $env:CLUSTERPILOT_API_KEY `
  -TenantId $env:CLUSTERPILOT_TENANT_ID `
  -ClusterId $env:CLUSTERPILOT_CONFORMANCE_CLUSTER_ID `
  -IdempotencyKey $env:CLUSTERPILOT_CONFORMANCE_IDEMPOTENCY_KEY `
  -OutputDirectory $env:CLUSTERPILOT_CONFORMANCE_EVIDENCE_ROOT `
  -CandidateCommitSha "<candidate-commit-sha>" `
  -PollIntervalSeconds 15 `
  -MaximumRetries 4 `
  -PlanOnly
```
Expected result
The script validates and records the intended environment request without starting conformance execution.
Stop conditions and operator notes
- Remove -PlanOnly only in the controlled evidence job.
- Certified mode additionally requires -ConfirmCertifiedMaintenance and -ConfirmCertifiedCost.
- AllowProductionImpactOverride requires separate explicit governance and is not a normal release command.
Reviewed sourcetests/conformance/Invoke-ClusterConformanceEnvironmentValidation.ps1

Admission before execution

✓
Cluster exists, is reachable, and has no conflicting conformance run.
✓
Selected Kubernetes version and Sonobuoy catalog entry are eligible.
✓
Signed runtime bundle and digest-pinned images/tools are available online, through Harbor, or in the air-gap bundle.
✓
Artifact storage capacity, retention, timeout, and maximum output are acceptable.
✓
The operation-scoped cluster-admin-equivalent credential can be created and later removed.
✓
Certified mode has explicit operator confirmation and the required impact override.

Commands for this step

Build and verify the conformance image evidence

These protected release steps pin, scan, evaluate, sign, and validate every conformance image before a customer run can consume it.

Protected workflowRun only on an approved maintainer, security, or release system.

01
Changes statePowerShell
Generate the conformance image verification plan
Resolve every catalog image and architecture that must be verified for the candidate release.
Before you run it
- PowerShell 7
- Signed candidate catalog staged inside the allowed repository root
PowerShell · release-conformance-image-plan
```
& ./scripts/new-conformance-image-verification-plan.ps1 `
  -CatalogPath ./conformance/catalog.json `
  -OutputPath ./artifacts/conformance/image-verification-plan.json `
  -AllowedRoot (Resolve-Path .).Path
```
Expected result
A deterministic plan lists every required image, digest and supported architecture.
Reviewed sourcescripts/new-conformance-image-verification-plan.ps1
02
Read-onlyPowerShell
Validate the catalog tool image inventory
Verify that the signed catalog and materialized tool bundle agree on every required executable and bounded version probe.
Before you run it
- Signed catalog
- Fully materialized conformance tool bundle
- PowerShell 7
PowerShell · release-conformance-catalog-inventory
```
& ./tests/conformance/Invoke-ConformanceCatalogImageInventoryValidation.ps1 `
  -CatalogPath ./conformance/catalog.json `
  -ToolBundleRoot ./artifacts/conformance/tool-artifact `
  -CommandTimeoutSeconds 180
```
Expected result
Every catalog entry resolves to the expected tool artifact and its bounded validation succeeds.
Reviewed sourcetests/conformance/Invoke-ConformanceCatalogImageInventoryValidation.ps1
03
Changes statePowerShell
Verify and scan one conformance image
Verify registry digest/architecture identity, generate an SBOM and evaluate vulnerabilities for one planned conformance image.
Before you run it
- PowerShell 7
- Crane
- Syft
- Grype
- Registry reachability
- Exact candidate commit and digest from the plan
PowerShell · release-conformance-image-scan
```
$recordedAt = (Get-Date).ToUniversalTime().ToString("o")
& ./scripts/invoke-conformance-image-verification.ps1 `
  -ImageReference "ghcr.io/<owner>/<image>" `
  -DigestSha256 "<64-hex-digest>" `
  -Architectures "linux/amd64,linux/arm64" `
  -CandidateCommitSha "<candidate-commit-sha>" `
  -RecordedAtUtc $recordedAt `
  -OutputDirectory ./artifacts/conformance/scans/<image> `
  -CranePath crane -SyftPath syft -GrypePath grype `
  -CommandTimeoutSeconds 600 `
  -AllowListPath ./security/cve-allowlist.json
```
Expected result
Digest and architecture checks, SBOM, raw scan and policy summary are written for the exact immutable image.
Reviewed sourcescripts/invoke-conformance-image-verification.ps1
04
Read-onlyPowerShell
Assert CVE allow-list coverage across image summaries
Reject stale or unused image-scoped allow-list entries by comparing them with all candidate scan summaries.
Before you run it
- All planned image scan summaries
- Reviewed CVE allow-list
PowerShell · release-conformance-allowlist-coverage
```
& ./scripts/assert-conformance-image-allowlist-coverage.ps1 `
  -ImageReference "ghcr.io/<owner>/<image>@sha256:<digest>" `
  -AllowListPath ./security/cve-allowlist.json `
  -SummaryPaths @(Get-ChildItem ./artifacts/conformance/scans -Filter '*.summary.json' -Recurse | ForEach-Object FullName)
```
Expected result
Every applicable allow-list entry maps to retained evidence; stale or uncovered entries fail.
Reviewed sourcescripts/assert-conformance-image-allowlist-coverage.ps1

PrivilegedPowerShell

Sign conformance image verification evidence

Aggregate planned image scan outputs into candidate-bound signed evidence.

Before you run it

Complete successful scan root
Protected release signing context
Exact candidate commit

PowerShell · release-conformance-image-evidence

$recordedAt = (Get-Date).ToUniversalTime().ToString("o")
& ./scripts/new-conformance-image-verification-evidence.ps1 `
  -CatalogPath ./conformance/catalog.json `
  -ScanRoot ./artifacts/conformance/scans `
  -CandidateCommitSha "<candidate-commit-sha>" `
  -RecordedAtUtc $recordedAt `
  -SigningPrivateKeyPath "<protected-signing-private-key>" `
  -SigningPublicKeyPath ./release-signing-public-key.pem `
  -SigningKeyId "<release-signing-key-id>" `
  -OutputDirectory ./artifacts/conformance/image-evidence `
  -AllowedRoot (Resolve-Path .).Path

Expected result

Candidate-bound evidence and detached signature are produced without exposing private-key material.

Reviewed sourcescripts/new-conformance-image-verification-evidence.ps1

06
Read-onlyPowerShell
Validate signed conformance image evidence
Independently verify evidence schema, signature, candidate commit, signing identity and allowed-root containment.
Before you run it
- Signed image evidence
- Canonical release public key
- Expected commit and key identity
PowerShell · release-conformance-image-validate
```
& ./scripts/validate-conformance-image-verification-artifact.ps1 `
  -CatalogPath ./conformance/catalog.json `
  -EvidenceRoot ./artifacts/conformance/image-evidence `
  -PublicKeyPath ./release-signing-public-key.pem `
  -ExpectedCandidateCommitSha "<candidate-commit-sha>" `
  -ExpectedSigningKeyId "<release-signing-key-id>" `
  -AllowedRoot (Resolve-Path .).Path
```
Expected result
All trust, identity, content and containment checks pass.
Reviewed sourcescripts/validate-conformance-image-verification-artifact.ps1

Run and interpret

01
Preview admission
Review mode, versions, distribution source, privilege, expected artifacts, timeout, and cleanup plan.
02
Start with idempotency
Use the approved role and idempotency key; store the run and correlation IDs.
03
Observe bounded progress
Follow Sonobuoy setup, execution, result collection, analysis, and cleanup as explicit steps.
04
Verify artifacts
Download through the authenticated API, verify integrity, and review the immutable result snapshot.
05
Confirm cleanup
Ensure temporary cluster resources and operation-scoped credentials are removed or produce a recovery task.

Commands for this step

Start a Quick conformance run

Use this API workflow after admission succeeds. The command returns a durable run that must be observed through its normal OperationRun state.

Run in orderComplete each step successfully before continuing.

01
Changes stateHTTP API
Start a Quick conformance run
Admit a pinned, policy-authorized Sonobuoy Quick run through ClusterPilot.
Before you run it
- Conformance permission and policy
- Compatible assigned agent
- Pinned catalog/tool/images
- Capacity and cleanup admission pass
HTTP API · api-conformance
```
curl --fail-with-body -sS -X POST \
  -H "Authorization: Bearer $CLUSTERPILOT_API_KEY" \
  -H "Idempotency-Key: <uuid>" \
  -H "Content-Type: application/json" \
  --data '{"mode":"quick"}' \
  $CP_URL/api/v1/clusters/<cluster-id>/conformance/runs | jq .
```
Expected result
A conformance OperationRun is created; the run uses customer-cluster execution and retains immutable results.
Reviewed sourcedocs/api/cluster-conformance.md

Result semantics and retention

Outcome	Interpretation
Passed	The selected Sonobuoy mode passed for the recorded cluster/release snapshot.
Failed	One or more tests failed; inspect plugin results and run evidence.
Incomplete	Execution, collection, analysis, or cleanup did not produce a complete result.
Cancelled	The operator requested cancellation and cleanup reached a terminal state.

Purge is a separate, authorized lifecycle action. Retain the immutable result metadata, audit decision, and any evidence required by policy before deleting large raw artifacts.

Commands for this step

Download and verify the customer-visible result

Retrieve artifacts only through the authenticated API and preserve their integrity metadata with the acceptance record.

Run in orderComplete each step successfully before continuing.

01
Read-onlyHTTP API
Download a run artifact
Download protected evidence while retaining the response checksum and audit trail.
Before you run it
- runs.read permission
- Artifact is retained and policy permits download
HTTP API · api-artifact-download
```
curl --fail-with-body -sS -L \
  -H "Authorization: Bearer $CLUSTERPILOT_API_KEY" \
  --output run-artifact.bin \
  $CP_URL/api/v1/runs/<run-id>/artifacts/<artifact-id>/download
sha256sum run-artifact.bin
```
Expected result
The downloaded bytes match the recorded artifact SHA-256.
Reviewed sourcedocs/api/run-artifacts.md

Commands for this step

Resolve protected release proof and adoption evidence

These commands belong to release engineering and independent validation. They are not required for a normal customer Quick run.

Protected workflowRun only on an approved maintainer, security, or release system.

01
Changes statePowerShell
Resolve release proof requirements
Resolve required GitHub or approved offline conformance proofs into a candidate-bound evidence index.
Before you run it
- PowerShell 7
- Authenticated GitHub CLI for online resolution or a complete approved offline export
- Requirements file
PowerShell · release-conformance-proof-resolve
```
& ./scripts/resolve-conformance-release-proof-requirements.ps1 `
  -RequirementsPath ./conformance/release-proof-requirements.json `
  -CandidateCommitSha "<candidate-commit-sha>" `
  -Repository "ClusterPilot-System/clusterpilot" `
  -OutputPath ./artifacts/conformance/resolved-proofs.json `
  -AllowedRoot (Resolve-Path .).Path

# Offline alternative: add both arguments below.
# -OfflineRunIndexPath ./offline/run-index.json -OfflineArtifactRoot ./offline/artifacts
```
Expected result
Every required proof resolves to the exact candidate or the command fails without silently accepting a gap.
Reviewed sourcescripts/resolve-conformance-release-proof-requirements.ps1
02
PrivilegedPowerShell
Generate signed conformance adoption inputs
Create the signed state-transition inputs for candidate, pre-production, supported, deprecated or blocked catalog adoption.
Before you run it
- Approved requested state
- Signed image verification evidence
- Resolved proof index
- Protected signing context
PowerShell · release-conformance-adoption-inputs
```
$recordedAt = (Get-Date).ToUniversalTime().ToString("o")
& ./scripts/new-conformance-release-adoption-inputs.ps1 `
  -CatalogPath ./conformance/catalog.json `
  -VerificationEvidencePath ./artifacts/conformance/image-evidence/evidence.json `
  -ProofsPath ./artifacts/conformance/resolved-proofs.json `
  -RequestedState preprod-validated `
  -CandidateCommitSha "<candidate-commit-sha>" `
  -ApprovalReference "<change-or-approval-reference>" `
  -RecordedAtUtc $recordedAt `
  -SigningPrivateKeyPath "<protected-signing-private-key>" `
  -SigningPublicKeyPath ./release-signing-public-key.pem `
  -SigningKeyId "<release-signing-key-id>" `
  -OutputDirectory ./artifacts/conformance/adoption `
  -AllowedRoot (Resolve-Path .).Path
```
Expected result
Signed adoption inputs bind catalog, proofs, state, candidate, approver reference and timestamp.
Stop conditions and operator notes
- RequestedState accepts candidate, preprod-validated, supported, deprecated or blocked.
- Use -PreviousCatalogPath and the dedicated proof/verification allowed roots when performing a governed transition from prior state.
Reviewed sourcescripts/new-conformance-release-adoption-inputs.ps1

Read-onlyPowerShell

Validate a conformance adoption transition

Verify catalog, image evidence and adoption signatures plus the requested support-state transition.

Before you run it

Complete signed adoption bundle
Canonical release public key
Expected state and candidate identity

PowerShell · release-conformance-adoption-validate

& ./tests/conformance/Invoke-ConformanceReleaseAdoptionValidation.ps1 `
  -CatalogPath ./conformance/catalog.json `
  -CatalogSignaturePath ./conformance/catalog.json.sig `
  -VerificationEvidencePath ./artifacts/conformance/image-evidence/evidence.json `
  -VerificationEvidenceSignaturePath ./artifacts/conformance/image-evidence/evidence.json.sig `
  -AdoptionEvidencePath ./artifacts/conformance/adoption/adoption.json `
  -AdoptionEvidenceSignaturePath ./artifacts/conformance/adoption/adoption.json.sig `
  -PublicKeyPath ./release-signing-public-key.pem `
  -RequestedState preprod-validated `
  -ExpectedCommitSha "<candidate-commit-sha>" `
  -AllowedRoot (Resolve-Path .).Path

Expected result

Every signature, identity, transition and containment check passes.

Reviewed sourcetests/conformance/Invoke-ConformanceReleaseAdoptionValidation.ps1

04
Changes statePowerShell
Resolve a reusable conformance release bundle
Select and download a governed bundle for the exact candidate and allowed support states.
Before you run it
- PowerShell 7
- Authenticated GitHub CLI
- Governed release artifact available
PowerShell · release-conformance-bundle-resolve
```
& ./scripts/resolve-conformance-release-bundle-artifact.ps1 `
  -Repository "ClusterPilot-System/clusterpilot" `
  -CandidateCommitSha "<candidate-commit-sha>" `
  -AllowedSupportStates candidate,preprod-validated,supported `
  -OutputDirectory ./artifacts/conformance/resolved-bundle `
  -RepositoryRoot (Resolve-Path .).Path
```
Expected result
Only a bundle whose evidence satisfies candidate and support-state policy is materialized.
Stop conditions and operator notes
- Add -AllowCompatibleAncestor only when the governing reuse policy explicitly permits an ancestor candidate.
Reviewed sourcescripts/resolve-conformance-release-bundle-artifact.ps1

PrivilegedPowerShell

Build the signed conformance tool artifact

Download catalog-pinned tools, verify them and produce the signed release-owned tool artifact.

Before you run it

Approved egress or mirror
Signed catalog and schema
Protected release signing context

PowerShell · release-conformance-tool-artifact

& ./scripts/new-conformance-tool-release-artifact.ps1 `
  -CatalogPath ./conformance/catalog.json `
  -CatalogSignaturePath ./conformance/catalog.json.sig `
  -CatalogSchemaPath ./schemas/v1/conformance-catalog.schema.json `
  -ReleaseSigningPublicKeyPath ./release-signing-public-key.pem `
  -ReleaseSigningPrivateKeyPath "<protected-signing-private-key>" `
  -ReleaseSigningKeyId "<release-signing-key-id>" `
  -OutputDirectory ./artifacts/conformance/tool-artifact `
  -AllowedRoot (Resolve-Path .).Path `
  -DownloadTimeoutSeconds 300

Expected result

Every tool matches its catalog digest and the resulting artifact and metadata are signed.

Reviewed sourcescripts/new-conformance-tool-release-artifact.ps1

Conformance validation

Purpose and boundary

Validation modes

Admission before execution

Run and interpret

Preview admission

Start with idempotency

Observe bounded progress

Verify artifacts

Confirm cleanup

Result semantics and retention