Skip to main content

Multi-Agent Delivery Verification

Use when agents implement, review, and gate changes. Coordinate isolated implementation workers, opposite-family reviewers, and a final verification owner without accepting self-reported success as evidence.

Core rule​

Agent summaries and exit codes are routing signals, not proof. A deliverable is complete only when the orchestrator inspects the artifact and reruns verification after the latest mutation.

Workflow​

  1. Establish dependency gates

    • Identify the canonical task registry and prerequisites.
    • Start only slices whose dependencies are merged and green.
    • Make review dependencies verdict-aware: a review card becoming done is not an approval signal. Release final verification only after reading an explicit APPROVE bound to the current immutable head. On REQUEST CHANGES, keep final verification gated, create a bounded remediation writer and fresh exact-head review, and link that fresh review as a prerequisite before anything dispatches.
    • Count a writer only after verifying route, worktree, live process state, non-overlapping ownership, and a fresh task-scoped tracked mutation. Reading/discovery, stale labels, wrapper state, and self-reported intent do not satisfy a writer floor.
    • Give every writer an isolated branch/worktree.
    • Before manually dispatching into a worktree governed by a periodic supervisor, acquire/register its sole-writer lease or pause/exclude that worktree in the supervisor. A process snapshot is not sufficient: a timer can launch a second writer after the snapshot. Recheck ownership immediately after dispatch and before long gate or packaging commands; if overlap occurred, stop both mutation paths and inspect timestamps plus the live diff before continuing.
    • Assign one owner to each migration chain or central contract.

  2. Constrain the implementation brief

    • State exact scope, acceptance criteria, required RED/GREEN evidence, and forbidden side effects.
    • Require exact files changed, commands executed, outputs, residual risks, and no commit/push until review.
    • Bound dependency-source archaeology; after a few focused probes, require a small adapter, a concrete blocker, or implementation progress.

  3. Inspect the handoff

    • Re-read the authoritative live issue/spec acceptance criteria before declaring a handoff or launching review. Build an acceptance matrix across the whole issue: a green bounded slice is only partial progress when a paired boundary, documentation update, platform, or negative path remains. Preserve the slice and immediately route the missing criteria; do not stage or describe the issue as review-ready.
    • Read git status, changed-file scope, and the actual diff.
    • For manifest-backed or generated fixture corpora, prove the payloads are actually tracked. git status and ordinary git add <directory> can silently omit ignored binaries. Compare manifest entries to filesystem files and git ls-tree, run git check-ignore -v on every payload, verify byte length and digest, then force-add only intentional ignored artifacts. Rerun the exact candidate tests after the tracked tree contains the payloads. See references/manifest-backed-fixture-corpora.md.
    • Run git diff --check <base>..HEAD against the complete candidate, not only the unstaged worktree. Normalize newly added text assets (including licenses) to LF and remove trailing spaces without altering their substantive text.
    • For HTML-to-PDF work, audit each rendering context separately. Main-document resources may support constrained local file: URLs, while Chromium native header/footer templates cannot load external resources and therefore need validated inline raster-image data URIs prepared by the integration layer. Add negative tests proving native fragments reject file:, network, script, HTML, SVG, malformed, and CSS-breaking inputs. See references/html-pdf-template-boundaries.md.
    • Remove bookkeeping, probes, generated scratch output, and unrelated edits.
    • When a writer is interrupted or exhausts its turn budget, first prove no writer remains alive, then inspect the worktree as unknown state. A wrapper exit or completion notification is not proof that its spawned agent died: reconcile the tracked-process registry with the OS process tree, inspect any surviving process's working directory and stdout/stderr destinations, and adopt the existing same-task worker instead of launching a duplicate. Preserve coherent RED tests and task-scoped edits; resume the same session with an edit-first prompt naming only the remaining invariant. Reinspect after every continuation rather than equating error_max_turns with either failure or completion. See references/wrapper-exit-process-survival.md.
    • Treat internal agent readiness/review-policy failures as incomplete even if useful code exists.
    • Audit the invariant behind new guards and allowlists. A green drift test is invalid if it merely records known-bad production values as exceptions while customer-facing output still presents them as authoritative.
    • For golden fixtures that disagree with canonical domain math, do not force production logic to match contradictory expected values. Establish the authoritative calculation first, correct the fixture and conflicting prose, and keep the fixture gate exact afterward. Build fixture inputs independently from expected JSON, preserve rich representative content needed by downstream renderers, and review runtime types, JSON schemas, loader validation, fixture prose, and customer-facing causation as one contract. See references/semantic-golden-fixture-reconciliation.md.
    • For source remapping, compare evaluator behavior to the exact authoritative source meaning—not titles alone—and fail closed on unmapped or partially verified entries. Never bless a historical identifier that now names a different current control.
    • For benchmark/framework completeness claims, reconcile the versioned source universe, runtime rule inventory, mapping ledgers, manifests, imported evidence, issue/PR state, worktrees, active processes, and Kanban separately. Count only rules belonging to the claimed catalog—never subtract unrelated rule packs from its denominator. A structurally valid partial ledger, a dirty worktree, or an importer does not make controls available in the product. See references/benchmark-control-completeness-reconciliation.md.

  4. Run skill-bound, opposite-family review

    • Before inspecting the change, the reviewer loads the core code-quality and security-hardening skills plus the scope-specific workflow (pre-commit, GitHub PR, or whole-codebase review). Named reviewer/final-verifier role assignments persist across runs; do not silently collapse them into one self-reviewing agent.
    • Use a reviewer from a different model/agent family for security-sensitive changes. Two differently named roles from the same model family do not satisfy this independence gate.
    • Ask for an explicit APPROVE or REQUEST CHANGES verdict with severity-ranked file:line findings.
    • If a broad reviewer is OOM-killed or times out, retry with a bounded scope: changed files plus focused tests. The final owner runs heavy gates.
    • A reviewer outage, missing skill load, or unparseable verdict is never approval.
    • An abnormal process exit can still leave a complete actionable rejection report. Read the saved report before retrying, verify exact candidate/clean checkout and surviving process ownership, and route concrete REQUEST CHANGES findings to remediation rather than blindly repeating the review. Record process health separately from verdict completeness; do not infer the crash cause from a terminal diagnostic or turn an abnormal exit into approval.
    • An exit-zero reviewer process with an empty report is also no verdict, even when its session shows completed tools and clean final identity checks. Recover the exact persisted session, inspect its last messages, and resume that same session with a bounded “emit the report from evidence already gathered; do not rerun or mutate” prompt. Accept the continuation only when it produces a readable verdict bound to the unchanged head; otherwise keep the gate pending. If that continuation is killed because the persisted session is oversized, do not keep resuming it: launch one fresh, bounded exact-head review that uses targeted reads and summarizes command output. The earlier tool transcript remains routing context, not inherited verdict evidence.
    • Make repository identity the reviewer's first executable gate: record the absolute working directory, observed HEAD, branch/detached state, and tracked status before inspecting or testing. Workspace-aware runners may restore a canonical checkout despite being launched from another linked worktree; an exit-zero review of the wrong branch is not candidate evidence. Retry from an isolated local clone at the exact commit, force an explicit workdir for every command, and verify identity again after the gates. See references/exact-candidate-review-isolation.md.
    • The final verification owner independently loads the applicable review/security skills, inspects the current commit and diff, resolves all blockers, and reruns evidence after the latest mutation; it does not inherit the reviewer's approval as proof.

  5. Resolve findings with TDD

    • Review handoff authority independently from schema validity: a digest over classifier-supplied paths does not prove those paths belong to the candidate. Require Git-backed scope evidence and candidate-bound gate verification; preserve dirty work without sending an old HEAD to review. At external verification boundaries, inject ordinary I/O, timeout, and unexpected exceptions and prove the affected handoff fails closed without starving unrelated work. See references/handoff-sealing-authority-review.md for negative probes and the distinction between established findings and still-unverified fixes.
    • For recovery-authority corrections, review the invariant across both live writers and persisted replay, not just the reviewer's example. Inject crashes between durable transitions; invalid live calls must leave bytes unchanged and invoke no callbacks, while valid historical interrupted states remain readable and resumable through the public API. Recent ownership evidence is not necessarily causally valid: bind its observation time to the authority it supports. See references/recovery-authority-causal-review.md for the matrix and evidence limits.
    • Add a regression test for every required finding and watch it fail for the expected reason before changing production code.
    • Apply the smallest fix, rerun focused tests immediately, then rerun all applicable gates after the latest mutation.
    • For streaming/concurrency code, cover normal completion, bodyless response, cancellation, source/reader error, forwarding failure, lease expiry, crash recovery, and overlapping terminal paths. Cleanup must be awaited when suspension could strand state and must be exactly-once.
    • For durable action journals, test the whole attempt lifecycle across processes and historical schema versions—not only immediate retry denial. Fresh readback must not admit a second same-key attempt while the first is active; recovery must preserve unknown old-version attempts until fenced reconciliation. For release evaluators, reconcile platform requirements against structured artifact identities and test delimiter collisions. See references/durable-state-and-release-contract-review.md.
    • For hard process deadlines, do not unref() the timer that guarantees a required terminal exit: unresolved Promises do not retain the Node event loop, so an unreferenced deadline can permit an early natural zero exit. Bound primary work and cleanup separately, and test both a no-extra-handles child process and the real native callback adapter. Callback errors and synchronous native throws must become handled failures without double close, double exit, or unhandled rejection. See references/process-liveness-and-native-cleanup.md.
    • For lifecycle/file races, persist the concrete artifact path before the first write and retain that reservation when cleanup fails so retention can converge.
    • Make lifecycle regressions reproduce the actor and failure window literally. If the contract says “the writer created an artifact and failed before reservation,” create the bytes inside the writer callback before it returns or throws—not in test setup—and separately prove no database/registration handle opened, the filesystem artifact was cleaned, and no durable row remains. A pre-created fixture can make correct cleanup code look covered while missing the actual boundary.
    • A post-check directory race cannot use regular-file hard-link restoration. If a foreign directory is atomically claimed, persist its exact quarantine path in the existing durable cleanup row before returning/throwing; never delete the association or leave the directory discoverable only by a random filename. Test replacement after the precheck and before the claim.
    • A lock around singleton database selection/switching is insufficient. Hold one reentrant-safe operation lease from project selection through complete SQL query/write completion, audit every connection consumer, and test a real held query against a cross-project purge plus success/error release.
    • Security mutation tests must restore production files in finally/a shell trap, compare the restored bytes or digest, and rerun GREEN after restoration. If a worker exits mid-mutation, treat the worktree as unknown state and inspect the live diff before any other action.
    • For CI supply-chain findings, never turn an unknown action commit into a permanent allowlist or call an unsigned digest manifest an attestation. Resolve immutable action SHAs from the publisher's official Git reference/release, record the version-to-SHA binding, pin every third-party uses: to the full reviewed SHA with no exception path, and use the platform's native OIDC/Sigstore attestation action with only its required job-scoped permissions. If official identity material cannot be verified, keep the candidate blocked rather than weakening the checker. See references/ci-supply-chain-handoffs.md.
    • Reject check-then-act filesystem corrections when the final operation can overwrite or unlink: lstat/exists followed by rename is still racy. Prefer an atomic no-overwrite primitive, inject a competitor immediately after any pre-check, and verify regular-file, symlink, and hard-link/path-swap variants required by the finding. A focused GREEN test for the originally cited interleaving does not close the finding if source inspection reveals a narrower residual window.
    • For async UI reused across resource identities, guard every post-await continuation with immutable request identity plus a current token; identity changes must invalidate old request slots without allowing late finally blocks to clear new loading state.
    • For newline-bearing secrets such as inline PEM bundles, verify the UI control itself preserves internal newlines through blur/state handling and into the exact IPC/request payload. Single-line password inputs sanitize line breaks, and a one-line BEGIN marker test is insufficient. Use a multiline-capable masked control where the product contract advertises inline content; keep one bounded credential-size constant across all entry routes, and never narrow the advertised contract merely to make a test pass. Use a cryptographically valid, matching test certificate/key pair; validate the pair, mock the external credential constructor beneath the real resolver, assert exact transport, and prefer opaque DER/PKCS#8 fixtures when literal PEM would leak into scanners or agent transcripts.
    • For privileged local-file imports, removing a path from the typed request is not a runtime boundary. Reject smuggled own or inherited path fields before the native dialog and every read/import/parser/store side effect; consume only the dialog-returned path, treat cancellation as terminal, and keep content validation separate from extension filters. Test absolute, traversal, UNC, file://, and inherited-property payloads with zero-side-effect assertions. See references/privileged-file-dialog-boundaries.md.
    • See references/concurrency-and-artifact-race-review.md for deterministic artifact/purge and stale-UI interleaving matrices.

  6. Reconcile dependencies without creating a moving-base gate

    • Update from current main after prerequisite or materially overlapping PRs merge, and before first delivery when required by repository policy.
    • Once a PR is open, bind review, verification, and CI validity to its exact head SHA. Unrelated movement of the base branch does not invalidate head-bound evidence and does not by itself require a rebase/merge-main cycle.
    • Reconcile again only when the PR head changes, GitHub reports conflict/non-mergeability, branch protection explicitly requires an update, or an overlapping base change materially affects a touched contract.
    • Rerun applicable tests after the latest PR-head mutation; earlier-head evidence is stale.
    • Verify clean install, unit/integration tests, lint, formatting, coverage, audits/signatures, type checks, builds, Worker smoke, generated types/config, secret scan, and diff checks as applicable.

  7. Protect stateful test isolation

    • Never run concurrent worktrees against the same mutable database/container namespace.
    • Serialize stateful integration gates, or provision unique project IDs, ports, volumes, and container names per worktree.
    • If interference occurs, stop treating the failure as product evidence, let the owning task release the environment, reset from migrations, and rerun cleanly.

  8. Ship through PR gates

    • Commit only reviewed scope.
    • Open a PR whose body records review fixes, exact verification, rollback/residual risks, and dependency context.
    • Monitor CI with the least-privileged accessible API. If a token cannot read Checks annotations, poll Actions workflow/run/job REST endpoints instead of interpreting the 403 as workflow failure.
    • Query Actions by the PR's exact live head SHA; branch-wide or repository-wide run listings can mix unrelated executions and are not exact-head evidence.
    • Treat the legacy combined commit-status endpoint carefully: state: pending with total_count: 0 means no legacy statuses were published, not that GitHub Actions is still running. Determine Actions CI from exact-head workflow runs and their jobs; require every applicable run/job to be completed successfully.
    • Before declaring readiness, re-read live state, merged, draft, head/base SHAs, and mergeability in the same run. Match every required reviewer and final-verifier artifact to that head SHA. An older-head approval, vanished reviewer, missing verdict, or unparseable output keeps the PR pending.
    • If a user says a newly announced PR is missing, query the exact pull endpoint or an all-state listing before retrying creation; an open-only list can hide a PR merged or closed between turns. For a merged result, report the direct Closed/Merged link plus merged_at, merged_by, source head, and merge commit, then verify the requested file/content on the current default branch and applicable post-merge CI. Never recreate, reopen, or append to the finished PR.
    • A durable local review report can satisfy an internal review gate only after its exact head and complete verdict are read back; an empty GitHub reviews list neither proves nor disproves that internal verdict. State clearly whether approval is persisted on GitHub or only in a verified internal artifact.
    • When review evidence must be auditable by repository users, post each verdict visibly on the PR as soon as it is verified. Include reviewer route/identity, exact head SHA, explicit APPROVE or REQUEST CHANGES/HOLD, commands and counts, findings, and residual gates. Read the comment back before claiming it exists. After any head mutation, treat earlier comments as historical evidence—not current approval—and post fresh exact-head Terra/reviewer evidence before launching or recording final verification.
    • Do not treat a PR description's claim that an agent approved as the review artifact by itself. Recomputing a recorded diff digest proves only that the current content matches the described content; it does not prove reviewer identity, independence, or that the review occurred. Require a readable persisted verdict (GitHub review, durable local report, signed receipt, or equivalent) that identifies reviewer, exact head or base-plus-diff digest, and explicit verdict. If only the PR description survives, report the claimed verification separately and keep the review gate pending.
    • Merge only after every required repository workflow and non-waived human/role gate is green, then advance dependent work.
    • Honor an explicit owner merge override. Internal agent gates are governance defaults, not a veto over an authorized repository owner's direct instruction to merge. Re-read live PR state/head/mergeability and exact-head CI in the same turn; preserve repository branch protections; use the observed head SHA as an optimistic merge lease. Record which internal gates were waived, split unrelated findings into their own issues/current-main branches, and do not keep enlarging the PR.
    • Treat an interrupted merge command as an unknown side effect, not a failed merge. Read the PR back before retrying; if it already merged, verify the resulting default-branch content and continue post-merge reconciliation instead of issuing a second merge.
    • Finish post-merge reconciliation before reporting completion. Read back the merged PR and live default-branch ref; verify landed content rather than assuming a squash source SHA is an ancestor. Resolve the closing issue, remove stale claimable/ready labels, terminalize the human-action or delivery card with merge and CI receipts, and read those targets back. Follow the exact merge commit's post-merge Actions run through completion; if it is still running, say so rather than calling the delivery fully verified. Trigger or confirm the durable supervisor/dispatcher only after terminal state is recorded, and never infer a successful trigger from a timed-out command or stale execution row. See references/post-merge-reconciliation.md.
    • See references/owner-merge-override-and-moving-base-convergence.md for the bounded override, merge-readback, and unrelated-finding split workflow.

  9. Surface human action explicitly

    • Do not bury ready PRs inside a general fleet status. Put each under “Action required — ready for your review and merge” with a direct link, exact head, green-gate summary, and concise reason it is ready.
    • List non-ready PRs separately with do not merge yet and the single remaining gate or blocker.
    • Assess multiple PRs independently: one pending PR must not hide another that is ready for human action.
    • Separate human-only evidence from agent-owned integration/review work. Prepare the correct integrated candidate before asking the user to test, give a bounded privacy-safe checklist and copyable attestation, and make any waiver explicit and risk-scoped. See references/human-acceptance-gate-handoffs.md.

Urgent named-work recovery​

When the user explicitly says “run it now” or asks for a named reviewer NOW, perform bounded candidate/ownership discovery and actual supported dispatch in the same turn; do not stop at a proposed handoff. Preserve an already-live exact worker rather than spawning a duplicate. Report the concrete blocker and active phase, not a replay of discovery.

Before remediating old PR review comments, compare the remote head with newer immutable local candidates and complete exact-candidate verdicts. An approved local correction awaiting integration/publication is not missing implementation. Confirm material integration conflicts against the verified remote main SHA, preserve accepted fixes, and scope a distinct reconciliation task rather than rerunning completed fixes. Conflict markers demonstrate an attempted merge, not successful reconciliation.

An explicitly requested immediate final-verifier assessment may run as a read-only diagnostic alongside the primary reviewer in separate immutable worktrees. Label it diagnostic/acceptance assessment, not final delivery approval before its prerequisite. Preserve ordered final signoff and distinguish slice acceptance from broader live issue criteria.

Verification checklist​

  • Dependencies and materially overlapping base changes reconciled; no moving-base refresh was triggered merely because main advanced
  • Actual diff inspected; scratch artifacts removed
  • Catalog completeness uses a pinned source/version and catalog-specific numerator; manifests, mappings, runtime rules, automation, and imported evidence are reported as distinct delivery states
  • New guards/allowlists prove the customer-facing semantic invariant and fail closed on exceptions
  • Opposite-family verdict obtained and findings resolved, or the authorized owner explicitly waived that internal gate
  • Focused regression tests pass after latest fix
  • Full applicable gates pass after the latest PR-head mutation
  • Stateful integration environment was isolated or serialized
  • Generated files/config and lockfile scope verified
  • Live PR state and exact head re-read immediately before readiness or merge action
  • Every non-waived review/final-verification artifact matches that exact head
  • Exact-head PR CI green before merge
  • After merge or an interrupted merge call, PR state and default-branch content were read back
  • Closing issue state/labels and the delivery or human-action card were reconciled and read back
  • Exact merge-commit post-merge CI reached a terminal result, or its still-running state was reported explicitly
  • Durable supervisor/dispatcher continuation was confirmed without treating a timeout or stale row as success
  • Human-ready PRs surfaced in a distinct action-required section with direct links

References​

  • See references/verdict-aware-review-dags.md for verdict-gated review transitions, writer-floor receipts, and safe continuation of interrupted coding sessions.
  • See references/security-sensitive-handoffs.md for concrete review prompts, cleanup-path cases, and CI monitoring patterns.
  • See references/html-pdf-template-boundaries.md for Chromium document versus native header/footer asset rules, context-specific URI validation, and text-asset diff hygiene.
  • See references/symlink-quarantine-restoration.md for portable symlink rollback, post-claim failure propagation, durable quarantine discoverability, and deterministic Darwin regression coverage.
  • See references/fail-closed-drift-guards.md when validating source remaps, exception lists, and customer-visible identifiers.
  • See references/benchmark-control-completeness-reconciliation.md when auditing whether every control in one or more benchmark/framework catalogs is actually available.
  • See references/semantic-golden-fixture-reconciliation.md when a required fixture conflicts with canonical calculations, schemas, or derivable inputs.
  • See references/concurrency-and-artifact-race-review.md for durable pre-write artifact reservation and deterministic async-identity race tests.
  • See references/interrupted-writer-continuation.md for safely preserving RED tests and resuming a timed-out or turn-exhausted coding-agent session.
  • See references/wrapper-exit-process-survival.md when a wrapper exits but its spawned coding agent may still own the worktree.
  • See references/process-liveness-and-native-cleanup.md for hard timeout, event-loop retention, and real native callback cleanup verification.
  • See references/multiline-secret-input-transport.md for newline-preserving masked controls, bounded credential transport, valid opaque crypto fixtures, and behavior-bearing PEM regressions.
  • See references/privileged-file-dialog-boundaries.md for renderer-to-main import boundaries, runtime extra-field rejection, native-dialog ordering, and hostile-path zero-side-effect tests.
  • See references/remote-branch-retirement.md before deleting merged or apparently stale remote branches in a repository with parallel agents, worktrees, or recovery state.
  • See references/post-merge-reconciliation.md for API-first merge/content verification, issue-label cleanup, board terminalization, exact-merge CI, and safe supervisor continuation.
  • See references/subscription-backed-agent-invocation.md when a user requires proof that a coding worker used their authenticated subscription/team account rather than API-key or alternate-cloud billing.
  • See references/human-acceptance-gate-handoffs.md for separating human-only validation from agent-owned work, preparing the correct integrated candidate, privacy-safe attestation templates, and scoped waivers.
  • See references/manifest-backed-fixture-corpora.md for proving ignored/binary fixture payloads are tracked, hash-bound, encrypted as claimed, and exercised through production migration paths.

Supporting files: this skill's supporting files are held in the docsite at docs/15-skills/_support/engineering/multi-agent-delivery-verification/ — fetch them fresh from jknash/docsite main alongside this page. Source: jknash/hermes-shared-skills · branch hermes-jkdev001 @ 1d0d545c3970 · skills/engineering/multi-agent-delivery-verification/ · view source · Imported 2026-10-03. Supporting files (references, scripts) remain in the source repository.

Published by Muse · 2026-10-03.