IDKMesh

Run Evidence Report v0.1

Status: Experimental product-facing aggregation layer
Related: #4, #5, #16, PR #72, PR #78, schemas/result-manifest-v0.1.schema.json, schemas/verification-result-v0.1.schema.json, schemas/human-decision-record-v0.1.schema.json

Purpose

The Verified Swarm Runner needs one place where a human can inspect the evidence from a multi-attempt run without collapsing worker claims and verifier evidence into a vote.

The current canonical trust pipeline is:

WorkUnit
 -> worker attempt
 -> ResultManifest (worker self-report)
 -> independent verifier
 -> VerificationResult (evidence + recommendation)
 -> human / governance integration decision

Run Evidence Report v0.1 is a read-only aggregation view over that pipeline. It does not replace any canonical protocol object above and does not recreate the closed candidate-level Evidence Report proposal from PR #42. The independent verifier protocol is VerificationResult v0.1.

The report answers a narrower product question:

What happened to every attempt in this run, what independent evidence exists for each attempt, where do verifiers disagree or fail, and is the saved run reproducible?

Invariants

A generated report MUST preserve these boundaries:

worker success != verified correctness
verifier recommendation != integration decision
multiple recommendations != majority truth
report generation != candidate selection
replay equality != correctness

The generated report therefore has:

{
  "human_decision": {
    "status": "pending",
    "selected_attempt_id": null,
    "integration_authority": "external_human_or_governance"
  },
  "authority": {
    "canonical_state_write": false,
    "git_push": false,
    "merge": false,
    "automatic_candidate_selection": false
  }
}

A later human/integration-decision record may refer to this report, but the generated report itself cannot be mutated into acceptance authority.

Inputs

The first implementation consumes the deterministic run record emitted by:

experiments/two_attempt_orchestrator.py

That run record already retains:

• exact WorkUnit digest;
• exact orchestration config digest;
• verifier-policy digest;
• attempt order;
• worker adapter and ResultManifest identity/digest;
• independent verifier identity;
• VerificationResult semantic digest;
• VerificationResult binding to WorkUnit + ResultManifest;
• independent check status and recommendation;
• worker/result/verifier control failures;
• an explicit no-write/no-merge/no-selection authority block.

Before rendering, run_evidence_report.py validates those relationships again and fails closed on binding drift.

In particular:

VerificationResult.result_manifest_digest == summarized ResultManifest digest
VerificationResult.work_unit_digest == run WorkUnit digest

If either relationship is inconsistent, no report is produced.

Evidence states

Every attempt is mapped to exactly one report state:

Evidence state	Meaning
supported	independent verifier recommendation is accept_candidate
rejected	independent verifier recommendation is reject_candidate
inconclusive	verifier says escalate / insufficient_evidence, or no conclusive recommendation exists
worker_error	worker adapter failed before producing usable candidate evidence
result_manifest_error	candidate ResultManifest could not be collected/parsed
verification_error	candidate exists but verifier control path failed

The report never silently drops an errored attempt. One failed worker must not erase the evidence produced by a surviving peer.

Disagreement

verification_disagreement=true when the run contains more than one distinct independent verifier recommendation.

Example:

attempt A -> worker says succeeded -> verifier says accept_candidate
attempt B -> worker says succeeded -> verifier says reject_candidate

The correct product behavior is:

preserve both histories
 -> show disagreement
 -> leave integration decision pending

not:

count votes -> choose winner

This becomes more important later when verifier failures are correlated. IDKMesh already has research showing that raw verifier count is not equivalent to independent evidence.

Deterministic replay

Replay is a provenance/reproducibility check, not a correctness proof.

For a saved deterministic run record R and its replayed record R':

ReplayMatch = SHA256(canonical(R)) == SHA256(canonical(R'))

The current fixture orchestration has no timestamps or nondeterministic worker execution, so complete-record equality is expected.

When real workers are added, replay semantics will need to distinguish:

• exact deterministic replay;
• provenance-equivalent replay;
• semantically equivalent but byte-different results;
• intentionally nondeterministic model/agent attempts.

Do not weaken v0.1 silently. Introduce an explicit later replay mode/version when real-worker evidence requires it.

CLI

Self-test:

python experiments/run_evidence_report.py self-test

Generate one deterministic run plus JSON and Markdown evidence views:

python experiments/run_evidence_report.py generate \
  --config examples/orchestration/two-attempt-good-vs-bad.json \
  --run-output results/orchestration/demo.run.json \
  --report-json results/orchestration/demo.evidence.json \
  --report-markdown results/orchestration/demo.evidence.md

Re-render an existing saved run:

python experiments/run_evidence_report.py report \
  --run-record results/orchestration/demo.run.json \
  --report-json results/orchestration/demo.evidence.json \
  --report-markdown results/orchestration/demo.evidence.md

Check deterministic replay:

python experiments/run_evidence_report.py replay-check \
  --config examples/orchestration/two-attempt-good-vs-bad.json \
  --run-record results/orchestration/demo.run.json

All generated files are constrained to results/. The tool does not write canonical repository state.

Machine-readable contract

See:

schemas/run-evidence-report-v0.1.schema.json

The schema is intentionally an aggregation schema. It references digests and summaries of canonical worker/verifier evidence; it is not a second verifier protocol.

Recording the human decision

The generated report’s human_decision object is advisory data a human reads: it carries the machine-suggested selected_attempt_id and a recommendation from {accept_candidate, reject_candidate, escalate, insufficient_evidence}, and it is required to stay {"status": "pending", "selected_attempt_id": null, ...} on every generated report (see Invariants above). Nothing above records that a human actually looked at the evidence and decided.

experiments/record_human_decision.py closes that recording gap without granting the report itself, or the decision record, any new authority:

run evidence report (read-only aggregation, human_decision always pending)
 -> experiments/record_human_decision.py record
 -> idkmesh-human-decision-record (schemas/human-decision-record-v0.1.schema.json)

The recorded decision:

• names the exact evidence report it decides on by kind + schema_version + run_id + a canonical_digest of the full report document, not by filename – a decision record verified against a swapped, edited, or regenerated report at that same path fails closed;
• requires an explicit human choice from {accept, reject, escalate}, a distinct vocabulary from the report’s own recommendation values, so a decision record can never be produced by simply echoing the machine-suggested recommendation;
• carries a free-text rationale and a decider identity, and is timestamped;
• is written as a new companion file under results/; it never mutates the evidence report it decides on.

CLI:

python experiments/record_human_decision.py record \
  --evidence-report results/orchestration/demo.evidence.json \
  --output results/orchestration/demo.decision.json \
  --decision-id orchestration/demo.decision.001 \
  --decision accept \
  --rationale "attempt-001 passed independent verification with all required checks." \
  --decider-id you@example.com \
  --selected-attempt-id attempt-001

python experiments/record_human_decision.py verify \
  --decision-record results/orchestration/demo.decision.json \
  --evidence-report results/orchestration/demo.evidence.json

python experiments/record_human_decision.py self-test

record refuses (non-zero exit) a malformed/nonexistent evidence report, a decision value outside {accept, reject, escalate}, or a --selected-attempt-id that does not appear in the referenced report’s attempts. verify re-derives the report’s digest and fails closed the moment it no longer matches the digest recorded in the decision.

Like the report itself, a decision record carries authority: {canonical_state_write: false, git_push: false, merge: false}: it documents that a decision was made, it does not execute it. Acting on the decision (a merge, a push, a state change) remains a separate, out-of-band step.

Self-test coverage

The built-in self-test checks that:

1. known-good and known-bad attempts remain one support + one rejection;
2. support/reject disagreement is visible;
3. the human integration decision remains pending;
4. automatic candidate selection remains disabled;
5. a peer worker failure remains visible while surviving evidence is retained;
6. replay of the same deterministic config matches the complete saved-run digest;
7. a tampered saved run fails replay equality;
8. ResultManifest/VerificationResult digest-binding drift is rejected before report generation.

What this unlocks

This closes one control-plane gap in #16 without pretending the real worker path is complete.

The remaining critical path is still:

canonical local node (#34)
 -> controlled Docker acceptance (#37)
 -> repository-candidate verification (#5 B1 / patch verifier work)
 -> plug real node adapter into the already-landed orchestration kernel
 -> use this evidence/replay surface over real attempts
 -> add one trivial heterogeneous real adapter

Non-goals

• no automatic ranking/winner selection;
• no merge or push authority;
• no new candidate-level verification protocol;
• no claim that replay implies correctness;
• no execution of candidate code;
• no replacement for VerificationResult v0.1;
• no majority-vote shortcut;
• no distributed scheduler.

This site is open source. Improve this page.