idkmesh

IDKMesh continuation — independent review evidence validation

Date: 2026-08-28

User direction

Continue development of MSKazemi/idkmesh from the current repository state and preserve the substantive output in the public repository.

Repository state observed

• PR #196, which introduced the independent IDKGraph P1 review protocol and JSON session template, is merged.
• Issue #152 remains open because the repository correctly lacks an eligible independent human review with honest active-review minutes.
• Growth Seed #167 already exposes that human review as a good first issue / help wanted contribution, so a duplicate recruitment issue should not be created.
• Public branch metadata still reports main as unprotected. This continuation therefore adds no stronger integration, repair, approval, or merge authority.

Bounded next step chosen

Make a future human review submission machine-checkable without machine-classifying the cohort.

The implementation adds tools/idkgraph_review_session.py, a deterministic validator and descriptive scorer for completed reviewer-supplied JSON sessions. It freezes the cohort identity/order and validates:

• experiment/schema identifiers;
• public seed and frozen source revision;
• exactly 15 ranked paths in frozen order;
• reviewer public identifier, independence statement, and anchoring disclosure;
• finite positive active-review minutes;
• allowed labels and finite confidence in [0, 1];
• explicit evidence notes and change recommendations.

For valid human-entered evidence, it computes only:

• exact-label agreement with the frozen PR #166 reference labels;
• action/no-immediate-action/unresolved confusion matrix;
• label counts;
• active minutes per candidate;
• recommended-change count and minutes per recommended change;
• explicit disagreement records.

It does not inspect document content to produce labels, fill missing judgments, infer reviewer time, repair repository structure, or authorize integration.

Verification surface

tests/test_idkgraph_review_session.py covers:

• complete valid sessions;
• preserved disagreement and uncertainty;
• the checked-in placeholder template failing as incomplete evidence;
• frozen cohort order/revision enforcement;
• non-finite/out-of-range measurement rejection;
• explicit other labeling;
• non-mutation of reviewer evidence.

The existing .github/workflows/idkgraph-observatory.yml is extended to include the new module/test/template in its canonical IDKGraph CI surface rather than introducing another workflow.

Research boundary

Agreement is descriptive evidence, not a correctness target. One 15-item cohort cannot estimate global warning-detector precision. Human attention must be reported by the reviewer rather than inferred from GitHub timestamps or generated by an AI-assisted process.

Next evidence gate

After this validator is independently integrated and one eligible reviewer completes Growth Seed #167, validate that session and publish the descriptive metrics as a new audit artifact. Preserve disagreement rather than modifying the original cohort audit to make labels converge.

This site is open source. Improve this page.