This document turns three field-defining questions into the first coordinated IDKMesh research program.
Two of the three tracks are finished. They are kept here because this document is a record of what the program was, not a list of what is currently claimable.
E032-population-scaling.md.
Hypothesis 3 is tested by the budget-matched coordination-topology arms in
R1_COLLECTIVE_SCALING.md, which find topology
shifts the exponent by roughly an order of magnitude less than error
correlation does. Hypothesis 2 is tested by the issue #30 help/hurt sweep
(R1_HELP_HURT_SWEEP.md) and by
E040-diversity-correlation-threshold.md,
whose joint result is about the harnesses rather than the hypothesis: the
equal-budget advantage is proportional to retained independence at every
assumed correlation short of 1.0, so a grid that does not charge for
diversity has no failure mode for hypothesis 2 and cannot supply evidence
either way. The help/hurt sweep does charge for it, and every one of its
hurts cells carries a non-zero structural-worker quality penalty —
correlation sets the size of the advantage, worker quality sets its sign.
The track stays open on the real held-out corpus, which is
issue #70.E022-verification-scaling-matrix.md.WORK_UNIT_RESEARCH_TRACK_COMPLETION.md.These tracks should not become three isolated projects. They share one experimental substrate.
A large human+AI mesh becomes useful only when three things scale together:
collective capability
/\
/ \
/ \
/ \
/ \
Work Units ---------- verification
composability trust
More workers are useful only if work can be decomposed into composable units and verification can keep up with the resulting output.
A compact hypothesis is:
VerifiedProgress = GeneratedValue * Composability * VerificationConfidence - CoordinationCost - FailureCost
This expression is a research scaffold, not a settled model. Every term should eventually have an operational definition.
Every experiment should produce a common event record so results from different orchestration strategies can be compared.
Candidate event fields:
experiment_id: string
run_id: string
work_unit_id: string
parent_work_unit_ids: []
worker_id: string
worker_type: human | ai | compute
worker_family: string
strategy: string
started_at: timestamp
finished_at: timestamp
input_context_bytes: number
output_bytes: number
compute_cost: number
human_attention_seconds: number
messages_sent: number
validation_cost: number
validator_ids: []
validation_result: pass | fail | uncertain
hidden_test_score: number
escaped_defects: number
regressions: number
security_findings: number
confidence: number
provenance: object
The schema should evolve through versioned proposals rather than silently changing.
Before trying to prove that large collectives work, establish a minimal scientific harness.
No architecture should be declared better from a single demonstration.
Each comparison should specify in advance:
The mesh needs a reference point before adding complexity.
Run representative tasks with:
Measure quality, time, compute, human attention, and verification cost.
This establishes the denominator for every later claim about collective advantage.
Hold the task representation approximately fixed and vary worker count:
N = 1, 2, 5, 10, 20, ...
Compare homogeneous and heterogeneous groups.
Primary goal: estimate marginal verified value:
MV(N+1) = VerifiedUtility(N+1) - VerifiedUtility(N)
Also measure marginal cost and communication burden.
Important: majority agreement must not be treated as ground truth. Hidden tests or independent evidence remain the primary correctness signal.
For the same workloads and generation fan-out, compare verification policies:
Track the verification queue as a first-class system variable.
A useful stability indicator is:
rho = GenerationArrivalRate * MeanVerificationCost / EffectiveVerificationCapacity
If rho >= 1 for sustained periods, unverified work should be expected to accumulate. Real implementations will be more complicated than this queueing approximation, but the approximation provides a falsifiable starting point.
Now vary how the same larger objective is decomposed.
Candidate decomposition levels:
Search for the granularity that minimizes total cost:
TotalCost = Execution + Coordination + Context + Verification + Integration + Rework
The optimal Work Unit is not necessarily the smallest one.
The flagship experiment should vary all three dimensions:
The key question becomes:
For a fixed compute and human-attention budget, which combination produces the largest amount of independently verified useful work?
This is closer to the real IDKMesh problem than optimizing any component alone.
IDKMesh should avoid optimizing raw token generation, number of commits, number of agents, or number of completed tasks.
A candidate headline metric is:
Verified Useful Work per Unit of Scarce Resource (VUWSR)
Report it separately against at least:
No single scalar should permanently replace the underlying vector of metrics.
The research program should deliberately look for conditions under which IDKMesh fails:
Negative results are project assets.
Useful independent contributions include:
Contributors do not need to solve the whole architecture. Small, independently verifiable pieces are preferred.
A useful first milestone is not “build a million-node mesh.”
It is:
Run one reproducible benchmark that compares a strong single-agent baseline with several multi-agent configurations, records full cost and verification metrics, and publishes enough evidence for another contributor to reproduce or falsify the result.
From there, scale only when measurement demonstrates what should be scaled.