Date: 2026-09-23
Status: product/research proposal grounded in a current competitor scan
Scope: identify high-value capabilities that are not merely “more agents”, “better prompts”, generic evaluation, tracing, or another connector layer.
This document does not claim that no research prototype, startup, or private system anywhere has explored these ideas.
The narrower and defensible claim is:
In a 2026-09-23 review of the mainstream public product surfaces of Goose, Google ADK, LangGraph/LangSmith, CrewAI, OpenHands, Microsoft Agent Framework, OpenAI Agents SDK, and adjacent agent products, the features below were not found as first-class end-to-end product capabilities in the combined form proposed here.
That wording matters. Independent verifier agents, verification debt, uncertainty propagation, delayed-feedback routing, and correlated LLM-judge research all exist elsewhere. IDKMesh should therefore innovate at the systems combination and operating-policy level, not market a commodity idea as globally unique.
The competitor scan found strong coverage of:
Examples:
A useful warning from the wider ecosystem is that independent verification itself is no longer sufficient differentiation. New systems and practices explicitly advertise separate verifiers. IDKMesh should move beyond “worker A + critic B”.
The repository already has an important in-flight research branch that overlaps several ideas below:
AVE already proposes and synthetically studies:
Therefore this product document must not create a second competing control-policy architecture.
The useful new product slice is narrower:
Convert AVE’s qualitative/correlation-penalty idea into an explicit marginal independent-evidence operator: given the evidence already selected for one candidate, estimate how much effective evidence each additional verifier is expected to add, and allow the router to reject a redundant verifier even when that verifier is individually strong.
This can become a reusable primitive beneath AVE, the Connector Control Plane, and the GUI. It should be tested against simple family-diversity heuristics rather than assumed better.
Route the next worker or verifier by marginal independent evidence, not by nominal agent count or standalone accuracy.
Most routers ask:
IDKMesh should additionally ask:
Given the evidence already collected for this exact task, which available worker/verifier is most likely to contribute information we do not already have?
Each candidate worker/verifier has an evidence-lineage fingerprint:
Select the next evidence producer using a quantity such as:
Expected marginal trust gain
---------------------------------------------
compute cost + latency + human attention + risk
The “trust gain” should not be raw agreement.
It can initially be approximated by:
The router should be allowed to say:
“Do not call another agent. The available choices are too correlated with evidence already collected.”
This is strategically important. Most orchestration products are rewarded for executing more agents. IDKMesh should sometimes produce less agent activity because additional activity has near-zero evidence value.
A 2026 paper, Nine Judges, Two Effective Votes, directly shows that nominally diverse LLM judges can collapse to a much smaller effective jury because of correlated errors. Research on correlation-aware LLM routing also exists.
The product gap is turning those observations into a live orchestration policy across workers, verifiers, tools, and human review.
Relevant external work:
Relevant IDKMesh foundation:
idkmesh gate-audit;RoutingDecision.Continuously inject hidden known-bad candidates into the verification stream so the system measures whether its reviewers are still capable of detecting failure.
Evaluation suites are usually run periodically. A verifier can drift, a prompt can change, a model can update, or multiple reviewers can silently converge on the same blind spot between test runs.
IDKMesh can act more like an immune system.
Maintain a rotating bank of controlled probes:
Some probes should be:
Verifier performance is updated from probe outcomes.
If a reviewer or panel misses probes:
probe miss
-> reliability downgrade
-> increased verification requirement
-> route to a more independent verifier
-> optional human escalation
-> potentially halt generation if proof capacity is compromised
Mainstream frameworks have evaluation datasets and online evaluators. The proposed difference is making hidden, rotating known-bad probes part of the runtime trust protocol, directly controlling routing and acceptance policy.
The current gate-audit already supports known-bad probe candidates. This feature turns that diagnostic idea into continuous operational calibration.
SentinelProbe contract.When proof capacity is saturated, slow down or stop generation.
Agent systems increasingly make generation cheap while review remains scarce. Many products expose budgets, concurrency limits, queues, or human approvals, but the proposed IDKMesh controller should tie generation admission directly to risk-weighted unverified work.
Track:
Examples:
low-risk + deterministic proof available
-> continue
high-risk + independent verifier available
-> continue with stronger evidence requirement
high-risk + only correlated reviewers available
-> hold generation
review queue above risk-weighted WIP limit
-> throttle new worker dispatches
sentinel catch rate degraded
-> reduce autonomy tier
This turns “human attention is scarce” from a dashboard observation into a control-system input.
The repository already contains verification-debt and backpressure research. The innovation is to make it the runtime admission controller for heterogeneous coding-agent generation.
VerificationDebtSnapshot.admit, throttle, hold, human_required.Do not call evidence “independent” merely because it came from different agent names.
Two agents can have different labels while sharing:
IDKMesh should maintain a graph of evidence lineage and common-cause exposure.
candidate-42
|
+-- worker: goose
| +-- model-family: X
| +-- recipe: security-review-v3
| +-- retrieval-set: R17
|
+-- verifier-A
| +-- model-family: X
| +-- test-suite: T5
|
+-- verifier-B
+-- model-family: Y
+-- test-suite: T5
A and B are model-diverse but share the same test blind spot. The graph should make that visible.
Report:
Important: structural overlap is a risk signal, not proof of statistical dependence.
Extend provenance records with bounded lineage fields and render a non-authoritative common-cause report.
Verification should choose the next test that is most likely to disprove the candidate, not simply run a fixed checklist.
For a candidate change, identify competing failure hypotheses:
H1: happy-path works, permissions broken
H2: permissions work, migration rollback broken
H3: functional behavior works, provenance is invalid
H4: tests pass only because the fixture misses a boundary case
Then choose the next verifier/test by expected ability to distinguish those hypotheses.
This is active experiment design applied to software verification.
expected reduction in residual risk
------------------------------------
verification cost + latency + reviewer attention
The planner can stop when:
Frameworks commonly allow custom evaluators. This proposal makes evaluation selection itself adaptive and falsification-driven.
Start with deterministic test families and known failure classes rather than LLM-generated proof plans.
A verifier should be judged by what happens after its approval, not only by agreement with a benchmark.
Most evaluation systems stop near deployment. IDKMesh should connect later outcomes back to the exact worker/verifier/routing decision.
Outcome signals can include:
For each task class:
worker produced candidate
verifier recommended accept
human merged
later defect observed
|
v
update worker + verifier calibration
|
v
future routing changes
Reliability should decay over time and after model/prompt/runtime changes.
Delayed-feedback routing exists in research and routing systems. The proposed IDKMesh feature ties delayed outcomes to software integration authority and verifier calibration, not merely model selection.
Use GitHub-native signals first: revert commits, reopened linked issues, failed post-merge workflows, and maintainer-labeled regressions.
Spend scarce human review minutes where they change the decision most.
Instead of sending every ambiguous candidate to a human, estimate where human review has the highest expected value:
The scheduler should prefer a 10-minute human review that resolves a high-risk uncertainty over a 40-minute review of a candidate already covered by strong independent deterministic evidence.
This is a longer-term research feature, not a near-term autonomous policy.
Adaptive Evidence Portfolio
Why:
Sentinel Blind-Spot Probes
Why:
Verification Backpressure Controller
Why:
Evidence Lineage and Common-Cause Graph
Why:
Counterfactual Proof Planner
Post-Integration Truth Loop
Human Attention Value-of-Information Scheduler
These become substantially more useful once real worker/verifier outcome histories exist.
The strongest concise positioning created by these features is:
Other platforms orchestrate agents. IDKMesh should orchestrate evidence.
A more technical version:
IDKMesh chooses workers and verifiers by the marginal trust they add, continuously probes for shared blind spots, applies backpressure when proof capacity is saturated, and learns from post-integration outcomes.
That is more defensible than:
“IDKMesh has multiple agents and an independent verifier.”
task / issue
|
v
risk + uncertainty
|
v
Adaptive Evidence Portfolio
/ \
/ \
worker(s) verifier set
\ /
\ /
candidate evidence
|
common-cause graph
|
sentinel probes
|
counterfactual proof planner
|
residual-risk estimate
|
+----------------+----------------+
| |
v v
enough evidence insufficient proof
| |
v v
human/governance decision throttle / reroute / escalate
|
v
integration
|
v
post-integration outcome
|
+------> reliability + routing calibration
The system objective becomes:
maximize verified useful work
subject to:
residual risk
proof capacity
human attention
compute
authority
evidence independence
Do not:
Tracked implementation/research issue: #693 — measure marginal evidence contribution before adding a verifier.
The first measurement slice is implemented as idkmesh gate-marginal and
documented in
MARGINAL_EVIDENCE_V0_1.md.
It measures add-one verifier contribution under the exact gate rule without
granting routing or integration authority.
The next evidence gate is the held-out benchmark documented in MARGINAL_EVIDENCE_BENCHMARK_V0_1.md. It freezes the marginal selector plus four simpler baselines from design rows only, then evaluates those frozen selections on disjoint holdout rows.
The cross-cohort synthesis layer is documented in MARGINAL_EVIDENCE_SYNTHESIS_V0_1.md. It aggregates multiple already-frozen benchmark reports descriptively, rejects exact duplicate cohort evidence, and preserves unresolved outcomes without emitting a winner or promotion decision. This closes the tooling gap between a single held-out fixture and a broader preregistered cohort, but observed cohort collection remains a separate research gate.
This progression deliberately moves through:
measure marginal contribution
|
v
compare against simple baselines
|
v
require held-out stability
|
v
synthesize multiple frozen cohorts
|
v
require broader observed evidence
|
v
only then consider AVE / Connector Control Plane dry-run integration
The related AVE research remains in #621 / PR #622. The held-out benchmark does not duplicate that controller and does not promote any selector to production.
The product hypothesis remains:
“IDKMesh can use measured reviewer dependence to decide who should review next — but only after the decision rule survives simpler baselines and held-out evidence.”