A verifier panel is only as strong as the independent evidence its members contribute. Ten reviewers that fail on the same cases may be worth much less than ten independent votes.
This problem is central to IDKMesh because agentic systems can cheaply create both candidates and reviewers. Counting votes without measuring dependence can make a review gate look stronger while adding little real protection.
A panel’s nominal size is the number of reviewers. Its effective size asks how much independent information the panel actually contains.
IDKMesh’s gate-audit diagnostic accepts a matrix of observed verdicts and reports:
The tool is diagnostic. It does not grant acceptance or merge authority.
Reviewers can correlate because they share the same tests, training data, architecture, prompt, blind spot, or task misunderstanding. Majority voting assumes more than “different names in the panel”; it depends on how errors line up across items.
That is why IDKMesh treats verifier diversity as an empirical property rather than a label.
Raising a quorum can help only when the panel contains useful discriminating evidence. If reviewers systematically miss the same defect, requiring more of the same votes does not create new information.
The right response may be to change the evaluator, add a different kind of evidence, improve hidden tests, or route difficult cases to a human specialist. For automated judges, see LLM-as-a-judge reliability.
E017 measured a panel of 25 independently seeded partial test oracles over a 72-candidate corpus whose ground truth came from hidden tests. These verifiers were programs, not people or LLM judges, and every recorded error was a genuine missed defect.
The observed panel looked large but carried far less independent evidence:
| Measurement | Observed result |
|---|---|
| Mean verifier accuracy | 0.7956 |
| Mean pairwise error correlation | 0.5873 |
| Majority-vote panel error | 0.2083 |
| Single-verifier error | 0.2044 |
| Measured effective panel size | 1.00 of 25 |
| Common N/(1+(N-1)rho) heuristic | 1.66 |
| Error with a 24-of-25 acceptance quorum | 0.0556 |
Under majority vote, the 25-verifier panel was therefore no better than a single member on that corpus. The common effective-size heuristic still overstated the measured panel by 1.66x.
The highest-leverage change was not adding reviewers. Because the test-oracle errors were one-sided missed defects, changing the aggregation rule from majority acceptance to a much stricter quorum cut error from 0.2083 to 0.0556, about 3.7x. Four defects remained invisible to every verifier, so no quorum could eliminate the floor.
It is evidence about this measured programmatic panel, not a universal constant for human reviewers or LLM judges. The verifier diversity structure was constructed, the corpus contains 72 candidates from 24 problems, and the one-sided error property comes from partial test oracles.
IDKMesh attempted a live LLM-verifier measurement earlier in E016, but the 20 LLM verifiers did not discriminate above chance strongly enough to make their correlation estimate meaningful. The repository therefore does not claim that a 25-LLM panel is worth one vote.
The transferable lesson is narrower and more useful: measure reviewer competence and shared error structure before treating reviewer count as independent evidence.
Reproduce the retained votes, analysis, and acceptance metrics with the E017 reproduction guide.
It is a way of expressing how much independent information a correlated panel contains relative to an idealized set of independent reviewers.
Collect verdicts on cases with known or externally established outcomes, inspect per-item errors, measure dependence/correlation, and include seeded probes that the gate should reject.
No. More reviewers increase value only when they add sufficiently independent useful evidence relative to their cost and latency.
A quorum is the threshold or rule used to turn individual verification results into a panel-level recommendation. Its usefulness depends on reviewer quality and dependence.
Run the gate-audit quickstart and read the Gate Audit v0.1 specification.
Choose evaluators that differ in evidence source, implementation, model family, tests, or expertise where those differences are relevant to likely failure modes. Diversity is valuable when it reduces shared blind spots, not when it is only cosmetic.
It means multiple reviewers are wrong on the same items more often than independent reviewers would be. Correlation reduces how much new information each additional vote contributes.
Set it from measured reviewer performance, risk tolerance, and the cost of false acceptance versus false rejection. A quorum should be validated on representative cases rather than copied from panel size alone.
Abstention is appropriate when required evidence is missing, reviewers disagree beyond the calibrated decision boundary, or the case is outside the evaluators’ demonstrated competence.
Compare item-level error patterns, prompts, tools, data sources, model/provider lineage, and evaluator design. Nominally different reviewers that fail on the same cases are not providing much independent protection.
Browse all AI-agent trust topics.
Last reviewed: 2026-09-24.