An LLM-as-a-judge is an evaluator, not an oracle. Its verdict becomes useful only to the extent that its error rate, calibration, independence, task fit, and authority are understood.
IDKMesh’s broader verification work applies the same discipline to any automated evaluator: record what the evaluator saw, what plan it used, what artifacts it judged, and what evidence supports its conclusion.
A panel can contain many judges and still behave like far fewer independent judges if they fail on the same items. IDKMesh therefore distinguishes nominal panel size from effective independent votes.
The project’s E017 experiment measured this effect using independently seeded partial test oracles—programs, not language models—on a real defect corpus. It is evidence about correlated verification errors and quorum design, not a claim that the same numeric result applies to LLM judges.
See verifier panels and the retained E017 record.
A serious evaluator study should separate several questions:
A high average agreement score cannot answer all of these.
Sometimes, for a defined task and measured evaluator. Reliability should be established on representative cases and reported with limitations; it should not be inferred from model reputation.
Not necessarily. Different models can share training data, prompts, abstractions, and item-level failure modes. Measure error dependence rather than counting brands.
Calibration asks whether the evaluator’s confidence or score meaningfully corresponds to correctness or outcome frequency. It is distinct from simple agreement.
It can contribute evidence. Final integration authority should remain a separate policy decision, especially for security-sensitive or high-impact changes.
See the EvaluatorPlan and VerificationResult specifications and schema index.
They can be used as evaluators when the task, rubric, model/version, prompt, and observed reliability are recorded. Brand or model size alone does not establish that a judge is accurate for a particular evaluation.
Use explicit rubrics, blinded ordering where possible, counterbalanced prompts, held-out calibration cases, multiple evidence types, and measurements for systematic preference rather than relying on one prompt format.
Often it is safer to hide irrelevant producer identity when measuring output quality, because model names can introduce preference bias. Keep identity available in provenance even when it is blinded from the scoring prompt.
There is no universal number. Add judges only while they contribute useful independent information relative to cost and latency; correlated judges can make a large panel behave like a much smaller one.
Preserve or expose the uncertainty, request additional evidence, use a different evaluator class, or escalate to a human rather than forcing every case into an accept/reject verdict.
Browse all AI-agent trust topics.
Last reviewed: 2026-09-22.