Continue improving MSKazemi/idkmesh after the pull-request convergence pass, while keeping substantive project work in the public repository.
The repository had already advanced beyond the earlier planning snapshot:
main;pending;WorkerAdapter / result-bundle boundary;main fixed the task-001 repository-path bug, but benchmark outcome evidence remained explicitly separate.The frozen benchmark source remains 9c53bb4069a5db1c0688dbbe7a8f028540cbf7c2. Any candidate-generation experiment must use that snapshot rather than exposing later solutions to the producer.
The accepted PR #91 node is an execution sandbox, not a goal-to-code reasoning engine. Its demonstrated smoke WorkUnit contains an explicit org.idkmesh.node.execution container command. The frozen benchmark WorkUnits contain coding objectives and constraints but deliberately do not contain such commands.
Therefore directly feeding the benchmark tasks to #91 would misrepresent what the node does. The missing experimental capability is a goal-level candidate producer that can emit a canonical candidate bundle for the existing verifier/coordinator path.
A previously attractive option, GitHub Models, cannot be used: GitHub’s current documentation says the GitHub Models playground/catalog/inference API were fully retired on 2026-07-30.
GitHub documentation also states that standard GitHub-hosted runners are free for public repositories. The experiment therefore uses only a standard public-repository runner and requests no larger runner, paid inference API, model secret, artifact upload, or cache.
References:
GitHub Copilot cloud agents were not selected for this experiment because current documentation describes them as a paid-Copilot capability that consumes AI credits. The goal here is to preserve the repository’s zero-project-spend experiment boundary.
The smallest probe uses:
Qwen/Qwen2.5-Coder-0.5B-Instruct;bbf27711794f58ebd1796058f4280b53c32e19fc;do_sample=false, seed 0;001-cohort-path-boundary;single-worker-baseline-v1.Model reference:
This first 0.5B model is intentionally weak enough to make failure plausible. The experiment should be able to discover that a small/free producer is not useful; a negative result is evidence rather than a reason to weaken the benchmark.
The model must not see the already-landed solution or evaluator-owned semantic expectation.
The inference container receives only:
It does not receive:
main checkout;GITHUB_TOKEN;Provisioning and candidate generation are separated.
During trusted provisioning, the workflow may use network access to install pinned Python packages and fetch the pinned public model snapshot. Candidate inference then runs in a separate Docker container with:
none;no-new-privileges;Model output is untrusted text. The host harness accepts only one textual Git diff for the exact WorkUnit allowed path, applies it only to a disposable checkout of the immutable source revision, normalizes the resulting patch with Git, and never executes candidate code.
If the producer emits a usable patch:
frozen WorkUnit
-> isolated open-weight model text generation
-> bounded normalized candidate patch
-> ResultManifest v0.1 (worker self-report only)
-> frozen public EvaluatorPlan v0.2
-> independent metadata-only unified-diff verifier
-> VerificationResult v0.1
-> probe evidence
If the model emits malformed or out-of-scope output, that is recorded as producer_output_rejected. If the independent verifier rejects a structurally valid candidate, that is recorded as a real rejection. Neither case is converted into CI harness failure merely to manufacture a green model result.
Harness/invariant errors remain CI failures.
The experiment has no authority to:
human_integration_decision_required remains true.
The workflow uploads no artifact and uses no Actions cache. Concise evidence is written to the job summary/logs. This first probe does not change the frozen cohort evidence fields. Attaching evidence is a separate step after the run is inspected and provenance/reproducibility are adequate.
Use one bounded, zero-secret, open-weight producer probe to test whether free public-repository compute can create a real goal-level coding candidate while preserving IDKMesh’s existing WorkUnit -> ResultManifest -> EvaluatorPlan -> VerificationResult trust chain.
Do not create a new task protocol, do not modify the frozen benchmark definition after observing the result, and do not bypass PR #91’s separate human-review gate.