How tests run in IDKMesh, why the tiers are drawn where they are, and what to do when a gate complains. The measurements quoted here were taken on 2026-09-27; re-measure before treating any of them as current.
make setup # once: create .venv and install test dependencies
make test # the gate: unit tier (see dated baseline below)
Everything else is automation around those two commands.
For contributor setup and testing, use the current repository state rather
than historical issue or pull-request prose. The maintained source of truth is
the combination of CONTRIBUTING.md, this document, the
executable Makefile targets, scripts/testkit.py, and pytest.ini. If an older
issue says a Makefile target is unmerged or should not be assumed, keep that text
as provenance but follow the current files above.
On Linux/macOS and other POSIX-style development environments, the supported convenience path is:
make setup
make test
make integration
The Makefile is not the only supported path. A direct-Python workflow remains
available when make is inconvenient. After creating the virtual environment,
Linux/macOS can run without shell activation:
python -m venv .venv
.venv/bin/python -m pip install -r requirements-phase0.txt pytest
.venv/bin/python -m pytest -q
On Windows PowerShell, use the virtual environment’s interpreter directly:
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-phase0.txt pytest
.\.venv\Scripts\python.exe -m pytest -q
Neither path needs a PYTHONPATH=. prefix for pytest: pytest.ini sets the
repository root on pythonpath. These commands define supported entry points;
they are not evidence that every platform has been independently exercised.
Every number in this section is a measurement with a date attached, not a constant. Re-measure before trusting any figure here, and re-date the line above when you do.
The timing rows were measured on 2026-09-27 on 20 cores at an idle load
average of about 2 on the commit that added
docs/decisions/ADR-0019-run-subresource-suffix-reservation.md, so they are
not directly comparable to a figure from a busier or wider machine; the
section below on CPU-seconds explains why. Load is not a second-order effect
here: the same unit-tier selection measured 91-94 CPU-seconds at load 15-22
and 56.5 at load 2-9 on byte-identical content, so uptime belongs beside
any figure you record. The counting rows are properties of the
tree, not of the machine, and tests/test_documented_tier_scopes.py re-derives
the marker expressions this document publishes directly from
scripts/testkit.py.
Numbers first, because the tier boundaries are derived from them rather than copied from a blog post:
| Quantity | Measurement |
|---|---|
make test (unit tier) |
66.8 s wall, 56.5 CPU-s, 3098 passed / 2 skipped / 436 deselected / 5942 subtests |
Whole suite, no marker filter (python -m pytest -q) |
3528 collected |
Selected by the nightly leg (-m "sim or slow") |
436 |
| Slowest single test in the unit tier | 1.13 s (test_e016_corpus.py::test_build_returns_deterministic_tasks_with_unique_ids, setup) |
| Affected-test run after a one-file edit | 0.1–0.4 s |
| CI, mean run / slowest run / daily volume | not re-measured since PR Gate moved from the full suite to the unit tier — the figures that stood here predate that change and would understate PR Gate’s new speed and overstate its old one |
On 2026-09-27, tests/test_replay_run.py’s five real-orchestration-replay
tests (one spawning two real replay_run.py CLI subprocesses) moved from
unit to slow – together they cost about 26 CPU-s of the unit tier’s 90
CPU-s budget, which is exactly the margin gate (3.13) was failing into on a
busier CI runner with zero test failures. They still run every night; see the
nightly row below.
On 2026-10-04 gate (3.13) on main failed with zero failing tests at 91.7 CPU-s
against the 90 CPU-s ceiling (a re-run of the same commit passed). Eight
bootstrap-resampling tests in tests/test_marginal_evidence.py and
tests/test_gate_audit_uncertainty.py – about 11 CPU-s together, measured
19.3 to 8.8 CPU-s for those two files – moved from unit to slow. They
still run every night; the ceiling is unchanged. Re-measure the whole tier at
idle before trusting the new margin.
This is not a static state — the suite keeps growing, and the response to a
tight budget is always to make the tier cheaper, never to raise the ceiling.
Marking a test slow is one way; making it genuinely cheaper without moving it
is better, because the required gate keeps the coverage. On 2026-09-25
tests/test_cli_tools_help.py stopped spawning an interpreter per discovered
tool and now runs each tool’s --help through runpy in-process, which removed
the single largest entry from the durations table while still exercising every
tool’s real __main__ path.
Before acting on a budget failure, re-measure on an idle machine. A unit
tier red is more often a measurement artefact than a suite that grew: on
2026-09-25 gate (3.13) failed one pull request at 92.1 CPU-s against the
90-second ceiling with zero test failures, and the next push of the same
branch — carrying nine more tests — passed the same leg at 63.8 CPU-s. Across
comparable content that leg has reported 63.8, 65.9, 88.5, 89.5 and 92.1
CPU-seconds. Re-run the gate and measure locally at idle before you re-tier
anything; moving tests out of the required gate to satisfy a noisy measurement
costs real coverage and buys nothing.
The important consequence: this suite is not slow. A full run costs about as much as reading the diff you just wrote. Test selection is therefore a convenience for sub-second feedback, never a substitute for running everything before a commit — skipping a test you should have run costs far more than the time it would have taken.
Each tier owns an explicit CPU-time budget. This is the size-and-budget model large monorepos use (Google’s small/medium/large test sizes; Bazel-style affected-target selection). A tier that exceeds its budget fails, which is the only mechanism that reliably stops a fast suite from decaying into a slow one over a few quarters.
| Tier | Scope | Budget | Runs |
|---|---|---|---|
smoke |
only tests affected by your uncommitted changes | 25 CPU-s | after every edit |
unit |
the whole suite except what’s marked sim or slow (-m "not sim and not slow") |
90 CPU-s | before every commit, and the PR Gate’s required check |
integration |
unit + schema JSON syntax + Markdown link integrity + reference resolution |
600 CPU-s | before every push |
nightly |
integration + everything marked sim or slow (-m "sim or slow") |
none | scheduled — see .github/workflows/nightly-full-suite.yml |
nightly is not equivalent to integration. The two tier markers are in
use: 311 tests carry sim, and -m "sim or slow" selects 436 (the other 125
carry slow — mostly meta-tests that shell out to unittest discovery or
pytest collection as subprocesses, plus the five test_replay_run.py real
orchestration-replay tests, where the subprocess spawn or the real replay
rather than the assertion is what’s slow) — exactly the 436 the unit tier
deselects in the table above. Every one of them runs only in nightly, so a
change that breaks one is invisible to the pre-commit and pre-push gates
until the scheduled run.
This paragraph previously said the opposite — that no test carried
@pytest.mark.sim and that the two tiers therefore did the same work — while
the baseline table in the section immediately above already recorded 369
deselected tests. The document contradicted itself on arrival, which is why the
marker expressions are now re-derived from scripts/testkit.py by a test rather
than retyped here.
make smoke # ~0.4 s what you just changed
make test # the real gate (unit tier)
make integration # unit + link/schema/reference checks
make nightly # the long tail
make gate # picks the cheapest tier that covers your changes
make profile # the 25 slowest tests, when a budget is exceeded
All of them delegate to scripts/testkit.py, so the Makefile, the Claude Code
hooks, and CI execute the same code path and cannot drift apart.
PR Gate runs unit (scripts/testkit.py unit) plus the same Markdown-link
check (scripts/check_links.py), schema backward-compatibility check
(tools/schema_compat_check.py, ADR-0020), schema migration-note check
(tools/schema_migration_note_check.py, the ledger in schemas/README.md),
and OpenAPI/schema reference resolution check (tools/openapi_ref_check.py,
API Conventions section 19) as its required, always-on checks — seconds, not minutes, so a documentation fix
isn’t held up by the health of an unrelated simulation. It does not run
integration as a single delegated call: each shared script stays its own
explicit, stdlib-only step (tests/test_ci_local_gate_parity.py pins that
shape for all of them) so they can run before pip install and cannot
silently diverge into a second, inline copy.
Inside that unit tier, tests/test_api_contract_conformance.py is the API
contract conformance suite (issue #737): every documented examples/api/
fixture is bound to the exact response it documents and validated against the
schema openapi.yaml advertises for it (the catalog is text-scanned, so no
YAML parser is required), keeping catalog, example, and test from drifting
apart silently. Its runtime leg serves representative responses over real
loopback HTTP and validates them against the advertised schema, with an
exhaustiveness guard over every response the catalog advertises. Every
envelope the runtime serves – including the store-unavailable 503s and the
inspect endpoint’s api-error 406/413/415 bodies – is advertised and
validated through its representative, so catalog and runtime cannot drift
apart silently. The documented-examples leg is exhaustive in both
directions: every schema the catalog advertises on a request or response
body must have a committed example in examples/api/, and digest fields
that name embedded fixture content must recompute to the real canonical
digest of it. tests/test_example_contract_coverage.py resolves cross-file
$refs through a schema registry, so the resource-referencing response
contracts validate in depth rather than being skipped or crashing.
The complete suite — nightly, everything unit excludes included — runs
on a schedule in .github/workflows/nightly-full-suite.yml, decoupled from
the merge path. A failure there means the research content regressed, not
that a specific pull request is unsafe to merge.
This machine is shared with other projects. During development of this document a neighbouring repository’s test run pushed the load average past 25 on 20 cores, and the identical IDKMesh suite went from 34 s to 124 s of wall-clock without a single line of test code changing.
A wall-clock budget would have failed the gate for a reason that had nothing to do with IDKMesh. CPU time is the load-independent measure of “is the suite getting slower”, so that is what the budget polices. Wall-clock is still reported, and still used as a hang timeout.
CPU time is not perfectly isolated either — under that same contention the suite measured 53 CPU-s against an idle baseline of 36, because cache pressure and context switching are real costs. The 90 CPU-s budget is set with that in mind.
Per-test hang ceilings are deliberately not used. pytest’s
faulthandler_timeout looks like a free upgrade, but arming it starts a
faulthandler watchdog thread for the whole run, and
sim/e033_goal_distance.py parallelises with
multiprocessing.get_context("fork") because each job re-points the
environment’s future goal in module globals. Forking a multi-threaded process is
the documented deadlock case: a thread holding a lock at fork time leaves it held
forever in the child. Measured — with faulthandler_timeout = 300,
tests/test_e033_goal_distance.py emitted two such warnings; at 0, the same 69
tests emitted none. The watchdog is a C-level thread, so it never shows up in
threading.enumerate(); the warning is the only symptom. pytest.ini records
this so the setting is not reintroduced.
Do not raise the budget. Run make profile, find what got slow, and either
fix it or mark it for a later tier:
@pytest.mark.sim # excluded from unit; runs in nightly
@pytest.mark.slow # measured above the tier-2 per-test budget
Raising a budget converts a one-time cost into a permanent one, and there is no natural point at which anyone ever lowers it again.
This is a local, opt-in setup, not repository content. .gitignore excludes
.claude/ — it holds per-agent configuration and personal notes that must never
be committed to a public repository — so the hook files described here are not
in your checkout and git pull will never bring them. They are reproduced in
full below, because that is the only way this page can hand them over.
Set up this way, the tiers behave like a continuous test runner (NCrunch, Wallaby, Infinitest): feedback arrives while the change is still in working memory.
| Hook | Event | Tier | Behaviour |
|---|---|---|---|
test-on-edit.sh |
PostToolUse on any edit |
smoke |
silent on success; exit 2 hands the failure back |
test-on-stop.sh |
Stop |
auto |
blocking — a turn does not end on a red tree |
.claude/settings.json:
{
"hooks": {
"PostToolUse": [
{
"matcher": "Edit|Write|NotebookEdit",
"hooks": [{ "type": "command", "command": ".claude/hooks/test-on-edit.sh" }]
}
],
"Stop": [
{ "hooks": [{ "type": "command", "command": ".claude/hooks/test-on-stop.sh" }] }
]
}
}
.claude/hooks/test-on-edit.sh:
#!/usr/bin/env bash
# Tier 1 after every edit. Silent on success; exit 2 hands the failure back.
set -uo pipefail
cd "$(git rev-parse --show-toplevel)" || exit 0
if ! output=$(python3 scripts/testkit.py smoke --quiet 2>&1); then
printf '%s\n' "$output" >&2
exit 2
fi
.claude/hooks/test-on-stop.sh:
#!/usr/bin/env bash
# Whichever tier the change requires, before the turn ends.
set -uo pipefail
payload=$(cat)
# Claude Code sets stop_hook_active once this hook has already blocked in this
# turn. Blocking again would loop forever on a failure nobody can fix.
case "$payload" in *'"stop_hook_active":true'*) exit 0 ;; esac
cd "$(git rev-parse --show-toplevel)" || exit 0
if ! output=$(python3 scripts/testkit.py auto --quiet 2>&1); then
printf '%s\n' "$output" >&2
exit 2
fi
chmod +x both. What makes this cheap enough to run constantly is the result
cache: scripts/testkit.py fingerprints the content of every tracked file plus
uncommitted changes, so re-running a tier that already passed on an identical
tree costs ~0.1 s instead of 50 s, and a conversational turn that touched no
code is not taxed.
The fingerprint deliberately has no extension allowlist. Hashing only
.py/.json/.ini looks like a cheap optimisation and is actually unsound
here: the integration tier link-checks 418 tracked .md files and the workflow
guards read 51 .yml files, so a real workflow violation reported
cached pass -- tree unchanged while the guard, run directly, failed. Hashing
all 1320 tracked files (~29 MB) costs ~0.1 s warm, against tiers that run for
tens of seconds. A cache that can hide a genuine failure is worth less than the
time it saves.
To make the edit hook cost nothing on the happy path, run it in the background
and let it interrupt only on failure; the Claude Code hooks reference documents
the key for that (asyncRewake). The scripts above are correct either way, and
the blocking form is what the measurements below were taken against.
Verified on 2026-09-10 against a deliberately failing test added to tests/,
in the checkout where they were authored: on a green tree both exit 0; on a red
tree both exit 2 with the failure text on stderr; and with
"stop_hook_active":true the Stop hook exits 0, so an unfixable failure blocks
once and then lets the turn end rather than looping. Nothing in the suite can
re-check that for you — these files are not in the repository — so treat it as a
report, not a guarantee, and re-run the same probe after editing them.
smoke maps changed sources to tests two ways: the tools/foo.py →
tests/test_foo.py naming convention, and a reverse-import scan for any test
importing the changed module. Changing tools/idkgraph_link_check.py, for
instance, selects both test_idkgraph_link_check.py and
test_idkgraph_health_checks.py.
Selection deliberately fails open: a change to a conftest.py, an
__init__.py, or pytest.ini cannot be attributed to specific tests, so the
runner widens to the full suite instead of trusting a partial answer.
pytest.ini sets pythonpath = ., so a bare pytest now works. The old
PYTHONPATH=. pytest prefix is no longer required (plain unittest discovery
still under-collects; prefer pytest).
Repository-wide workflow hygiene is now part of current main and has an
executable regression contract in tests/test_workflow_ci_hygiene.py. The guard
requires every workflow to declare a concurrency group, rejects a bare
workflow-level cancel-in-progress: true that could destroy branch/scheduled
evidence, and requires explicit timeouts for ordinary jobs. Workflow-level
cancellation may instead be disabled or gated to pull-request events.
Job-level concurrency remains a deliberate exception surface: workflows such as the evolution controller can use separately keyed advisory/canonical groups when the group expression itself preserves the required isolation. The tests pin those repository-specific cases rather than flattening every workflow into one concurrency policy.
When changing .github/workflows/, run the focused workflow/security tests in
addition to the normal repository gate and treat the current workflow files plus
their regression tests as the source of truth; do not rely on an older issue or
PR description of CI behavior.
Conventions are unchanged: files are test_*.py, classes end in Tests,
methods begin with test_, fixtures are deterministic and seeded. pytest.ini
sets --strict-markers, so a typo’d marker is an error rather than a silent
no-op.
Mark anything long-running:
@pytest.mark.sim
def test_full_sweep_replays_the_committed_evidence(): ...
| Symptom | Cause |
|---|---|
pytest: command not found |
Run make setup. |
| A tier passes instantly without running | Cached pass on an unchanged tree. make clean-cache or pass --no-cache. |
BUDGET EXCEEDED |
Run make profile; mark the offender sim/slow. Do not raise the budget. |
| The Stop hook keeps blocking | The suite is genuinely red. It blocks once per turn, never in a loop. |
| Hooks do not fire | Claude Code watches .claude/ only if a settings file existed at session start. Open /hooks once, or restart the session. |