Status: working architecture direction, not a frozen implementation.
IDKMesh should not try to become one giant cluster, one giant scheduler, one giant database, or one giant multi-agent conversation.
The architecture should be a fractal federation of autonomous cells:
node -> cell -> region/fabric -> federation
Each level uses similar concepts — identity, task queues, artifact exchange, verification, summaries, and policy — but most work is completed at the lowest level that has enough capability.
The system should scale by adding cells, not by making one control plane infinitely larger.
This is consistent with lessons from existing large-scale systems. Kubernetes v1.37 documents a supported cluster envelope of up to 5,000 nodes, 150,000 pods, and 300,000 containers; going to internet-scale IDKMesh therefore requires federation/multi-cluster architecture rather than assuming a single Kubernetes cluster can represent millions of machines. Ray similarly uses a cluster-level head/control service and distributed workers, making it a useful execution substrate inside a cell rather than a natural single global control plane. libp2p provides modular peer-to-peer transports, encrypted links, NAT traversal, discovery and multiplexing that are relevant to federation across heterogeneous networks.
A useful conceptual latency model is:
T(N) = T_serial + T_parallel/N + T_coord(N) + T_verify(N)
Adding agents or computers only helps while the decrease in parallel work is larger than the growth of coordination and verification overhead.
The Universal Scalability Law is also a useful warning model:
C(N) = N / (1 + alpha*(N-1) + beta*N*(N-1))
where alpha represents contention and beta represents coherence/coordination cost. At very large N, even small coherence costs dominate.
Therefore the main architectural goal is not merely parallelism. It is to drive global contention and coherence toward zero.
A node may be a laptop, workstation, server, GPU box, cluster gateway, browser-capable worker, or organizational endpoint.
A node advertises bounded capability metadata:
capability = {cpu, gpu, memory, storage, bandwidth, locality, trust-domain, skills, runtimes, privacy constraints}
It should not publish unnecessary private details globally.
A cell is the primary operational failure and coordination domain.
A practical early cell might contain 10-200 workers; mature cells might contain hundreds or thousands depending on workload and infrastructure. A cell can be a lab, company, university, Kubernetes cluster, home-device group, geographic region, or temporary compute swarm.
Each cell owns:
Most tasks should start and finish inside one cell.
A fabric joins cells with compatible trust, geography, latency, organizational ownership, or workload characteristics.
It exchanges aggregate information such as:
It should not continuously mirror the full internal state of every cell.
The global layer should be thin.
Candidate responsibilities:
It should not schedule individual tasks globally.
Stores goals, hypotheses, task graphs, requirements, evidence, decisions, provenance, and artifact relationships.
Key idea: represent work as a graph, not as a chat transcript.
Creates tasks, decomposes work, matches tasks to capability classes, handles leases, retries, backpressure, budgets, and cross-cell overflow.
Control is hierarchical and sharded.
Runs code, agents, models, simulations, tests, compilers, training, data processing, and other workloads near the appropriate data/compute.
Execution workers should remain useful even if higher control layers temporarily disappear.
Validates outputs through tests, independent replicas, adversarial agents, policy checks, provenance, signatures, sandbox results, human review, and formal verification where appropriate.
Verification is risk-adaptive rather than uniform.
The core abstraction should be a Task Contract, not an unrestricted agent conversation.
A task should contain something close to:
Task {
id
objective
inputs[]
expected_outputs[]
acceptance_tests[]
resource_requirements
trust_requirements
privacy_constraints
budget
deadline
retry_policy
verification_policy
provenance
parent_task
dependencies[]
}
Outputs become immutable/versioned Artifacts with hashes and provenance.
This contract-first model is critical for both scalability and agility: workers can be replaced without changing the task semantics.
Use hierarchical scheduling.
The node chooses among local runnable tasks and can use work stealing between nearby workers.
Matches tasks to nodes based on hard constraints and soft scores.
Candidate score:
score(i,j) = capability_fit + locality + reliability + diversity + expected_information_gain - cost - latency - risk
If a cell cannot satisfy a task, it publishes a summarized demand envelope to neighboring/federated cells.
Only unsatisfied or strategically distributed work leaves the cell.
Used for rare discovery, not per-task scheduling.
This architecture avoids one global scheduler whose work grows with every participant.
Ray’s current scheduler is a useful reference for cell-level capability/resource scheduling: tasks specify resources, nodes are filtered for feasibility, and locality/resource utilization influence placement. IDKMesh should borrow the principle, not necessarily the exact implementation.
Do not choose one consistency model for the entire platform.
Use sparingly for:
Use for:
Use for:
Immutability dramatically reduces distributed-state conflict.
Use efficient broker/RPC mechanisms. Candidate technologies can include NATS/JetStream, gRPC, local actor runtimes, or Kubernetes-native mechanisms.
Use federation gateways with bounded connections and summarized state.
libp2p is worth evaluating because its current stack explicitly supports modular transports such as TCP, QUIC, WebSocket, WebRTC and WebTransport, encrypted connections, NAT traversal, peer discovery, routing and protocol multiplexing.
Use gossip for soft state and discovery, not for every artifact payload.
Large artifacts should move directly between selected peers/object stores after control-plane negotiation.
Prefer content-addressed immutable artifacts.
artifact_id = hash(canonical_content + metadata_version)
Benefits:
Move compute to data when possible. Scheduling should price network transfer explicitly.
Assume:
Every task must define at least:
Durable workflow systems such as Temporal are relevant references for long-running control operations because they persist workflow history and resume execution after process or infrastructure failures. IDKMesh should evaluate such durable-execution semantics for control workflows, while keeping the execution mesh itself substrate-independent.
The platform’s central quality hypothesis should be implemented structurally.
For each task define a verification class:
Verification should be selected according to expected loss:
verification_budget proportional to probability_of_failure * consequence_of_failure
Do not send every result to one central reviewer. Verification must itself be distributed.
Replication is only valuable when failure modes differ.
When possible, verification replicas should vary one or more of:
The scheduler should eventually estimate pairwise failure correlation and prefer independent validators.
Every queue should have bounded capacity or explicit admission control.
When overloaded:
Never let infinite queues turn temporary overload into a system-wide collapse.
Scalability and agility are not opposites if contracts are stable and components are replaceable.
Keep a very small stable protocol kernel:
Everything else should be pluggable:
Use explicit schema/API versions with compatibility tests.
A new scheduler can observe the same workload and produce hypothetical decisions without controlling execution. Compare it against the production scheduler before rollout.
Deploy protocol/runtime changes to a small set of volunteer cells first. A cell is the natural canary unit and blast-radius boundary.
simulation -> replay -> shadow -> 1 cell -> 1% cells -> 10% -> broad deployment
A deployment should define rollback thresholds before rollout.
Every major architectural commitment should link to evidence, benchmarks, and an Architecture Decision Record. Decisions are revisable when evidence changes.
Run all four planes in one process or a few local processes. No distributed-system complexity unless required by the interface.
Goal: prove Task -> Artifact -> Verify.
Introduce real queueing, leases, capability scheduling, local artifact cache, sandboxed workers, retries, and distributed validation.
Goal: prove heterogeneous reliable execution under churn.
Introduce federation gateways, overflow scheduling, cross-cell artifact discovery, cell autonomy, and partition tests.
Goal: prove that cross-cell coordination grows much slower than node count.
Shard directories, use gossip/summary exchange, automate cell creation, hierarchical telemetry, stronger trust mechanisms, abuse resistance, and large-scale simulation.
Goal: demonstrate no global bottleneck in the normal work path.
Cells become administrative, security, network, and social domains. Global services contain only small indexes, protocols, attestations and routing hints. Most raw state stays local.
Goal: adding another 10x participants should mostly add capacity rather than 10x central coordination burden.
Track these from the first prototype:
Reject architecture proposals that require any of the following in the normal task path:
Build the smallest system that already has the final scaling shape:
idkmesh-node — worker daemon with capability advertisement and sandboxed task execution.idkmesh-cell — local task broker/scheduler, artifact index, event log, verifier and API.Task and Artifact schemas — versioned and content-addressed.Do not begin by implementing blockchain, global consensus, distributed model training, tokens, or millions-node networking. First prove that the architecture’s coordination cost remains bounded as simulated node count grows.
Simulate N = 10, 100, 1,000, 10,000 and 100,000 workers.
Compare:
A. one global queue + scheduler; B. sharded queues with one global directory; C. autonomous cells + overflow federation.
Measure:
The first major architectural milestone should be evidence that model C keeps per-task coordination roughly bounded as N grows.
Adopt Fractal Autonomous Cells as the current default hypothesis for IDKMesh, subject to benchmark falsification.
The project should deliberately attempt to disprove this architecture through simulation and prototypes before treating it as permanent.