Per-benchmark coverage score — audit note (23 September 2026)

The overview's lead figure scores each primary benchmark by the number of distinct stage × layer cells, out of 35, in which at least one of its task verifiers executes the real LLM workload (a training run, a finetune, or a kernel on the declared hardware). This is the measured evidence class of the 5 September method note. A lighter extension adds cells reached only by bounded (unit or artifact checks in a narrower harness) or adjacent (a fixed dependency, a simulation, or a nearby non-LLM task) evidence. Family entries count once. The score counts cells, never tasks, so a task that reaches two cells is not double-counted and per-cell task counts are never summed.

Scores

Why four benchmarks score zero (verifiers re-read on 23 September)

• Terminal-Bench 2.1 (commit 7131e43): every task declares gpus = 0; GPU tasks arrived in 3.0. The tensor-parallel task starts gloo process groups on CPU at world sizes 1, 2 and 4 and compares a 2 × 64 input against nn.Linear with allclose. The pipeline task builds a 4-layer LLaMA (hidden 512) on CPU and compares hooked activations with a reference. gpt2-codegolf compiles the C file, generates a few tokens and string-matches. hf-model-inference loads a Hugging Face sentiment model and checks six POST responses. The batching task is scored by an analytical cost model. Five cells are bounded, one adjacent, none measured.
• DeepSWE v1.1 (commit 0b9fabb): all 113 tasks declare gpus = 0. The three infrastructure-adjacent tasks run go test -run TestCoalesceValues (Helm), the Arcane drift-detection service and handler tests, and pytest test_coalesce.py (LangChain). Real deployment tooling, but unit tests, and adjacent to LLM infrastructure.
• WeirdML v3: a real GPU sandbox (four CPUs and one GPU per the task page of 16 September 2026), but the tasks are point-cloud classification, ship detection, radio-telescope calibration and the 17-task "Bonanza". Real training of non-LLM models; no LLM life-cycle stage is touched.
• EdgeBench: the 51 public tasks in the README table are Scientific & ML (4: bipedal-walker RL, gravity inversion, GNN node classification, source inversion), Systems & SE (12: ANN vector search, VLIW kernel optimization, ffmpeg swscale, git in Zig, and others), Optimization (15), Knowledge (4), Formal (8) and Games (8). None trains, serves or checkpoints an LLM or writes a GPU kernel.

The 35 cells and the 10 that no task reaches

A cell is one life-cycle stage (data; pretrain & midtrain; post-train; agent post-training; serve & deployment) crossed with one infrastructure layer (model & algorithm; data pipeline & evaluation; training & inference engines; parallelism & communication; kernels & compilers; cluster orchestration & sandboxing; memory & state management). Each layer carries a contract stating what a test there has to do; a task lands in a cell when its verifier reaches that stage and exercises code at that layer. Fifteen of the 52 entries sit in two stages.

Sources for the production examples are the pipeline, RL-loop and agentic tables of the overview and deep-dive pages.