Verifying the Verifier

Summary: This report puts an LLM-as-a-verifier harness through a self-audit. The headline result is 85.4% (76/89), gate PASS. Before trusting that number, the authors ran the exact same benchmark against a dead endpoint and got a plausible-looking 79.8% out of pure connection errors, because the framework silently scores failed calls as a 0.5 tie. That single discovery reframes everything else: a probability-weighted grading method that only matters on the 24 "swing" tasks where three attempts disagree, a prefill-bound hardware profile (not decode-bound), a thermal monitoring gap where the visible GPU-die sensor lags the actual trip sensor by over 20°C, seven separate bugs found only by running the code (not reading it), including a tokenizer edge case that silently voided 22% of all scores, and an eight-check negative-control gate that had to prove it could fail before its "PASS" was allowed to mean anything. The published cloud-API number this project targets (86.5%) is not reproduced here (different API path, thinking disabled, local checkpoint). This run lands at 85.4%, one swing task apart, within noise.

Terminal-Bench 2.1 · Self-Verification · 2× DGX Spark GB10

Can a model grade its own homework? And, the harder question, how would you know if your measurement of that was made of nothing at all?

DeepSeek-V4-Flash-0731   vLLM 0.21.1rc1 · TP=2 · 1M ctx   fp8_ds_mla · MTP k=5   576 scoring calls / 1,728 requests   85.4% · gate PASS · 2026-08-18

The finding that matters most

A dead endpoint scores 79.8%

Before trusting any number this harness produced, we unplugged it entirely, pointed it at a closed port, and ran the full benchmark against nothing.

Negative control · executed, not reasoned about

Every one of the 576 scoring calls failed with Connection error. The framework scores a failed call as a 0.5/0.5 tie by default, so the run completed normally, printed a clean table, and exited 0.

The result sits exactly where a plausible result should sit: above the floor, below the ceiling. Nothing about the headline reveals that no model was ever consulted.

Verifier tokens (0 verifier calls)
$ verify_run.py --strict
RESULT: FAIL, exit 1 ← the gate refuses it

The deliverable of this run is not the percentage. It is the gate verdict sitting next to it.

The only honest tell in that output is 0 verifier calls, buried in a token-accounting block most readers skip. So the score stopped being the deliverable. A separate gate, which re-derives every cache key that should exist and refuses anything incomplete or poisoned, became the thing that decides whether a number gets published.

What is actually being measured

Only 24 tasks are in play

The model attempts each Terminal-Bench task three times. Of 89 tasks, 58 succeed on every attempt and 7 fail on every attempt, those are decided regardless of any grader. The benchmark lives entirely in the remaining 24 "swing" tasks, where the attempts disagree and picking correctly is the whole skill.

The grader never sees an answer key. It reads two attempts side by side and rates them A–T, then, instead of taking the letter, the harness reads the model's probability distribution over the 20-letter scale at that exact position and takes the expectation. A far finer signal than a verdict.

Each swing task = 1.12pp

Result: 76/89 = 85.4%, i.e. 18 of 24 swing tasks selected correctly, closing 47% of the gap between blind picking (12.67/24) and the oracle. The published cloud-API result, 86.5%, is exactly 19 of 24. One swing task apart: 1.1pp, against single-seed noise of ±2.2pp. Statistically indistinguishable, and not a reproduction, see the caveats below.

This is not a reproduction

The repository targets DeepSeek's paid cloud API. We pointed it at two Sparks on a tailnet instead, a different code path inside the project's own source (call_openai, not call_deepseek), thinking disabled rather than enabled, grades read by constrained prefill rather than emitted by the model, and a local 0731 checkpoint judging trajectories generated by the cloud model. The honest label is "local 0731 as verifier," not "86.5% reproduced."

Measured on the hardware

It is a prefill benchmark, not a decode one

Each comparison ships two full agent trajectories into one prompt. Median prompt: 77k tokens. p95: 180k. Largest: 215k. That reframes everything, earlier timing chaos (30.9 s/it → 133.9 s/it) was never concurrency contamination. It was prompt size varying sixfold.

Single request · unique cache-defeating filler · clocks uncapped · 2 reps per point, within 3%

Throughput degrades gracefully, 1,884 → 1,338 tok/s across an eightfold depth range, near-linear in wall-clock rather than quadratic. Sparse MLA is doing its job. At the benchmark's real span the machine holds ~1,750–1,850 tok/s, putting a worst-case 215k-token prompt at roughly two minutes uncontended.

Two independent probes · healthy k=5 decays; a broken drafter collapses flat

A bare acceptance percentage is close to useless here, it swings ~20 points on prompt style alone. The shape is the signal, and both probes decay cleanly inside the healthy band.

The monitoring blind spot

The stage-1 trip prompted a full sensor survey. Every surface below exists on the machine and is readable without special privilege, and none of it appears in nvidia-smi, which is what a GPU dashboard shows you.

Two things fall out of that table. The board/die gap is +10°C at steady state and was +22°C at the trip, so the die sensor lags the one that actually governs throttling, and lags it furthest exactly when it matters. And the 200GbE fabric NICs sit at 68–71°C with nothing anywhere watching them.

There is a labelling trap here too. All seven board zones report the type string acpitz, with no labels and no way to tell CPU from SoC from PCIe rail. They cannot be named from the system, so a dashboard that prints "CPU temp" next to one of them is asserting something it did not measure. They have to be identified behaviourally, by watching which zones move under a CPU-only load versus a GPU-only load.

A claim this page made, and then had to retract

An earlier version of this section reported that "spark-2 runs consistently hotter than spark-1, 89.3°C against 84.4°C, a persistent five-degree delta." That came from a single snapshot. Once the collector had logged 101 paired samples, it did not survive:

The paired difference is +0.64°C mean in the opposite direction to the claim, and it ranges from −8.4 to +9.0°C, the two nodes trade places constantly, with spark-1 hotter in 66% of samples. There is no persistent asymmetry. What is true, from the guard's own log, is narrower: spark-2 was the node that happened to reach 95.4°C at the one trip observed.

This is left in rather than quietly edited out, because it is the same error this page is about, a number that looked solid, was read once, and generalised. The only reason it was caught is that something was logging continuously instead of sampling when curious. Over this 101-sample window the board peaked at 94.1°C against a 95°C trip point; across the full 1,924-sample run it reached 95.7 / 95.8°C, and the guard's own 5-second polling recorded 95.4°C at the trip. Three sampling regimes, three peaks, quote the window or the number means nothing. A spot check would have called any of them comfortable.

Notable findings

Seven things worth knowing

1. An off-by-one silently destroyed 22% of all scores (found live)

The first run of this benchmark cached 78 of 357 scores (22%) as an exact 0.5 tie, with zero errors raised and nothing in the output to suggest a problem. A fifth of the evidence was being thrown away silently. (What that actually cost the headline number is measured below, and it is less than this paragraph originally claimed.)

Cause: the grade is read by prefilling <score_A> and taking the model's letter distribution at that position. When the grammar-constrained sample lands on a bare space, a legal prefix of " A", and common with tokenizers that emit a standalone space token, content.strip() yields an empty string. The tag-lookup walks the token list accumulating text and keeps the last match; the empty token leaves that text unchanged, so the tag matches a second time and the lookup advances one slot past the real distribution onto the closing tag's zero-logprob placeholder.

The distribution was present and correct every single time; 19 of the 20 returned alternatives were valid score letters in every case observed. It was retrieved, then overwritten. The regex fallback cannot rescue it either, the reconstructed text holds <score_A></score_A> with nothing between, and the pattern requires at least one character.

Fixed in one line, skip whitespace-only tokens when advancing. Verified three ways: the bare-space case now returns a score bit-identical to the same distribution sampled as a letter; a genuinely empty distribution still correctly ties; and across the completed 576-call run, 35 bare-space events occurred and produced zero poisoned entries. Reported upstream as issue #10 with a self-contained repro.

This is plausibly the concrete instance of a gap the authors themselves flagged as unstudied: "score-token mapping sensitivity unanalyzed across languages and tokenizers." The trigger is purely a tokenization property of the verifier model.

2. Failed calls are cached as real data (silent failure)

The framework swallows exceptions in its grade-reading path and returns a 0.5 tie as a normal value. That tie is then written to the score cache indistinguishably from a measurement. A run in which every grade-read failed produces a complete-looking cache that passes any count-based check.

Detection: the 1–20 scale maps to (v-1)/19, which can never equal exactly 0.5, nearest values are 0.4737 and 0.5263. So an exact 0.5 is a signature, not a coincidence. The gate flags every one.

In fairness to the authors: against the paid cloud endpoint this project targets, with SDK retries and a rate-limited API, degrading a rare failure to a neutral tie is a defensible default. It becomes dangerous in the configuration we chose: a single self-hosted endpoint where a systematic fault takes out every call at once, rather than a few.

3. The expected-count check was itself wrong (self-inflicted)

The tournament runs a 3-pair ring plus a 2-pair pivot round, but those sets always overlap by exactly one directed pair, provably, across all six Hamiltonian cycles and all three pivot choices. So 576 calls, not 720.

The original gate asserted 720. A flawless run would have failed it, sending us hunting a phantom. Caught by exhaustive enumeration, not by reading.

4. One slow request could become six (amplification)

The SDK defaults to a 600s read timeout with 2 retries, and the calling code re-issues the whole request on any exception. A single 200k-token prefill running long could therefore fire up to six full attempts, piling the heaviest work onto an engine already struggling, before being silently scored a tie. Patched to an explicit ceiling with retries disabled and connect split out at 10s.

5. This run doubles as a crash reproduction (environment)

The cluster's last engine death left a worker wedged on an xgrammar structured-output request, though the postmortem lists at least two candidate mechanisms and ranks neither, so this is a suspect, not a cause. Every grade-read here uses structured outputs, the run issues 1,152 of them. So before launching, container logging moved off json-file, which docker rm destroys, and which is why no prior engine death on this cluster left a surviving traceback, onto journald, and a stack-dump path was installed and proven against the live process.

6. Two thermal sensors, 22°C apart (measured)

One minute into the run the thermal guard tripped stage 1 at 95.4°C and capped both nodes to 1750 MHz for five minutes. At that same moment nvidia-smi reported a GPU die temperature of 73°C.

The guard reads board thermal zones; the die sensor lags them by more than twenty degrees. Anyone monitoring GPU temperature alone would never see the trip coming. Stage 1 (95°C) and stage 2 (96°C) are one degree apart, and stage 2 refuses new tailnet connections, the path this benchmark's own requests arrive on.

7. Restoring a correctness invariant cost real headroom (measured)

A one-line patch had cut a spin-wait window from 1s to 2ms, buying −4°C and ~100pp of CPU. But that window is what makes a deliberately lossy notification channel safe, and the same default governs a worker RPC read that blocks forever with no timeout backstop. It was reverted before this run, as invariant restoration, explicitly not as a fix for the observed death, whose own timeline defeats that theory.

The predicted cost arrived on schedule: sustained prefill now trips thermal stage 1 where it previously did not.

Self-inflicted

Every bug found today was found by running, not reading. Five defects in our own tooling surfaced only under execution: a synthetic-cache builder that re-randomized a shared pair after selection; an exit code masked by a pipe; a contention check written (x or 1) > 0, where a legitimate 0.0 is falsy and became 1, skipping every trial; a gate that returned "cannot verify" before reporting poison it had already found; and the 720 above.

All five looked correct on the page.

The gate

Proven by making it fail

A check nobody has watched fail is not yet a check. Each control below was constructed and executed; the gate had to produce the right verdict on all eight before the real run was permitted to start.

Acceptance signature

The final step replays the finished cache against an endpoint pointed at a dead port. A complete cache makes zero API calls; an incomplete one fails loudly against the closed socket. That removes the last piece of reimplementation that could drift from the thing it audits, verified to report 0 verifier calls on a complete cache.

Caveats

What this does not establish

Everything above is one cluster, one session, one seed. Specifically:

Not a reproduction. Different API path, thinking disabled, a local checkpoint judging trajectories generated by a different model. The published 86.5% is not the quantity being measured here and the two should not be compared decimal-for-decimal.

Single seed, 24 tasks. No error bars. A ±2-task swing is ±2.2 points, and nothing here distinguishes a genuine 2-point difference from noise.

The crash link is a suspect, not a verdict. The last engine death coincided with a structured-output request, and this run issues 1,152 of them; that makes it a plausible trigger, not a confirmed cause; the postmortem itself ranks no mechanism above another.

That is the honest version, and it is a weaker claim than "it would have looked broken." The real poisoning was also clustered, 9 of 18 entries on one task, touching 23 of 24 tasks, which costs more than uniform loss, but still only 1.1pp. The reason to fix it is that a fifth of the measurement was being thrown away silently, not that the headline would have collapsed.

The number to read first was never the percentage. It was whether RESULT: PASS sat beside it.