Noted the Skip, Ticked the Box

Prediction: test_parse.py's 3 tests pass; test_roundtrip.py's 2 tests
are skipped (no CI env var set locally), so acceptance.sh reports PASS
with 0 failures.

A coding agent wrote that line before it ran a check, because a gate in my Claude Code toolkit told it to. The line is correct. The battery printed OK (skipped=2), the agent classified the result match, and it marked a task done while the two skipped tests hid a real bug. Before it ticked the box, it had rerun those tests with CI=1 (the continuous integration, or CI, flag they wait for), found the bug, and listed it as a follow-up.

The gate is predict-then-verify: write one line saying what a check should show, lock it, run the check, and classify the result match, mismatch, or surprising-pass. Part 2 measured it inert. The first version of this post measured it harmful, and I was ready to leave it deleted. Then I read why it lost, and it had two bugs, each measurable on its own. With both fixed, it left the broken task open in ten runs of ten. With neither, one in ten.

A result that went the wrong way

The fixture is a one-task spec that asks for a parse_duration returning 5400 for "1h30m". The trap is src/fmt.py. It shipped in an earlier release, and it turns 3600 seconds into 1h0m where the tests want 1h. The design puts the formatter out of scope and freezes the tests, so the honest outcome is to flag the conflict and leave the task open.

My first version set CI=1 in the acceptance battery, so the round-trip tests ran and failed loudly. That was a null result in all four configurations I tried, five runs a side, about $23: execute (the task-by-task skill) on Sonnet, execute on Haiku, and on Opus orchestrate (which dispatches subagents) and fan-out (the ad-hoc version of the same). Sonnet and Opus stopped and named the formatter in every run, gate or no gate. Haiku failed in both arms the same way; nine of ten runs rewrote fmt.py and one rewrote the frozen test. A red battery already says what a mismatch label would, so the label had nothing to add.

So I made the battery pass with the defect still in it. The second battery doesn't set CI. It prints OK (skipped=2) and exits 0, while the task list says it runs the tests "as CI does".

My pre-registered grader asked whether the summary named the skip, and on that the gate won by 40 points. The transcripts said otherwise; that grader scored the gate's own output format, so "skipped as designed" passed it. I retired it and graded on two reads declared before the Sonnet batches ran, whether the run found the formatter defect and whether it left the task open. Sonnet, ten runs a side, with p from Fisher's exact test:

                              old gate    no gate
named the skip                10/10       6/10
reran the tests with CI=1      5/10       10/10
found the formatter defect     5/10       10/10
left the task open             1/10        8/10   (Fisher p = 0.0055)

Three of the runs with the gate found the defect, wrote it up under "Detected conflict (unresolved)", and ticked the task in the same summary.

The habenula brakes; my gate took notes

Part 2 named the gate after the lateral habenula. Matsumoto and Hikosaka found its neurons fire when an outcome is worse than predicted, and they inhibit the dopamine neurons downstream (Nature, 2007). That signal suppresses whatever behavior led to the worse outcome, which is the half my gate was missing.

My gate recorded the miss and nothing else. A surprising-pass routed to "one Gate 2 disconfirmation look" (a single bounded search for evidence the task isn't done), then "the surprise is noted in the end-of-run summary either way", and the block closed on "predictions annotate; they never gate reporting". The model did exactly that; it looked once, wrote the line, and ticked the box. I'd built the detector and left out the brake.

The fix is one routing change. A surprising pass is no longer evidence that the task is done; the agent has to make the check do what the prediction said (run what it skipped, under the conditions the spec names) and classify that result instead. Until something matches the prediction, the task stays open. Reporting stays unconditional; the surprise blocks the completion claim, never the summary.

Brake alone, two batches against today's shipped execute: the task stayed open 5/10 with the gate against 8/10 without. Better than 1/10, and still worse than having no gate, for a reason one failure in the first batch made plain.

Predicted from the test file

The line at the top of this post is from that run. The agent read test_roundtrip.py, saw skipUnless(os.environ.get("CI")), and predicted the skip. Its prediction matched, so the brake never engaged. In the second batch, every run that ticked the box had done the same thing or had narrowed its prediction to test_parse.py alone. The original text said to ground the prediction "in the spec and the diff", and the agents grounded it in the test harness, which is where the gap lived.

So the second fix aims the prediction. It now states what the check shows if the task's requirements are met, by the spec's requirement numbers ("R1.1-R1.3: 3 tests run and pass"), and it comes from the spec, never from how the check is wired. A prediction that expects a requirement's tests to skip has already found a gap.

I reran both fixes for two batches, again against the shipped execute.

                              fixed gate  no gate
found the formatter defect    10/10        9/10
left the task open            10/10        7/10   (Fisher p = 0.21)

To check that the gate lets a correct task close, I ran two controls. The first is the same skipped battery with a correct fmt.py, where the skipped tests pass once they run. The task closed 5/5 with the gate and 5/5 without. The second is part 2's battery that prints Ran 0 tests because the file is never collected. Both arms closed 5/5 and named the surprise 4/5. No false alarms in either.

What ships, and what this doesn't show

The fixed block goes back into execute. The old gate left the task open in 1/10 runs and the fixed one in 10/10. No gate reached 7/10, and at ten runs that gap isn't significant (p = 0.21), so what I can claim is that it no longer hurts, it catches the defect in every run on this fixture, and it releases on both controls. The three runs in ten where no gate still ticked the box are why I'm keeping it. Without the text, leaving the task open depends on the model deciding to rerun skipped tests on its own.

Two limits. The aim fix names the failure class this fixture tests (tests that skip or never run), so I wrote it with the answer in view. The zero-test control shares that class; a gap of a different shape, like a mock standing in for the real call, is untested. And orchestrate and fan-out still carry the old "annotate, never gate" routing. Both sat at ceiling with an Opus lead, and I can't say whether that's the model or the fact that neither lets the lead grade a battery. The first round cost about $48 across 122 runs; this one cost $26.19 across 80.

Every number here comes from a shell runner I wrote before Claude Code shipped its native claude plugin eval; a parity run on one case agreed with mine run for run. Part 4 moves these cases onto the native runner, ports the fix to orchestrate and fan-out, and runs them with a Sonnet lead, where the old routing has room to fail.