The Ablation I Haven't Run

Forty-three skills, twelve steering docs, ten hook scripts. That is the prompt surface of a Claude Code toolkit I maintain, counted with wc this week rather than recalled, and fifteen of the skills run past 300 lines. The largest, a PR reviewer, is 689 lines and 5,442 words. The day after Opus 5 shipped, Boris Cherny told a Y Combinator audience to delete their CLAUDE.md, their skills, and their hooks every six months and see what the model does without them. The advice was aimed at people using Claude Code rather than building on it, and it ended on "it might surprise you" (video). His team had just cut most of Claude Code's own prompt.

I can't tell whether that would help me, and the reason is narrower than "my repo is big". Following the advice means deleting a section, running the skill, and deciding whether what came back is worse. Nothing in my repo can make that decision. There is no behavioral eval for a single skill, so a run after the deletion and a run before it are two sessions that went differently, and I have no way to say which of the differences the deletion caused.

Anthropic measured the cut against a suite I don't have

The sentence that makes the advice work is the one that drops off when people repeat it. Anthropic's own write-up says the team "removed over 80% of Claude Code's system prompt for models like Claude Opus 5 and Claude Fable 5 with no measurable loss on our coding evaluations" (Shihipar, July 2026). Deleting 80% of a prompt takes an afternoon. "No measurable loss" took a suite of coding evals, run before and after, and the advice as people repeat it keeps the deletion and drops the suite.

Two long unattended runs came up in the same talk, and each had its own check. Bun's pre-existing test suite verified an 11-day rewrite of Bun's Zig code into Rust; a pixel-by-pixel comparison of the old app against the new one verified a longer Electron-to-Swift port. They show that a loop with a check in it lets a model run for days, and they say nothing about what the check is for a skill whose output is a review.

I wrote the part of my repo's CLAUDE.md that says how to validate a skill change:

Skills don't have a unit-test harness — they're prompts. To validate a change:

1. Load the plugin in a Claude Code session (`/plugin` → install from this
   directory) or symlink your dev copy.
2. Invoke the skill by name (`/<skill-name>`) and walk a real scenario
   end-to-end.

Walk a real scenario end-to-end. One run. Judged by the person who made the change, against no recorded run from before it, and two sessions on the same unmodified review prompt already produce different reports. A green run after a deletion is n=1, compared against n=0 from before.

Five batteries a skill can pass without ever running

When I first went looking I counted zero evals for the skills, which was wrong, and the correction sharpens the claim. Five of the fourteen feature specs in the repo carry an acceptance.sh, a shell script CI runs as that spec's gate, and each asserts something about a skill. One requires a blockquote about never suppressing findings to appear verbatim in the nine skills that inherit it; another requires a new skill to pin model: and effort: in its front matter and to scope its Bash matchers narrowly instead of allowing bare Bash. Those are real regression gates; cut the wrong paragraph and a green battery goes red.

All five test the file and none of them runs the skill. Each one greps a SKILL.md for a paragraph and fails when the paragraph is gone, which is the failure they were written for: someone trims a guardrail and the trim ships. The failure Boris's loop needs to see is a different one. If I delete a section and the review the skill writes gets worse, every battery stays green, because the paragraphs they pin are still in the file.

The hooks are tested by running them. There are ten of them, eight shipped with the plugin and two local to the repo, all shell scripts. Ten test scripts run them against fixed inputs with known expected output; pnpm hooks:test runs every one locally and a CI job runs the same command on every push. The hooks are cheap to test, and ten scripts test them. The skills are the product, and what covers them is five text assertions that never start a session.

So the repo answers "what can I delete without breaking CI" strictly, and it cannot answer "what can I delete without changing what the skill does" at all, and the second question is the one Boris is asking. Getting the coverage wrong in either direction costs me a wrong plan. When I assumed zero coverage I started rebuilding text assertions that already existed; assume instead that the batteries measure behavior, and the plan deletes a paragraph one of them pins and hears about it from CI.

Even without a baseline, part of the inventory is already settled. Some lines are facts about the repo that no model can read out of the code, and a smarter model is no better at guessing them. One hook tells every session that main is release-protected and that a bot writes the version bumps. The task files carry a small set of HTML-comment markers this repo made up, so somewhere has to say what each marker means. A reviewer skill holds a table of which standards doc to load for which kind of file. Those three are context, and they survive every model upgrade.

The candidates for deletion are the corrections, and after a full read there are fewer than I expected. Two, in fact. "Read the surrounding code before editing (never edit blind)" occurs once repo-wide, in a 588-line file. A five-line fallback chain for finding the diff base spells out what any model that can run git rev-parse origin/HEAD works out on its own. Sorting gets me as far as which six lines to test first; measuring what pr-review produces once they come out still needs the baseline.

What the reviewer refused to let me cut

I had a second reviewer fact-check the audit behind this post, and the corrections above are theirs. They also filed a dissent, which changes who has to prove what.

Boris's loop is delete, run, watch the model stumble, and add the line back, and every step assumes the person deleting can see the stumble. His team runs that loop on a product they ship, against sessions they observe. I ship a plugin. I never see the installer's session and I don't know their model tier. Once they have pulled a version with a guardrail missing, the line I add back reaches them on their next update, after whatever it was guarding against has already happened.

A redundant instruction costs tokens in every session that loads it. A missing one costs an installer a wrong diff reviewed as correct, or a security finding dropped without a trace. The reviewer's position is that the asymmetry sets the bar. A deletion in a distributed plugin has to clear "we verified before shipping that removing this doesn't bring back the failure it was added for", and that bar is the eval I don't have.

The reviewer refused two specific cuts from my first pass through the inventory, and I had bucketed both as scaffolding for a weaker model. The non-suppression blocks are the most verbose text in the review skills, which is why I reached for them. They are also the exact wording a passing acceptance battery asserts, and the repo added them after the current model read a stated severity filter as permission to drop findings entirely. Cutting them to save five lines deletes the evidence that the last ablation went wrong.

The other was the halt in the skill that works a spec's task list, after every third completed task, which I had read as a verification checkpoint. The skill states its own reason: continuing spends budget only the user can authorize. That halt survives any model upgrade, and I would have cut it for being one of the longest paragraphs in a 588-line file.

One skill, one fixture, five runs

My cost estimate got pushed too. I had called an eval for the 689-line PR reviewer afternoon-sized; the reviewer called it two to three days. The repo has no eval harness at all today, the skill fans out subagents so one run costs minutes and dollars, and n >= 5 per variant is the floor before a difference means anything. The built-in runner doesn't close that gap yet. On Claude Code 2.1.235, claude plugin eval prints its full option set and then, on any real invocation, "currently in early access", and runs nothing; the case format is authorable today and the runner is gated on the account side.

Boris's specific complaint about prompts like mine was choreography: "do one, then two, then three, then four" handed to a model that does better with a task, some guardrails, and an exit criterion. pr-review is that shape. Its numbered steps are the sections a baseline would let me remove one at a time.

The pilot is smaller: the smallest reviewer-shaped skill in the plugin, review, at 68 lines, single file, no fan-out. The fixture is a small directory with planted issues (a hardcoded credential, an unpinned GitHub Action, a wildcard IAM resource) and at least one lure. The lure is a wildcard on a test-only resource that a careless review would flag, so the eval measures precision and not just recall. The assertions are pass or fail on catching each planted issue and on leaving the lure alone. Then the baseline: the skill unmodified, five runs, scores recorded. Then, one section at a time, five runs each, to find out which sections the score depends on.

Nothing comes out of any skill before that baseline exists. Six months from the talk is late January, and by then I want the number I don't have now: five unmodified runs of review against a fixture I haven't built yet. Part 2 is that harness, the fixtures, and what the first ablation measured at five runs a side.