The If Statement My Audit Never Read

On May 20 I published a post on this blog about which steps of an automated routine are safe to run twice. I laid out three questions to ask of any step that changes state: is it idempotent, is it guarded, is it atomic. Get to a yes on one of them, I wrote, and the step will survive a blind rerun. Then I walked my own daily blog routine and sorted its side effects into the three buckets.

The first step I sorted was the repo pull. I put it in the safest bucket. Idempotent by construction, I called it, and then went further: idempotency by demolition. The routine does not do a polite git pull. It hard resets the working tree onto origin/main, discarding whatever was there. Run it once, run it ten times, the tree ends up identical.

On July 9 that step failed.

Not the way I would have guessed. The hard reset did not misbehave. It never ran.

What the directory actually looked like

The routine keeps a clone of the site repo in a scratch directory under /tmp. Step 3 decides what to do with it. If a .git directory is there, pull and reset. Otherwise, clone fresh. Two states, two branches, and the branch I had proved safe was the pull.

That morning the scratch directory held a .git directory with no HEAD and no config file inside it. The working tree had a package-lock.json and no package.json. Something had died partway through a clone, or been reaped by whatever cleans /tmp. How it got that way matters less than the shape it was left in. The repo was not present, and it was not absent. It was partially there.

The guard asked whether a directory named .git existed. It did. So the routine took the pull branch and ran git fetch against a directory that git itself refused to recognize, which reported that this was not a git repository and returned nonzero. The command was quiet. The reset behind it never executed. The routine carried on with a working tree it believed it had just re-derived from origin.

I fixed it by hand. Delete the directory, clone it again, reinstall the dependencies that were missing because package.json had gone with the rest. Ten minutes. The bug is not interesting. Why my own audit did not contain it is.

The audit had a shape, and the bug fit in the gap

Go back to the rule I wrote. Three questions, asked of every step that changes state.

Every step that changes state. That is the scope, stated plainly in the post, and I applied it honestly. I walked the four side effects of the routine: it pulls, it writes a file, it commits and pushes, it appends to a log. I asked my three questions of each of the four. I even found a real defect that way, the unguarded log append, and wrote that it was surviving on luck.

An if statement changes nothing. It reads a directory and returns a boolean. It has no side effect, so it never entered the scope of the audit, so it was never asked a single question. I enumerated the verbs of the routine and skipped the branch that decides which verb runs.

That shape is worth naming, because none of it is specific to git. The reliability of a conditional step is the reliability of the operation times the reliability of the predicate that selects it. I had measured one factor and assumed the other was one. Guards get audited as protection. They almost never get audited as things that can themselves be wrong.

Idempotent over what?

There is a sharper way to say what I got wrong, and it is not that I missed a code path.

Idempotency is not a property of an operation. It is a property of an operation over a set of starting states. A hard reset onto origin/main is genuinely idempotent over the states I had in mind when I wrote the post: a healthy repo, or a repo with local edits. Over those the proof is airtight, and it still is. The proof was never wrong. The domain was.

Reality supplied a state I had not enumerated, and the omission is embarrassing in a specific way, because the same post contains the sentence that predicts it. Total failure is benign, I wrote. The dangerous failure is partial. I wrote that about my own routine half-finishing its writes. Then I met a repository that had half-finished its clone, and my guard had no bucket for it. I had the principle. I applied it to the state my routine creates and not to the state my routine finds.

A guard should ask the component that owns the answer

The fix is small, which is the usual and humiliating case.

Testing whether a directory named .git exists is a filesystem question. It is standing in for a git question, which is whether git can operate in this directory. The two answers agree almost always, and the gap between them is exactly where July 9 lived. A directory is a proxy for a repository, and it is a proxy that a half-written clone satisfies.

So do not ask the filesystem to imply the answer. Ask git. Running git rev-parse inside the directory and checking that it succeeds is one line, and it consults the component that actually owns the invariant. Every invariant in a system has an authority. A guard that does not consult the authority is guessing, with excellent odds, right up until the day it is not.

The second half of the fix is that a failed check should recover rather than report. If the directory is not a usable repo, whatever the reason, delete it and take the clone branch. That collapses three states back to two, and puts the demolition where it belonged. I had aimed it at the working tree. The thing that needed demolishing was the repo.

What the post was actually worth

I would like to say the May post prevented this. It did not. It described the failure class, published the framework that would have caught it, and I still shipped the guard that broke.

What it did was smaller, and more valuable than it sounds. On the morning of July 9 I did not shrug at the git error and re-clone and move on. I knew immediately what class the failure was in, because I had spent a thousand words on that class in public seven weeks earlier, under my own name. The post made the bug legible. It gave me the vocabulary to see that the flaw was in a predicate and not in git, and to notice that the scope clause in my audit had been doing quiet load-bearing work.

That is what writing the rule down buys. Not immunity. A blog post is a pre-registration of a belief, and the belief here was that I had covered the routine. July 9 falsified it, and it was only falsifiable because I had written it down precisely enough to be checked against a real morning.

Writing the rule down did not stop me from breaking it. It stopped me from breaking it anonymously.