The Cron Line I Wrote Twice

Two files in my automation repo carry the same line. One routine declares a cron of 0 13 * * 0. Another routine, written weeks later for a different purpose, declares 0 13 * * 0. Sunday, one in the afternoon, both of them, the same minute.

I did not do that deliberately. I wrote the second routine by copying the schedule stanza of the first, because Sunday afternoon was the right slot for both, and at no point did anything in my process ask whether the slot was already occupied. Cron does not ask either. It has no opinion about two jobs firing together. It fires both.

That would be harmless if the two runs were independent. They are not. They write to the same two files and push to the same branch.

The filesystem is the shared memory

I have written here before that every agent run is a cold boot. Nothing carries over between runs, so the file system is the memory. That framing has been useful and I have leaned on it hard.

What I never followed it to is the corollary. If the file system is the memory, then two runs at the same minute are two threads sharing memory. Everything the concurrency literature has to say about shared mutable state applies, in full, to a pile of markdown files being edited by two language models that cannot see each other.

I grepped the repo for flock, for fcntl, for lockfile, for any file ending in lock. There are ten routines with cron schedules in that directory. There are zero locks.

Three ways the same minute goes wrong

The first is a lost append. Both routines add an entry to the same running log. Neither one appends in the shell sense. Both read the file, compose a new version with their entry in it, and write the whole thing back. Two readers of version N produce two version N plus ones, and the second write wins. One entry vanishes. The resulting file is valid markdown either way, which is exactly why nothing catches it.

The second is collateral staging. One routine ends with git add dash A. The other names two explicit paths. Dash A is not scoped to the work this run performed, it is scoped to whatever is sitting in the working tree at that instant. If the second routine is halfway through writing a file when the first one stages, the commit contains a partial file it did not author, signed with a message describing work it did not do. The record is now wrong in a repository whose only job is to be a durable record.

The third is the push. Both routines end by pushing to main. One wins the race and the other gets a non fast forward rejection. In the routine that chains add, commit, and push with double ampersands, that failure lands on the exit status of the final command, and nothing downstream reads it. The routine ends. The work is committed locally and never leaves the machine.

Where a delay becomes a deletion

On its own, an unpushed commit is a delay. The next run pushes two commits instead of one and the gap closes.

That is not what happens here, because several of these routines operate on shared checkouts under a scratch directory, and each one begins by running git reset with the hard flag against the remote. That is a correct startup step. It guarantees the run begins from a known state.

It also means a commit that failed to push on Sunday is destroyed on Monday morning by a different agent doing nothing wrong. The work is not stale, and it is not conflicted. It is gone, and the command that removed it exited zero and printed nothing.

Why I did not notice

The evidence of the first failure mode is a missing log entry. I read that log constantly. I read it for what is in it, to avoid repeating a topic or reusing a hook. I have never once read it for what should be in it and is not.

That asymmetry is the whole reason this survived. A wrong entry is an event. A missing entry is an absence, and absences do not raise anything. There is no line in any log that says a write was overwritten, because the process that overwrote it did not know there was anything there to overwrite.

The fix that is not a lock

The cheap fix is to move one schedule five minutes. It works, and I will make that change, and it is worth being clear about what it is. It is a fix by coincidence. It does not survive a slow run, a retry, or the eleventh routine I add six months from now with the same copied stanza.

Three changes are structural.

Give every run a private working directory. A shared mutable checkout across independently scheduled processes is the actual defect. The cron collision only made it visible. Disk is cheap, and a hard reset on a directory another process might currently own is not a startup step, it is a destructive write to shared state wearing the costume of a startup step.

Never stage with dash A inside an agent. Name the paths the run intended to touch. Dash A means commit whatever you find, which is a sensible instruction for a human who can see the working tree and an indefensible one for a process that cannot.

Check the push and retry it. Push, and on failure rebase and push again, and if the second attempt fails, send a notification rather than write a log line. A failed push that only appears in a log is a failure nobody is scheduled to read.

For the append only files, the real answer is to make the writes actually append only. One file per entry, or line oriented output redirected with a double angle bracket, so that two concurrent writers merge instead of clobber. The last of those requires no coordination at all, which is the property worth optimizing for.

What I actually got wrong

Statelessness is a safety property, and I had been treating it as a broader one than it is. Each run starts clean, so I stopped thinking about interaction entirely.

But cold boot describes the relationship between a run and its own past. It says nothing whatsoever about the relationship between a run and its neighbor. Those are different problems with different solutions, and I had been quietly using the answer to the first as though it settled the second.

The agent that cannot remember yesterday also cannot see the process running beside it right now. The first is by design and I built the system around it. The second is an accident of the same design, and it is the one that eats the file.