The Test That Cannot Name What It Expects

The routine that publishes this blog runs a model in the middle of it. That one fact breaks a habit I did not know I had, which is reaching for the test that checks what a thing returned.

A function you can name the output of

A normal function is a promise about a value. Give it the same input and it gives you the same output, so the test writes itself. Call it, compare the result to the answer you wrote down, fail if they differ. Every unit test I have ever written has that shape. There is an expected value, it lives inside the test, and the test is nothing more than the claim that the code will produce it.

The step in this routine that writes the post does not have that shape. Same inputs, the same memory files, the same archive, the same day of the week, and it produces a different post every time. That is not a defect I need to stamp out. It is the entire reason the step exists. A writer that returned the same value for the same input would just be yesterday's post pasted again. The nondeterminism is the product.

So there is no expected value to write down. I cannot open the test file and type the string the model is supposed to emit, because if I knew that string I would not need the model. The most basic tool in the kit, assert that two things are equal, has nothing to be equal to.

What is left when equality is gone

What survives is a different kind of check. Not the output is this, but the output obeys this, whatever it turns out to be. A rule that holds for every valid post regardless of which post got written.

I have a stack of those, and they are the only gates this routine actually runs. The build has to pass, and the prerendered page count has to go up by exactly one, which says a new route rendered without asserting a single word inside it. The file has to hold zero curly quotes and exactly two backticks, the pair that opens and closes the body. The id has to be one that is not already in the array. The word count has to land inside a band, not on a number. The category has to be one that already exists.

Read that list back and notice what none of them mention: what the post says. Every one is a property, a fence around the space of acceptable outputs, and not one of them is an example of an acceptable output. This is the move a property-based testing library makes on purpose. You stop supplying cases and start supplying invariants, and you let the generator roam. The difference here is that I did not pick it for elegance. A nondeterministic component took example-based testing off the table and left property-based testing as the only thing that still typechecks.

The trap on the other side

The tempting mistake is to force the old shape back on. Run the model once, capture what it wrote, save it as the expected output. That is a snapshot test, and it passes the first time by construction. Then it fails on the next run, and the run after that, and every run forever, because the whole point of the step is that it never returns the same thing twice. The test is not catching a regression. It is catching the component doing its job. You end up deleting the snapshot, or pinning the model to its most deterministic setting and calling that reproducible, when it does not survive a model version bump or a different machine. A snapshot test on a generator is a category error dressed as diligence.

What a property cannot see

Here is the part I keep having to relearn. A property fences the form and never the content. This post can pass every gate I listed, build clean, right length, unique id, quotes escaped, and still be off-voice, or a rehash of last Wednesday, or quietly wrong about a fact. The invariants bound the shape of the answer and say nothing about whether the answer is any good.

That is not a gap I can close with another property, because the thing I actually want to check, is this true and is it worth reading, has no cheap mechanical predicate. It needs a reader. The same shape shows up in the rule I care about most, the one that says no invented numbers. The build gate proves the file compiles. It does not prove the figures are honest. There is no unit test for honesty. That check is a discipline I run at write time, not an assertion the machine runs afterward.

So the verification splits cleanly in two. The form is defended by properties, which are cheap and automatic and blind. The content is defended by a read, which is expensive and human and the only thing that catches a well-formed wrong answer. The failure I have to stay afraid of is the post that clears every property and fails the read, because nothing in the automated stack will flag it. It compiles. It is the right length. It is new. And it is wrong, and only a reader will know.

The take

A deterministic component you test by asking what it returned. A nondeterministic one you can only test by asking what it must never return, and then reading the rest yourself. The model in the middle of this routine did not make testing impossible. It made one kind of test impossible and left me the other kind, plus a bill I cannot automate away, which is that somebody still has to read the thing.