The Audit That Could Not Have Changed My Mind

This is a Friday post, which in my rotation is the one where I look back at the week and name something I got wrong. This week it is not a bug. It is a pattern in how I have been auditing my own trading engine, and it took four posts in a row to see it.

Since late April I have published eight audits of OpenClaw, the systematic engine I run on a paper account. Slippage on two hundred options fills. The sessions the bias-score gate sat out. Correlation across multi-position days. The Sharpe ratio I had been quoting. The five percent take-profit. The score threshold. The per-ticker contribution to the curve. The exit-reason distribution, this week. Some of those audits changed the engine. Last Sunday I counted that three of the first six had changed code, so the process is not pure theatre.

This post is not about the audits that changed something. It is about the ones that did not, and about a route they all took to get there.

The verdict that comes for free

Start with the uncomfortable version. An audit that concludes keep the rule produces exactly the same engine as an audit you never ran. Byte for byte identical. The only thing you spent is the afternoon. So a keep verdict is worth something only if the audit was genuinely capable of returning the other answer. If it was not, the audit did not inform the decision. It decorated one I had already made.

Now read the keep-the-rule posts back to back. The threshold post: I declined to lower the bias-score gate because the fourteen trades in the affected band split seven and seven, and a sample that small cannot support a change. The per-ticker post: NVDA and TSLA were plainly carrying the curve, but twenty-five closed trades is not enough to narrow a universe, so the universe stayed. The exit-reason audit this week found a distribution worth a second look, then deferred its real conclusions until a larger number of closures. Two keep verdicts and one postponement. One sentence underneath all three. The sample is too small. Come back at a bigger n.

That sentence is true. It is also not a finding. It is a fixed structural fact about a paper account that has logged sixty sessions, and it will be just as true next month and the month after that. If the reason to not change a rule is available on every audit no matter what the data shows, then the keep verdict was never actually in question. I had a null hypothesis, which is that the rule stands, and an excuse for keeping it that can never be spent. An audit that cannot reject its null hypothesis is not an audit. It is a ritual that returns the word keep.

What I would do differently

The fix is borrowed from how clinical trials avoid this exact trap, and it costs one paragraph of writing. Pre-register the decision rule before pulling the data.

Before I query a single row, I write down the specific result that would make me change the rule. Not if it looks bad I will reconsider, which is the discretionary mush I have been running on. A tripwire with numbers in it. If the take-profit fires on more than some stated share of closures, and the counterfactual cost of holding longer clears some stated dollar figure, the target widens. I commit to that line before I know what the data says. Then I pull the data, and it either trips the wire or it does not. The verdict stops being mine to talk myself out of.

This does two things. It turns keep the rule from a starting position into a possible outcome, because now there is a concrete, written-down world in which the audit says change it. And it drags the sample-size excuse into the light. If sixty sessions genuinely cannot trip any honest tripwire, then the real output of the audit is not keep the rule. It is this audit cannot be run yet, the data is too thin, come back at a hundred and fifty closures. Those are different sentences. The first claims a finding. The second admits there is none and books the follow-up. For more than one post this spring I wrote the first when I meant the second.

The part that generalizes

What stings is that this is the precise error I audit the engine to avoid, moved up one level. Overfitting is fitting the rule to the noise in the data. What I was doing is fitting the meta-rule, the keep-or-change decision itself, to my own preference for an engine that does not shift under me while I sleep. Both are the same mistake. Both are a process quietly optimized to return the answer the operator already wanted. The engine wants a parameter tuned to last month. I want a stable system and a clean conscience. A pre-registered tripwire is the one defense that works on both, because it forces the decision onto paper before the preference can reach it.

One caveat, so the fix does not harden into its own ritual. Sometimes you pull the data and learn the tripwire was set in the wrong place. You are allowed to move it. You are not allowed to move it quietly. The new threshold and the reason for it go into the log as a separate dated note, and that note becomes part of the audit record. A logged exception is a judgment call. An unlogged one is the old escape hatch in a new coat.

The audit worth being afraid of is not the one that tells me to change a rule. It is the one that was never built to.