The Near-Misses My Engine Forgets

My trade log is a JSON-lines file. One line per trade, one record per state change. Entry timestamp, exit timestamp, ticker, direction, entry price, exit price, P and L, exit reason. Each field exists because, at some point, I needed to ask a question and the answer required that column.

The schema is honest about what fired. The trailing stop fired. The take-profit fired. The five-day time stop fired. The behavioral 200-SMA rule blocked an entry. Each of those events has a row, a tag, and an audit query that maps cleanly back to the rule that produced it.

The schema is silent about what almost fired. There is no column for the gate that came within five basis points of blocking the trade. There is no column for the rule that was the runner-up to the actual exit reason. There is no record, anywhere, of the entry gate that was closest to refusing this trade at 09:45 on the morning it was placed.

That silence is not free.

What a near-miss is, mechanically

An entry signal goes through a stack of gates before the engine sends an order. The bias score must clear 65. The behavioral checks must pass, not averaging down, not within ten days of a stop-out on the same ticker, not too far below the 52-week high. The risk gates must pass, daily-loss check, concurrent-position cap, position-size limit, entry-time cutoff. The signal must not be the third in a correlated cluster.

Each gate is a numeric threshold or a boolean. For every signal that does enter, every one of those gates returned pass. The trade-log row captures the trade. It does not capture which of the pass votes came closest to a fail vote. A signal with a bias score of 65.2 and a signal with a bias score of 84.7 produce identical entry-side rows.

That information is not unrecoverable from a deep dive into per-day scheduler logs and per-tick data, but it is also not queryable. It does not exist as a field anyone, including me, would think to filter on. So it does not get audited.

The audit I would actually run

Once a week, group every entered trade by the gate with the smallest pass margin. Plot the distribution.

If one gate dominates the near-miss list, that gate is doing more work than I think, and the trades it almost refused are the trades the engine is most marginal on. Their P and L is the right place to look for the next rule change. They are the population the rule was written to filter; if I am noisy on them, the threshold is in the wrong place.

If no gate dominates, if near-miss is spread evenly across gates, then no gate is binding in the marginal case, which means the rule that is actually deciding which trades happen is upstream of the gate stack, in the signal generator itself. That is a different conversation about a different file.

I cannot run either version of that audit today. The data is not in the log.

The one-field fix

Add one column to the entry record. Call it near_miss_gate. At entry time, after every gate has voted, compute the gate with the smallest pass margin and write its name. Optionally a second field, near_miss_margin, with the numeric distance from the threshold.

A string and a float per row. Three lines in evaluate_trade(). The cost of the change is small. The audit it unlocks did not exist before; the schema was the bottleneck, not the analysis.

Why the schema is the rate-limiter

This is the part worth writing down. I have spent the past month running audits. Exit-reason audit. Per-ticker contribution audit. Slippage audit. Threshold-defense audit. Every one of them was bounded above by what the trade log thought to record at the time the rule was written. The exit-reason audit only ran because exit_reason was already a field. The per-ticker audit only ran because ticker was already a field. The audit I cannot run is the audit whose field someone, two years ago, did not think to write down.

The schema is the cap on the question space.

I am going to add the column this weekend. The audit will not run for another month, because n needs to be at least a hundred entries before any distribution argument is anything other than a story. But the data starts collecting today, and that is the only move that changes anything a month from now.

A column that does not exist is not a missing audit. It is a missing year of audits.