Three Ways My Historical Data Was Written After the Fact

The usual suspect when a backtest does not survive contact with live trading is the strategy. Too many parameters, too few trades, a threshold tuned until the equity curve looked the way I wanted it to. That is the failure mode everybody writes about, and it is one I have written about here more than once. It also has a defense. Hold out a window, run the thing once, accept whatever comes back.

The failures I have a harder time defending against sit one layer below the strategy, in the data itself. Cross validation does not catch them, because the same corruption is present in the training window and the holdout window in equal measure. The test is blind to it by construction. Three of these are in nearly every historical dataset I have pulled, and all three share a property: the numbers were written after the dates they claim to describe.

The universe was picked with hindsight

The most common way to build a backtest is to take the current members of a broad index, pull ten years of daily history for each, and run a signal across the panel. Every step of that is reasonable, and the result is not a sample of the market ten years ago. It is a sample of the companies that made it to today.

Names that were delisted, acquired, or went to zero never enter the file. They left for reasons that are not independent of their returns, which is the entire problem. Membership runs the other way too. A stock generally joins a large index after it has already performed, so the constituent list is a momentum screen applied with perfect foresight. Backtesting a strategy across it is closer to grading an exam with the answer key stapled to the front than to testing an idea.

The fundamentals were revised

Earnings figures get restated. Macro series get revised, sometimes for years after the first print. My database holds the current value, which is the accurate one, which is exactly why it is the wrong one.

A signal that buys on a positive earnings surprise, computed against a restated figure, is trading on a number that did not exist on the day of the trade. Nobody could have acted on it. For the macro series the gap is worse, because the first print and the final revision can differ enough to flip the sign of the surprise, and the sign is the whole signal. The backtest is not optimistic here so much as fictional. It describes a trader with access to a later version of reality.

The prices were rewritten last night

This one took me longest to see, because it looks like a convenience rather than a distortion. An adjusted close is not a price that traded. Every dividend and every split retroactively rescales the entire history behind it, so the series I download today is arithmetically different from the series I downloaded last year for the same dates.

Two things follow. Any rule stated in dollars, a price filter, a minimum tick, a round number level, evaluates against a value that never printed on the tape. And a backtest run against a live data source is not reproducible. The same code, over the same date range, returns a different number six months later, and nothing anywhere raises an error. It quietly disagrees with the version I wrote down.

What I can actually fix

The honest counter is that clean point in time data is expensive, and vendors charge for it precisely because reconstructing what was known on a given morning is genuinely hard. The conclusion that follows is not to stop backtesting. A backtest with known biases beats an intuition with unknown ones.

The cheap fix is the third problem, and it is a snapshot. Freeze the input file, hash it, and store the hash next to the result. That does not make the data correct. It makes disagreement loud instead of silent, which is the difference between a number I can defend and a number I merely remember producing.

The first two I mostly cannot fix without paying for data I do not have. What I can do is stop pretending the direction is unknown.

The sign is the tell

None of these three errors makes the backtest look worse. Survivorship deletes the losers. Restatement swaps the noisy number for the accurate one. Adjustment smooths the series. Three independent defects in three separate layers of the pipeline, and every one of them flatters the result.

A bug with a random sign is a bug. It shows up as noise and it washes out as the sample grows. A bug whose sign is always favorable is not a bug, it is a selection effect, and it does not wash out with more data. It compounds with it. The longer the backtest window, the more delisted companies are missing from it and the more revisions have been folded in.

So the number at the bottom of the report is not an estimate of what the strategy would have returned. It is an upper bound on it. I have been reading those numbers as estimates for a long time, which means every one of them was too generous and I do not know by how much. That last part is the one I would fix first if I could.