The Hundred Backtests I Do Not Remember Running

Three days ago I wrote that it takes about four years of returns to trust a Sharpe ratio of 1.0. That post assumed one strategy, held up to the light, and asked how long its track record had to run before the number meant anything. It left out the more dangerous variable, which is not how long I ran the strategy but how many strategies I ran to find it.

The maximum is not the mean

Take a coin-flipping strategy. No edge, true Sharpe of exactly zero, pure noise. Run it for four years and its measured Sharpe will not land on zero. It lands somewhere near zero with a standard error of roughly one over the square root of the years, so about one half over a four-year window. Sometimes it prints positive, sometimes negative. That is one strategy.

Now run a hundred of them. A hundred worthless coin-flippers, each with a true Sharpe of zero, each measured over the same four years. Keep the best one. What does the Sharpe of that winner look like?

This is a question about order statistics, and the answer is not subtle. The expected maximum of N independent draws from a bell curve grows roughly like the square root of twice the natural log of N. For a hundred trials that factor is about three. Multiply it by the half-point standard error and the best of a hundred zero-edge strategies shows an apparent Sharpe of around one and a half. Over four years. From nothing.

Sit with the collision. The earlier post said a Sharpe of 1.0 over four years is a genuinely good strategy that just clears the bar for significance. This one says I can manufacture a Sharpe of 1.5 over the same four years by trying a hundred strategies that have no edge at all and reporting the winner. The track record is the same length. The number is higher. And it is pure selection.

Ten is enough to fool you

A hundred sounds like a lot until you count honestly. Take ten. The expected-maximum factor for ten draws is about two and a bit, so the best of ten zero-edge strategies over four years lands near a Sharpe of one. Ten tries. That is one afternoon of changing a lookback window from fourteen to twenty-one, swapping RSI for MACD, moving a stop from two percent to three, and keeping whichever version backtested best.

Nobody thinks of that afternoon as running ten hypothesis tests. It feels like refining one idea. But every parameter I nudged and re-ran was a fresh draw from the same noise, and the version I kept was the maximum of those draws, not a sample from them. The refinement was the overfitting. I was not improving the strategy. I was searching a pile of noise for its luckiest corner and calling the search progress.

The number I cannot report

The clean fix is to deflate: haircut the winning Sharpe by the expected maximum of N, report what survives. Bailey and Lopez de Prado built exactly this, the deflated Sharpe ratio, and the machinery is sound. The problem is not the formula. The problem is N.

To deflate honestly I have to know how many strategies I tried, and I almost never do. The trials I can name are the floor, not the count. Every parameter I swept, every variant I eyeballed on a chart and abandoned before I bothered to log it, every impulse to just check what happens if is a hypothesis test that inflated the maximum I eventually kept. The ones I discarded quickly count exactly as much as the ones I ran to completion. This is the garden of forking paths, and it means the true N is almost always an order of magnitude larger than the trials I remember running.

You cannot deflate by a number you refuse to count. And the act of searching does not feel like counting, which is precisely why the count runs away from you.

What I actually do about it

Three moves, in order of how much I trust them.

The first is to assume N is bigger than my memory of it, by a lot. If I can name ten tries, I deflate as though there were a hundred. The haircut is crude, but a crude haircut on an honest N beats a precise one on a fantasy.

The second is to quarantine data the search never touched. A slice of history the tuning loop was not allowed to see, tested once, at the end. That word once is the load-bearing one. The first time I look at the out-of-sample result and go back to adjust the strategy, that slice joins the search, N ticks up by one, and the quarantine is gone. Most blown holdouts do not die from a single dramatic cheat. They die from being peeked at, adjusted against, and peeked at again.

The third is to decide what I am testing before I test it. A pre-registered hypothesis bounds N by construction, because the strategies that count are the ones I named in advance, not the ones the search wandered into. This is the hardest to hold to, because the whole appeal of a backtest is that it lets you wander.

The take

A single strategy with a short track record is a wide error bar, and the earlier post was about the width of that bar. This is the other failure, and it does not shrink with more data. Trying more strategies makes it worse, not better, because more trials raise the maximum you select even as each individual estimate stays exactly as noisy as it was. Length and breadth are two different lies a Sharpe can tell, and the deflated Sharpe ratio is the only common statistic that docks you for both at once.

The version of me that runs a hundred quiet backtests and reports the best one is not doing research. He is running a lottery and showing you the winning ticket. The tell is that he cannot tell you how many tickets he bought.