Three Backtest Failure Modes That Cost Me Real Money

Every backtest I've built since starting OpenClaw looked fine on a Sharpe chart. Most of them still broke when I pushed them into paper trading. After enough of those failures, I stopped trusting results by reflex and started running the same three checks on every strategy curve before I let it anywhere near an order ticket.

None of these failure modes are novel. They show up in academic literature under names like "look-ahead bias" and "survivorship bias." The reason I'm writing about them anyway is that knowing about them does almost nothing to prevent them. They show up as small plumbing mistakes inside otherwise clean code, and they are expensive to find after the fact.

1. The lookahead leak you did not put there

The first failure mode is the one I catch most often, and it is almost never in the place I expect. You compute an indicator like a 20-bar SMA, you use it to generate a signal at bar t, you enter at the close of t, you exit at the close of t+5. Clean, right?

Usually, no. The leak hides in how the dataframe was assembled. Merging a daily options chain onto intraday bars, joining an earnings calendar onto the close, filling forward a missing volatility value: any one of these can silently carry information from the future into a row labeled with an earlier timestamp.

The check: shift every feature column by one bar on purpose and re-run. If your performance degrades catastrophically, the original backtest was reading tomorrow's data. If it barely changes, you are probably OK. The first time I ran this on a strategy I was about to promote, the Sharpe dropped from 2.4 to 0.6. That strategy never made it to paper.

2. Survivorship in your universe

The second leak quietly inflates every backtest I have seen published on forums. If you pull a list of tickers today and backtest against the last three years of their price history, you have already guaranteed you are only trading stocks that did not get delisted, acquired, or halted out of existence. That is not a market. That is a winners-only tournament.

For equities this is usually framed as survivorship bias in index constituents. For options, it is worse. Strikes that never traded, expired worthless, or had a chain restructured mid-quarter disappear from most commercial data vendors' histories. A bot that backtests "sell the 30-delta put" on SPY from 2020 onward looks like it prints money in part because the data it is being tested against has already been cleaned of the chains where that trade was untradeable in real time.

The check: record the full universe as it existed at the time of the trade, not as it exists today. If you cannot reconstruct it, mark the backtest speculative and size accordingly. OpenClaw's reconcile loop now logs a warning when a trade fires on a contract whose historical chain has fewer than 30 days of volume above 100 contracts. That warning alone has pulled two strategies out of my deployment queue that looked great on paper.

3. Costs, slippage, and the friction you assumed away

The third leak is the most boring and the most common. You model a fill at the midpoint of the bid-ask spread. You assume zero slippage. You charge a commission of a dollar per contract and move on. The backtest says the strategy makes 18 basis points per trade.

In live, the strategy makes nothing. Sometimes it loses money.

Vertical spreads on liquid names like SPY and QQQ are usually close to the model. Everywhere else, your fills will be materially worse than the midpoint, especially on exits, and especially on fast-moving days when you most want to be out. An 18-basis-point edge is well inside the noise of realistic slippage on a 50-cent-wide spread that quotes ten cents wide.

The check: run the backtest twice. Once at midpoint, once at the full spread on entry and half the spread on exit. If the curve between those two scenarios diverges meaningfully, your strategy is not a strategy. It is a slippage arbitrage against your own simulator.

The habit I kept

I have made all three of these mistakes on paper. Two of them I caught inside a week. The third, the costs leak, took three months and a small but real paper-trading drawdown to diagnose, because it showed up as "the strategy works, just not as well as the backtest." That is the worst kind of failure mode: the one that looks like normal regression to the mean rather than a bug.

My working rule now is that a backtest is a hypothesis, not a result. The curve only tells me a strategy is worth testing in paper. The paper results only tell me it is worth testing in a small live sleeve. Nothing about any of this is a conclusion.

If you build systems that trade on automation, run the three checks above on every strategy before you promote it. They will not catch everything. They will catch the three mistakes I am most likely to make twice.