Backtests, Data Mining, and Factor Decay

Every strategy you will ever be shown has an excellent backtest. That is not evidence of anything — it's a selection effect, and there's research measuring exactly how much of it survives contact with reality.

Last verified 2026-07-25

Why every backtest you see looks good

Nobody publishes, markets, or pitches a strategy with a bad backtest. So the backtests that reach you are not a sample of strategies — they're a sample of the ones that happened to look best on past data.

With enough variables and enough time periods, patterns that mean nothing will appear purely by chance. Test a hundred rules against history and several will look excellent, having discovered noise rather than a mechanism.

This is the base rate to hold in mind whenever a track record is presented. The impressive backtest isn't evidence in favour — it's the entry requirement for you being shown it at all.

The research put numbers on it

McLean and Pontiff tested 97 variables that academic studies had shown to predict cross-sectional stock returns, and tracked what happened to each after the study period ended.

Portfolio returns were 26% lower out-of-sample — that is, in data after the original sample but before publication. And 58% lower after publication. The roughly 32-point gap between those figures is attributed to investors learning about the mispricing from the research and trading it away.

Both halves matter. The first says a meaningful chunk of any backtest is fitting to its own sample. The second says even genuine effects shrink once enough people know about them.

What this does and doesn't say about factors

It does not say factors are fake. Fama and French's work identified return patterns that have held up across markets and long periods, and they remain a serious part of how academics model returns.

It says the honest expectation for any published, well-known factor is a smaller premium than the original paper reported — because part of that number was sample-fitting and part has since been arbitraged.

So both of the loud positions are wrong. 'Factors are free money, look at the backtest' ignores decay. 'Factors don't work at all' ignores decades of evidence. The measured answer is in between, and smaller than the brochure.