Backtesting
Testing a rule against historical data. The exercise is essential, straightforward to perform, and extremely easy to perform in a way that produces a meaningless result.
MadStockAlerts Research · Updated August 29, 2026
What to take away
- A backtest can only ever falsify a rule, never confirm it.
- The most common errors are systematic and have names.
- A result that requires precise parameters is a fitted result.
- Costs, slippage and capacity are frequently excluded and are frequently decisive.
- The number of tests run determines how impressive the best result has to be.
MAD Academy Training Video · 0:46
The Past Is Easy to Beat
A backtest is a claim about a strategy that was designed after seeing the data, and there are several specific ways it inflates.
This lesson is part of a Stock Alerts + Tools plan.
What a backtest can establish
A backtest applies a rule to historical data and reports what would have happened. That is genuinely useful for one purpose: it can demonstrate that a rule would not have worked, which is a definitive result.
It cannot demonstrate that a rule will work. The past is one realisation of a process, the rule was designed by someone who has seen that realisation, and the future is drawn from a distribution that may have changed.
The asymmetry is the whole point. A failed backtest is strong evidence; a successful one is weak evidence, and how weak depends entirely on how it was produced.
The named errors
| Error | What it does |
|---|---|
| Survivorship bias | Tests only the securities that still exist |
| Look-ahead bias | Uses information that was not available at the time |
| Overfitting | Tunes parameters until the past is described perfectly |
| Ignoring costs | Omits spread, commission, impact and financing |
| Selection of the period | Tests a window that happens to suit the rule |
| Ignoring capacity | Assumes size can be transacted that the market could not absorb |
Each has its own article or section in this pillar because each is independently sufficient to invalidate a result. A backtest with all six is not a weak test; it is a description of the data it was fitted to.
Multiple testing
The most underappreciated problem is not any single error but the number of attempts. If a hundred rules are tested at a conventional significance threshold, several will appear significant by chance alone, and the best of them will look excellent.
expected false positives = number of tests x significance level
- 100 tests at a 5 percent threshold produce about 5 apparently significant results from noise
- reporting only the best one conceals the 99 that were also run
Because the discarded attempts are rarely recorded, the researcher frequently does not know how many were run. This is the strongest argument for deciding the rule before looking, and for treating anything found by searching as a hypothesis rather than a finding.
Scroll the chart sideways to see all of it.
What a defensible test looks like
- 1State the hypothesis firstWith a mechanism, before the data is examined. A rule with no reason to work is a pattern in noise.
- 2Use point-in-time dataIncluding delisted securities and unrevised fundamentals as they were reported.
- 3Include realistic costsSpread, commission, impact at the intended size, and financing where leverage is used.
- 4Hold out dataTest on a period the rule was never examined against.
- 5Record every variant triedSo that the count of attempts is known when interpreting the best one.
- 6Check sensitivityA result that collapses when a parameter moves slightly was fitted to that parameter.
The last step is the cheapest and the most informative. A genuine effect degrades gracefully as parameters change; a fitted one has a sharp peak at exactly the values that were chosen.
What to report alongside a result
A backtest result is uninterpretable without the conditions that produced it. The list below is what any published result should state and what most do not.
| Item | Why |
|---|---|
| The universe, including delisted securities | Establishes whether survivorship bias is present |
| The data source and whether it is point-in-time | Establishes whether look-ahead bias is present |
| Every parameter and how it was chosen | Establishes how much fitting occurred |
| The number of variants tested | Determines how impressive the best result has to be |
| The cost assumptions | Determines whether the result survives implementation |
| Turnover and capacity | Determines at what size the result stops holding |
| The worst drawdown and its duration | Determines whether it could have been held |
The last row is the one that decides whether a result is usable rather than whether it is real. A strategy with an excellent long-run record and a four-year underwater period is a strategy almost nobody would have held to the end of it.