Overfitting
A model complex enough to describe the past perfectly has described the noise as well as the signal, and the noise does not repeat.
MadStockAlerts Research · Updated August 29, 2026
What to take away
- Any dataset can be fitted arbitrarily well by adding parameters.
- The fit describes the sample, and the sample includes randomness.
- More parameters require exponentially more data to justify.
- Sensitivity analysis is the cheapest test for it.
- The problem is worse in finance because the signal is small and the noise is large.
MAD Academy Training Video · 0:46
Fitting the Noise Perfectly
More parameters always fit history better and usually predict worse, and the gap between the two is the whole problem.
This lesson is part of a Stock Alerts + Tools plan.
Why it happens so easily
Financial returns contain a very small amount of predictable variation inside a very large amount of randomness. A model with enough flexibility will fit both, and it has no way to tell them apart within the sample.
The consequence is a model that performs superbly on the data it was built with and poorly on anything else. That gap is the definition of overfitting, and it appears whenever the fitting has more freedom than the data can support.
The counterintuitive part is that a better in-sample fit is evidence against a model rather than for it, past a certain point. A rule that describes history perfectly has almost certainly described its accidents.
Scroll the chart sideways to see all of it.
- In sample
- Out of sample
The forms it takes
- Parameter tuning: adjusting lookbacks and thresholds until the result improves.
- Rule stacking: adding conditions that each remove a specific historical loss.
- Period selection: choosing start and end dates that suit the rule.
- Universe selection: restricting to the securities on which the rule worked.
- Specification search: trying many formulations and reporting the best.
The second item is the most seductive because each addition is individually justifiable. A filter added to avoid a specific historical drawdown is a rule fitted to that drawdown, and the next one will be different.
Detecting it
| Test | What it reveals |
|---|---|
| Sensitivity to parameters | A sharp peak means the values were fitted |
| Out-of-sample performance | A large degradation means the fit was to the sample |
| Complexity against data length | More parameters than the data can support |
| Performance across sub-periods | A result driven by one period is not a result |
| Performance in other markets | An effect present in only one market is weaker evidence |
The first is the fastest. Vary each parameter by a modest amount and observe the result: a rule using a 47-day lookback that fails at 45 and 50 was not describing anything about markets.
The defences
- Require a mechanism. A rule with an economic reason to work is a hypothesis; one without is a pattern.
- Prefer fewer parameters. Simplicity is not elegance here, it is a constraint on how much can be fitted.
- Hold out data and test on it once. Testing on it repeatedly turns it into training data.
- Deflate any reported statistic by the number of attempts made.
- Expect out-of-sample performance to be materially worse than in-sample, and treat the gap as the estimate of the fitting.
The last point sets a realistic expectation. Even a well-constructed test typically degrades out of sample, and a strategy whose live results match its backtest exactly is more likely to be a coincidence than a validation.
Machine learning does not solve it
More flexible models are more prone to this rather than less, because flexibility is precisely the capacity to fit noise. The techniques developed to manage it are mitigations rather than solutions.
- Regularisation penalises complexity, which reduces fitting and requires choosing the penalty, which is itself a fitted parameter.
- Cross-validation estimates out-of-sample performance, and standard versions assume independent observations, which time series violate.
- Ensembles average many models, which reduces variance and does not remove a bias shared across them.
- More data helps, and financial history is finite and non-stationary, so more data means older data from a different regime.
The second item is a specific and common error. Randomly splitting a time series into training and validation sets leaks future information into the training set, because adjacent observations are correlated. Time-series cross-validation exists for exactly this reason.
The fundamental constraint is not the model. It is that financial data has a very low signal-to-noise ratio and a limited, non-repeating history, and no technique manufactures information that is not in the data.