A backtest showing +40% annual return means nothing on its own. The question that actually matters is: could this result have happened by pure chance if the strategy had zero real edge? Hypothesis testing is the formal framework for answering that question, and skipping it is why so many retail strategies that looked great in backtests fall apart live.
Setting Up the Test
Start with a null hypothesis: 'this strategy's average trade return is zero.' Your job is to gather enough evidence to reject that null with confidence. You never prove a strategy works — you only fail to reject that it's noise, with a stated confidence level.
t = (mean return − 0) / (standard deviation of returns / √n)A t-statistic above roughly 2 corresponds to a p-value below 0.05 for reasonably large samples — the conventional (if arbitrary) threshold for 'statistically significant.' A strategy with an average return of 0.3% per trade, a standard deviation of 2%, and 400 trades gives t = 0.003/(0.02/√400) = 3.0, comfortably significant.
Why Sample Size Dominates Everything
| Trades (n) | Minimum edge needed for significance (approx.) |
|---|---|
| 30 | Very large — noise dominates |
| 100 | Moderate edge required |
| 500 | Small, realistic edges detectable |
| 2000+ | Even marginal edges (0.1R) become detectable |
A strategy that wins 14 of 20 trades has a win rate that could easily arise from a fair coin flip due to sampling variance. Treat any backtest under roughly 100 independent trades as a pilot, not a conclusion.
The Multiple Testing Trap
If you test 100 random parameter combinations, roughly 5 will show 'significant' results at the p<0.05 level purely by chance — that's what a 5% false-positive rate means. Quants who scan thousands of parameter sets and cherry-pick the best-looking one are almost guaranteed to find noise dressed up as edge.
- ▸Apply a Bonferroni-style correction: divide your significance threshold by the number of variants tested.
- ▸Reserve a holdout period never touched during research, and validate only once on it.
- ▸Prefer strategies with a clear economic rationale over ones found by pure pattern search.
Out-of-Sample Validation
Split your data into in-sample (for building/tuning) and out-of-sample (touched exactly once, at the end). If performance collapses out-of-sample, the in-sample result was overfit, not a real edge — regardless of how impressive the p-value looked initially.
The Bottom Line
Statistical rigor is not academic overkill for retail traders — it's the difference between deploying real capital behind a genuine edge and confidently blowing an account on noise that happened to look like a pattern.