A value strategy that looks brilliant over thirty years of history can be an artifact of three data choices: which stocks were in the universe, when their earnings were actually knowable, and how many alternative specifications were quietly discarded before the winner was shown to you.
Backtesting approximates the real-life investment process by applying a strategy's rules to historical data and measuring what would have happened. The objective is to understand the risk and return trade-off of the strategy, not merely its average return. A backtest adds rigor and can serve as an acceptance or rejection criterion before capital is committed.
Two limits define its honest use. A strategy that backtests well may still fail out of sample, and a strategy that backtests poorly could still work, but few managers or clients will fund it. The implicit assumption is that the future will at least somewhat resemble the past. Backtesting fits quantitative and systematic styles naturally, but fundamental managers use it too, to check whether a screening criterion historically added incremental excess return.
Common mistakes
- Calling the rolling window a two-sample split. It produces many successive in-sample and out-of-sample pairs, and each period's out-of-sample data becomes part of the next window's in-sample data.
- Running the covariance estimate on the full history. In a risk parity backtest the second rolling window is mandatory. Using all data to set 1995 weights is look-ahead bias, and it typically inflates the reported Sharpe ratio.
- Treating point-in-time data as a cure-all. It removes survivorship and look-ahead bias but leaves data snooping untouched.
Bottom line
- Objective: understand the risk and return trade-off by approximating live investing; usable as an accept or reject screen
- Three steps: strategy design, historical investment simulation, output analysis
- Design parameters: investment universe, return and currency definition, benchmark, rebalancing frequency with transaction costs, start and end dates
- Rolling window: fit in-sample, hold out-of-sample, slide forward; multifactor strategies run it twice, once for factors and once for the covariance matrix
Exam shortcut
Classify the bias from the stem's noun. "Current index constituents" is survivorship. "Restated financials," "revised GDP," "reported three months later," or "vendor backfill" is look-ahead. "Highest t-statistic after testing dozens of models" is data snooping, and the fix is never point-in-time data.
The full lesson (about 2,840 words, 19 min read) adds 2 worked examples, all 6 common mistakes, a self-check, free in the app.
Learning objectives
- backtesting and simulation
Browse all free CFA Level II lessons or jump into free CFA Level II practice questions.