Exception-count backtesting is necessary but not sufficient. A model can pass Kupiec with seven years of clean data, then crater on the eighth when the regime shifts. Supervisors and modelers want validation that goes deeper: sensitivity testing, distribution-based goodness-of-fit, governance review. The probability integral transform turns the entire forecasting distribution into a uniform random variable; if the distribution is wrong anywhere, the PIT histogram bends.
Backtesting based on VaR breaches is binary: the realized loss either exceeded VaR or didn't. That throws away most of the information in the forecast distribution. A model that produces a 2-sigma forecast when the true outcome is right at the mean is wrong but never registers a breach. A model that nails the median and badly misses the tail might pass Kupiec for years before the tail event arrives.
Validation frameworks built around Lynch and post-2010 supervisory guidance push beyond exception counts in three directions:
- Conceptual soundness: does the model's structure fit the portfolio?
- Sensitivity analysis: does the output behave reasonably as inputs vary?
Common mistakes
- Treating Kupiec as model validation. Kupiec is one test. Models can pass Kupiec for years while being structurally wrong elsewhere in the distribution. Trap: a question contrasts a model that "passes backtesting" with concerns from validators about the distribution shape: the right answer is that PIT-based validation is required.
- Ignoring the difference between Kolmogorov-Smirnov and Anderson-Darling. KS finds the maximum gap; AD weights tail deviations more heavily. For a VaR application, AD is the tighter and more relevant test. Trap: a question reports both statistics; the test that should drive the decision is AD, not KS.
- Forgetting that PIT requires the full forecast distribution. Banks that only output VaR (a single quantile) cannot run PIT validation. They must produce the full forecast density. Trap: a question describes a parametric VaR system that "computes 99% VaR daily" without producing the underlying density: PIT cannot run.
Bottom line
- Exception-count backtesting (Kupiec) is necessary but not sufficient; full validation rests on three pillars: conceptual soundness, sensitivity analysis, and distribution-based backtesting.
- Conceptual soundness review asks whether the model's design assumptions match the portfolio's economics, independent of historical performance.
- Sensitivity analysis perturbs vol, correlation, distribution, window, and method. Stable models are credible; flip-flopping output signals overfitting.
- Confidence intervals on VaR require accounting for parameter estimation error, model uncertainty, and the tail's rare-event nature; bootstrap is standard. Point 99% VaR with 250 observations carries ~18% standard error.
Exam shortcut
When a question describes Kupiec passing but distribution-based tests rejecting, the correct interpretation is that the model has approximately correct exception counts but wrong shape. The remediation is distribution change (Student-t, historical) or vol input adjustment, not exception-count workarounds. PIT histograms with U-shape mean thin tails; peaked PITs mean fat tails. Read the deviation, then remediate.
The full lesson (about 2,955 words, 20 min read) adds 2 worked examples, all 6 common mistakes, a self-check, free in the app.
Learning objectives
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
Browse all free FRM Part II lessons or jump into free FRM Part II practice questions.