Two GLMs tie on Akaike information criterion (AIC). The incumbent costs nothing to keep; the challenger requires a system rebuild. Which actually writes better business? Predictive performance measures, especially the double lift chart, answer by simulating the rate change on real policies.
Why hold-out testing. Any model can memorize noise in its training data. The honest test is performance on data it never saw. Standard practice splits 70/30, fits on the training portion, and evaluates on the test portion. Cross-validation rotates the split.
Procedure on the hold-out set:
- Score every policy with the candidate model.
- Sort ascending by predicted pure premium.
- Bucket into equal-exposure groups (commonly 10 deciles).
- For each bucket, compute average predicted and average actual loss per exposure.
- Plot both on the same axes against bucket index.
KEY: A good model produces a monotone-increasing actual curve that hugs the predicted curve. Flat actuals across deciles means no signal. Crossings or reversals mean the model mis-orders risk.
Single lift tests one model; double lift compares two. This is the chart most state DOIs expect when an insurer files a rate plan replacing an incumbent.
Common mistakes
- Evaluating on the training set. In-sample RMSE always favors the more complex model. A common wrong answer reports R² ≈ 0.95 on training when test R² is only 0.40.
- Reading Gini as calibration. Gini measures ordering, not levels. Two models with identical Gini can have wildly different predicted means; pair Gini with a loss ratio chart.
- Single lift when double lift is needed. If the filing replaces an incumbent, the regulator wants double lift. Single lift proves the new model has signal; it does not prove the new model beats the old one.
Bottom line
- Always evaluate on hold-out data (test set or cross-validation); in-sample fit overstates accuracy.
- Single lift chart: sort hold-out by predicted, bucket into deciles, plot actual versus predicted; a monotone increase proves the model has signal and orders risk correctly.
- Double lift chart: sort by the ratio of challenger to incumbent prediction , bucket, compare actual loss costs; the curve tracking actuals proves the challenger beats the incumbent.
- Loss ratio chart should be flat across deciles for a rate-adequate model; a rising slope signals mispricing.
Exam shortcut
If the question asks how to demonstrate a new rate plan beats the incumbent, answer double lift chart on a hold-out, sorted by ratio of challenger to incumbent. If the question asks whether a single model has predictive power, answer single lift, Gini, or loss ratio chart. If the question gives AIC/BIC values, lower wins, but flag that hold-out lift is the regulator-facing measure.
The full lesson (about 2,252 words, 15 min read) adds 2 worked examples, all 6 common mistakes, a self-check, free in the app.
Learning objectives
- C11
Browse all free MAS-II lessons or jump into free MAS-II practice questions.