Calculate measures of model predictive accuracy (e.g., Lift, Gini index, AUROC).

You built a generalized linear model (GLM) that scores 50,000 auto policies for renewal lapse. The boss asks: "Is it any good?" Accuracy alone hides everything that matters. Lift, Gini, and AUROC answer the real question.

Why these metrics exist. A predictive model returns a score ${\hat{p}}_i$ for each observation. To use it for a decision (renew / non-renew, fraud / not fraud), you pick a threshold $c$ and classify positive when ${\hat{p}}_i\ge c$. Accuracy at one threshold tells you little; the metrics below summarize discrimination across all thresholds or rank-order quality.

For threshold $c$, classify each observation and count:

Key rates:

• Sensitivity (true positive rate, recall) = $\text{TP}/(\text{TP}+\text{FN})$.
• Specificity (true negative rate) = $\text{TN}/(\text{TN}+\text{FP})$.
• False positive rate = $1-\text{Specificity}=\text{FP}/(\text{FP}+\text{TN})$.
• Precision = $\text{TP}/(\text{TP}+\text{FP})$.
• Accuracy = $(\text{TP}+\text{TN})/N$.

TRAP: In an imbalanced book (say 2% fraud), a classifier that predicts "no fraud" always has 98% accuracy and zero sensitivity. Always look past accuracy for imbalanced targets.

Vary the threshold $c$ from $+\infty$ down to $-\infty$. At each $c$, plot the point (FPR, TPR). The curve sweeps from $(0,0)$ at $c=+\infty$ to $(1,1)$ at $c=-\infty$.

Common mistakes

• Reporting Gini $=$ AUROC. They are not equal. With AUROC 0.84, Gini is 0.68, not 0.84. The factor of 2 minus 1 is the whole point.
• Using accuracy on an imbalanced target. A 2% fraud rate gives 98% accuracy to the always-negative model. Always report AUROC, lift, or Gini alongside.
• Forgetting tie credit in AUROC. Ties count for 0.5, not 0 or 1. Skipping the tie term overstates a tree model's AUROC when many records share a leaf score.

Bottom line

• AUROC is the probability that a random positive scores higher than a random negative. Range $[0.5,1]$; 0.5 is random, 1.0 is perfect, 0.7 to 0.8 is the typical insurance band.
• Gini coefficient $=2\times \text{AUROC}-1$ for a binary classifier. Range $[0,1]$; also defined geometrically as twice the area between the Lorenz curve and the equality line.
• Lift at the top decile $=\frac{\text{response rate in decile 1}}{\text{overall response rate}}$. Lift $>1$ means the model concentrates positives in the top scores.
• Cumulative gains at depth $d$ is the fraction of all positives captured in the top $d$ of scored records. The gains curve plots cumulative gains vs. $d$.

Exam shortcut

If the question gives AUROC and asks for Gini (or vice versa), just apply $2\cdot \text{AUROC}-1$; do not rebuild the ROC. If the question gives a small confusion-matrix-style scoring table with no ties, count concordant positive-negative pairs and divide by $n_1{n}_0$; that is AUROC without drawing a curve.

The full lesson (about 2,087 words, 14 min read) adds 2 worked examples, all 6 common mistakes, a self-check, free in the app.

Learning objectives

• C10

Browse all free MAS-II lessons or jump into free MAS-II practice questions.