Assessing Model Accuracy

Picking the right accuracy metric is half the battle. Pick a method that ignores overfitting and your "best" model fails on new data.

Training error measures fit on the data used to estimate $\hat{f}$. Test error measures fit on independent data drawn from the same distribution. Flexible models drive training error toward zero but eventually overshoot, raising test error.

KEY: The training-error curve falls monotonically with flexibility. The test-error curve is U-shaped. The sweet spot sits at the U's minimum.

$$\text{MSE}=\frac{1}{n}\sum _{i=1}^n(y_i-\hat{f}(x_i){)}^2$$

Root MSE returns the metric to response units. Mean absolute error is more robust to outliers but harder to optimize.

$$E[(y_0-\hat{f}(x_0){)}^2]=\text{Var}(\epsilon )+[\text{Bias}(\hat{f}(x_0)){]}^2+\text{Var}(\hat{f}(x_0))$$

Increasing flexibility lowers bias but raises variance. $\text{Var}(\epsilon )$ is the floor you cannot beat.

The misclassification rate is $\frac{1}{n}\sum I(y_i\ne {\hat{y}}_i)$. The Bayes classifier assigns each x to its most probable class and achieves the lowest possible error rate, the Bayes error rate.

Common mistakes

• Reporting training error as model accuracy. Training error understates test error; the gap widens with flexibility. Always validate on held-out data.
• Using accuracy on an imbalanced dataset. With 99% class 0, predicting "always 0" scores 99% but is useless. Use AUC, F1, or class-weighted log-loss.
• Confusing LOOCV with k-fold. LOOCV has low bias, high variance; 10-fold has slightly higher bias, lower variance. They are not interchangeable.

Bottom line

• Training error always falls as flexibility rises; test error is U-shaped. Minimize test error, not training error.
• Regression uses mean squared error (MSE); classification uses misclassification rate, ROC/AUC, or log-loss.
• The Bayes classifier is the theoretical lower bound on classification error.
• Class imbalance breaks raw accuracy; report AUC, F1, or class-weighted log-loss instead.

Exam shortcut

If the question gives training error and validation error and asks "best model," ignore training error entirely and pick the lowest validation/CV error. If accuracy looks suspiciously close to the majority-class frequency, suspect imbalance and switch to AUC. When choosing between AIC and BIC for model selection with large n, BIC picks the smaller model because its $\ln n$ penalty dominates AIC's constant 2.

The full lesson (about 2,193 words, 15 min read) adds 4 worked examples, all 7 common mistakes, a self-check, free in the app.

Learning objectives

• 1b

Browse all free Exam SRM lessons or jump into free Exam SRM practice questions.