Resampling and Cross-Validation

Training error always looks better than reality. Resampling is how you force the model to confess what it really knows.

Training error underestimates test error because the model has already seen those observations. You need a held-out estimate of how the fitted $\hat{f}$ generalizes.

KEY: Resampling repeatedly draws samples from the training data, refits the model on each, and aggregates the test-error estimates.

Randomly split the data into a training set (commonly 70 to 80 percent) and a validation set (or test set). Fit $\hat{f}$ on training, score on validation.

• Drawbacks: the estimate depends on which observations land in each split (high variance), and the model trains on only part of the data, inflating estimated error (upward bias).

Randomly partition the data into k roughly equal folds. For each fold i, fit on the other k-1 folds and compute the test error on fold i. Average the k errors.

$${\text{CV}}_{(k)}=\frac{1}{k}\sum _{i=1}^k{\text{MSE}}_i$$

For classification, replace ${\text{MSE}}_i$ with the misclassification rate on fold i.

Common mistakes

• Reporting training MSE as model quality. Training error decreases monotonically with flexibility and tells you nothing about generalization. Always use a held-out estimate.
• Fitting feature selection on the full data, then cross-validating only the final model. This leaks the test folds into preprocessing. The entire pipeline (selection plus fitting) must be inside each CV iteration.
• Believing LOOCV is always better than 10-fold. LOOCV has lower bias but higher variance and $\approx$ n times the cost. Empirically, 10-fold often wins.

Bottom line

• Validation set approach: split once into training and test. Simple, but the estimate is high-variance and the model trains on less data.
• k-fold cross-validation: partition into k equal folds, train on k-1, test on 1, rotate, average the k test errors. Standard choices are k = 5 or k = 10.
• Leave-one-out cross-validation (LOOCV): k-fold with k = n. Nearly unbiased, very high variance across folds, expensive (n fits).
• Bias-variance tradeoff: validation set is highest bias, LOOCV is lowest bias but highest variance, 5- or 10-fold is the sweet spot.

Exam shortcut

If the prompt says "split the data once," it is the validation set approach; flag the high-variance, upward-bias drawbacks. If it says "k folds" with k between 5 and 10, expect a bias-variance balance question. If k = n or "leave one out," reach for the leverage formula when the underlying model is least-squares linear; otherwise count n refits.

The full lesson (about 2,183 words, 15 min read) adds 4 worked examples, all 7 common mistakes, a self-check, free in the app.

Learning objectives

• 1d

Browse all free Exam SRM lessons or jump into free Exam SRM practice questions.