The Bias-Variance Tradeoff

Two models can have identical training error and wildly different test error. The bias-variance decomposition tells you why, and which knob to turn.

$$E[(y_0-\hat{f}(x_0){)}^2]=\text{Var}(\hat{f}(x_0))+[\text{Bias}(\hat{f}(x_0)){]}^2+\text{Var}(\epsilon )$$

Bias is the systematic error from approximating a complex truth with a simpler model. A linear fit to a curved relationship has high bias. Variance is how much $\hat{f}$ jumps if you refit on a different training sample. A 1-nearest-neighbor model has high variance because every point swings the fit.

KEY: Bias is about model misspecification; variance is about sampling noise in the fit. Both are properties of the method, not the data.

Flexibility drives the tradeoff. Flexible methods (high-degree polynomials, small-k KNN, deep trees, neural nets) bend to match training data, so bias drops. They also chase noise, so variance rises. Rigid methods (linear regression, large-k KNN, shallow trees) underfit (high bias) but produce stable fits (low variance).

HIGH-FREQUENCY: Training MSE is monotone decreasing in flexibility. Test MSE is U-shaped. Picking the model with the lowest training MSE always overfits.

Common mistakes

• Reading training MSE as model quality. Training MSE drops to zero for any interpolating method; it never signals overfitting. Always score on held-out data, ideally cross-validated.
• Assuming "more flexible" always means "more bias reduction". Past the true complexity of $f$, extra flexibility adds variance with no bias payoff.
• Forgetting irreducible error. A test MSE of 1.2 on data with $\text{Var}(\epsilon )=1.0$ is near-optimal, not mediocre.

Bottom line

• Expected test MSE decomposes into the variance of $\hat{f}$, the squared bias of $\hat{f}$, and irreducible error $\text{Var}(\epsilon )$, all non-negative.
• More flexibility lowers bias and raises variance, and you cannot push both below a method-specific floor.
• Test MSE is U-shaped in flexibility, while training MSE falls monotonically and never marks the sweet spot.
• Irreducible error $\text{Var}(\epsilon )$ is a noise floor no model can beat.

Exam shortcut

If asked which method "minimizes both bias and variance," answer that no method does past the irreducible floor; the goal is total MSE. If the prompt gives training MSE far below test MSE, diagnose high variance / overfitting and prescribe less flexibility or more data. If training and test MSE are both high and close, diagnose high bias / underfitting and prescribe more flexibility.

The full lesson (about 1,563 words, 10 min read) adds 2 worked examples, all 7 common mistakes, a self-check, free in the app.

Learning objectives

• 1c

Browse all free Exam SRM lessons or jump into free Exam SRM practice questions.