How GLM Approaches Differ from OLS

Ordinary least squares is unbiased but greedy: with many correlated predictors it overfits, and with non-linear structure it just plain misses. Regularization and k-nearest neighbors (KNN) are the two main escape hatches, and they trade different things for that escape.

OLS picks $\hat{\beta }$ to minimize the residual sum of squares.

$$\text{RSS}=\sum _{i=1}^n(y_i-{\beta }_0-\sum _{j=1}^p{\beta }_j{x}_{ij}{)}^2$$

OLS is the best linear unbiased estimator (BLUE) under Gauss-Markov assumptions. It fails in three practical situations:

• $p$ close to or exceeding $n$: the design matrix is rank-deficient and $(X^{T}X{)}^{-1}$ blows up or does not exist.
• Multicollinearity: highly correlated predictors inflate coefficient variances. Estimates flip sign sample-to-sample.
• Pure prediction goals: a small dose of bias often cuts variance enough to lower test mean squared error (MSE). OLS refuses that trade.

KEY: Regularization sacrifices unbiasedness on purpose to reduce variance. The decomposition $\text{MSE}={\text{Bias}}^2+\text{Variance}+{\sigma }^2$ is the whole motivation.

Ridge adds a squared-coefficient penalty to the residual sum of squares (RSS).

Common mistakes

• Skipping standardization before ridge or lasso. The penalty acts on raw coefficient magnitudes. A predictor measured in dollars vs. thousands of dollars gets vastly different treatment. Always standardize to mean 0 and SD 1 first.
• Claiming ridge does variable selection. Ridge shrinks toward zero but coefficients only hit exactly zero at $\lambda =\infty$. For variable selection, use lasso (or elastic net).
• Picking $\lambda$ on training error. Training error decreases monotonically as $\lambda$ drops to 0, recovering OLS. Always pick $\lambda$ by cross-validation on held-out folds.

Bottom line

• OLS minimizes RSS unpenalized; ridge adds an $L_2$ penalty $\lambda \sum {\beta }_j^2$, lasso an $L_1$ penalty $\lambda \sum |{\beta }_j|$, and KNN abandons a functional form.
• Ridge shrinks coefficients toward zero but never to zero, while lasso shrinks and sets some exactly to zero, performing variable selection.
• Always standardize predictors before ridge or lasso, since their penalties are scale-sensitive.
• Choose $\lambda$ by cross-validation: $\lambda =0$ reproduces OLS and $\lambda \to \infty$ shrinks everything to zero; the one-SE rule favors simpler models.

Exam shortcut

If the question stem mentions "variable selection" or "sets coefficients to zero," answer lasso. If it mentions "shrinkage without selection" or "correlated predictors stay together," answer ridge. If it mentions "non-parametric," "no functional form," or "average of nearby points," answer KNN, and remember to flag the curse of dimensionality whenever the question gives a high $p$ with modest $n$.

The full lesson (about 3,570 words, 24 min read) adds 2 worked examples, all 6 common mistakes, a self-check, free in the app.

Learning objectives

• 2g

Browse all free Exam SRM lessons or jump into free Exam SRM practice questions.