Bagging, Boosting, and Random Forests

A single decision tree is a high-variance learner: change a few training rows and the splits move. Ensembles of trees fix this by averaging or by sequentially correcting errors. Knowing which knob does what is the exam payoff.

Why ensembles exist. A deep tree has low bias but high variance: small data perturbations swing the splits. Averaging B independent trees cuts variance by a factor of B. Real bootstrap trees are correlated, so the variance reduction is smaller, but still material.

Draw B bootstrap samples from the training data (sample n rows WITH replacement). Fit a deep, unpruned tree to each. For regression, average the B predictions. For classification, take the majority vote across the B trees.

$${\hat{f}}_{\text{bag}}(x)=\frac{1}{B}\sum _{b=1}^B{\hat{f}}^{\ast b}(x)$$

KEY: Bagging reduces variance but does NOT reduce bias. Each tree is already low-bias; the average inherits that bias.

Out-of-bag error. Each bootstrap sample omits roughly $(1-1/n{)}^n\to e^{-1}\approx 0.368$ of the rows, so about 37% of observations are unused for any given tree.

Common mistakes

• Saying bagging reduces bias. Bagging averages high-variance, low-bias trees. The average inherits the bias of a single deep tree; only variance falls.
• Setting $m=p$ in a random forest. This collapses the random forest into plain bagging and loses the decorrelation benefit. Use $m=\sqrt{p}$ or $m=p/3$ as the default.
• Drawing the predictor subset once per tree, or with replacement. The $m$-predictor subset is sampled WITHOUT replacement and FRESH at every split, not once per tree. Reusing one subset for a whole tree (or sampling predictors with replacement) weakens the decorrelation the random forest depends on.

Bottom line

• Bagging fits B deep trees on bootstrap samples and averages (or majority-votes) them, reducing variance but not bias.
• Random forests add a per-split predictor subsample ($m\approx \sqrt{p}$ for classification, $m\approx p/3$ for regression) that decorrelates the trees.
• Boosting grows shallow trees ($d$ = 1 to 6) sequentially on residuals with shrinkage rate $\lambda$, and can overfit if B is too large.
• A smaller $\lambda$ slows learning and requires a larger B to compensate.

Exam shortcut

When the prompt mentions bootstrap samples and averaging across trees grown to full depth, it is bagging or random forest: the only question left is whether $m$ restricts the per-split predictors. When the prompt mentions residuals, shrinkage $\lambda$, and shallow trees grown one at a time, it is boosting.

The full lesson (about 3,369 words, 22 min read) adds 4 worked examples, all 9 common mistakes, a self-check, free in the app.

Learning objectives

• 4c

Browse all free Exam SRM lessons or jump into free Exam SRM practice questions.