Understand the assumptions underlying different tree ensemble methods and the improvements they can make to decision trees.

A single deep CART model nails the training set and falls apart on holdout. Tree ensembles exist because every fix for that one weakness (variance, bias, correlation) implies a different assumption about how trees fail.

Why a single tree fails. CART (Classification And Regression Trees) is the standard recursive-partition base learner that ensembles are built to improve on. A deep CART tree partitions the predictor space into many leaves. Small changes to the training data can flip an early split, cascading into a completely different tree. That instability is high variance. A shallow tree is stable but misses interactions: high bias. Ensembles let you keep one of these traits and fix the other.

Draw $B$ bootstrap samples (size $n$, with replacement) from the training data. Fit a deep, unpruned tree to each. Aggregate: average for regression, majority vote for classification.

$${\hat{f}}_{\text{bag}}(x)=\frac{1}{B}\sum _{b=1}^B{\hat{f}}^{\ast b}(x)$$

Variance math. If the $B$ tree predictions had pairwise correlation $\rho$ and individual variance ${\sigma }^2$, the average has variance:

Common mistakes

• Confusing $m$ defaults. $\sqrt{p}$ is classification, $p/3$ is regression. Swapping them on a $p=36$ regression problem gives $m=6$ (wrong) instead of 12.
• Assuming boosting cannot overfit. Bagging and RF are overfit-proof in $B$; boosting is not. Too many stages at a high learning rate inflates test error. Always validate $M$.
• Treating bagging and random forests as the same method. Bagging uses all $p$ predictors at every split; RF restricts to $m<p$. The whole point of RF is to break the $\rho {\sigma }^2$ variance floor that bagging cannot.

Bottom line

• Single tree: low bias, high variance, unstable to small data perturbations. Ensembles attack one of those two failure modes.
• Bagging: averages $B$ bootstrap-trained trees; ensemble variance is $\rho {\sigma }^2+(1-\rho ){\sigma }^2/B$, so the $\rho {\sigma }^2$ floor persists even as $B$ grows. Assumes trees roughly unbiased.
• Random forest: bagging plus a random $m$-predictor subset per split ($m\approx \sqrt{p}$ classification, $m\approx p/3$ regression) to lower $\rho$ and break bagging's variance floor.
• Deep and unpruned base trees are required in bagging and RF so they stay approximately unbiased; averaging handles the variance.

Exam shortcut

If the problem asks "which ensemble reduces variance," answer bagging or random forest; "which reduces bias," answer boosting. If the problem gives $p$ and asks the default split-candidate count, write $\sqrt{p}$ for classification and $p/3$ for regression without hesitation. If asked whether more trees can hurt, the answer is no for bagging and RF, yes for boosting: that one phrase resolves most "which statement is true about ensembles" multiple-choice items.

The full lesson (about 2,306 words, 15 min read) adds 2 worked examples, all 6 common mistakes, a self-check, free in the app.

Learning objectives

• C4

Browse all free MAS-II lessons or jump into free MAS-II practice questions.