Bagging, Boosting, and Random Forests

Free SOA Exam SRM (Statistics for Risk Modeling) lesson in Decision Trees. 22 min read, ~3,369 words.

Bagging fits B deep trees on bootstrap samples and averages (or majority-votes) them, reducing variance but not bias. Random forests add a per-split predictor subsample ( for classification, for regression) that decorrelates the trees. Boosting grows shallow trees ( = 1 to 6) sequentially on residuals with shrinkage rate, and...