Classification and Regression Trees

A decision tree splits the predictor space into rectangles, then predicts a single value inside each rectangle. The mechanics are identical for numeric and categorical responses; only the splitting metric and the prediction rule change.

A tree partitions the predictor space into $J$ non-overlapping regions $R_1,\dots ,R_J$. Each region is a terminal node (leaf). Inside a region, the tree predicts a constant: the mean of Y for a regression tree, the majority class (or class probabilities) for a classification tree.

KEY: Every internal node is a yes/no question on one predictor. Every leaf is a prediction. The path from root to leaf is the rule that produced the prediction.

For a classification tree, the leaf stores the majority class and its within-leaf proportion. The tree below predicts loan default from two predictors: it first asks whether income is below 50k, then for the low-income branch asks whether the applicant is under 40. Each leaf carries a class label and the proportion of that class in the leaf, which doubles as the soft-classification probability.

Common mistakes

• Using misclassification error rate to grow a classification tree. It is too flat. Use Gini or cross-entropy for growing; reserve error rate for pruning and reporting.
• Forgetting the within-leaf prediction is a constant. A regression tree predicts the leaf mean, not a linear function of X within the leaf. Inside the leaf, every observation gets the same predicted value.
• Confusing larger $\alpha$ with bigger trees. Larger $\alpha$ means smaller trees (the penalty on $|T|$ is heavier). Smaller $\alpha$ means bigger trees.

Bottom line

• Trees partition predictor space into rectangles, predicting one constant per region: the leaf mean of Y (regression) or the majority class (classification).
• Regression splits minimize residual sum of squares (RSS); classification splits minimize Gini index or cross-entropy. Classification error rate is too coarse for growing.
• Trees grow greedy and top-down via recursive binary splitting, then prune back using cost-complexity, penalizing in-leaf error by $\alpha |T|$.
• Larger $\alpha$ yields smaller trees; choose $\alpha$ by cross-validation on the deviance or error, never by training error.

Exam shortcut

If the question asks for the predicted value in a terminal node, compute the leaf mean for regression or the majority-class proportion for classification. If the question shows two candidate splits and asks which is better, compute the weighted child impurity (RSS, Gini, or entropy) and pick the smaller one.

The full lesson (about 2,946 words, 20 min read) adds 3 worked examples, all 7 common mistakes, a self-check, free in the app.

Learning objectives

• 4b

Browse all free Exam SRM lessons or jump into free Exam SRM practice questions.