Construct, prune, and validate regression and classification trees.

Your assistant hands you an overgrown tree that scores perfectly on training data and poorly on the holdout. The fix is not a new model. It is knowing how splits are chosen, how the complexity parameter prunes, and how a confusion matrix tells you it was overfit.

How a split is chosen. A tree searches every predictor and every candidate cutpoint, then keeps the split that most improves node purity. For classification, purity is measured two ways.

Entropy measures disorder. A node with all one class has entropy zero. A perfect 50/50 split has entropy one (in bits).

$$H = -\sum_{k} p_k \log_2 p_k$$

Gini impurity measures the chance of misclassifying a random observation labeled by the node's class distribution.

$$G = 1 - \sum_{k} p_k^{2}$$

Both are zero at a pure node and peak at an even split. rpart uses Gini by default. The split with the largest weighted drop in impurity (the information gain) wins.

Common mistakes

• Swapping minsplit and minbucket. minsplit is the count needed to attempt a split; minbucket is the minimum left in each leaf. This exact confusion is the graders' most-cited error.
• Getting the cp direction backwards. Decreasing cp grows a larger tree. Candidates who said decreasing cp shrinks the tree lost credit on October 2025 Task 5(a).
• Choosing cp by training error. Prune on xerror (cross-validated), not rel error. Relative training error always falls as the tree grows, so it never signals overfitting.

Bottom line

• Classification splits maximize purity gain, measured by entropy or Gini impurity; both are zero at a pure node.
• Regression trees split to minimize within-node residual sum of squares and predict the node mean.
• rpart control parameters: cp, minsplit, minbucket, maxdepth all limit tree growth to fight overfitting.
• cp is the minimum improvement in relative error a split must earn; raising cp prunes, lowering cp grows.

Exam shortcut

Gini is faster to compute by hand than entropy: $1 - \sum p_k^2$, no logs. For information gain, always subtract the observation-weighted child impurity from the parent, never the simple average. When asked to fight overfitting, reach for one lever and state its direction: raise cp, raise minbucket, or lower maxdepth.

The full lesson (about 2,247 words, 15 min read) adds 2 worked examples, all 6 common mistakes, a self-check, free in the app.

Learning objectives

• 5a

Browse all free Exam PA lessons or jump into free Exam PA practice questions.