Calculate the summary statistics for a set of decision trees (e.g., Gini index, entropy, residual sum of squares).

A regression tree splits 600 policies into terminal nodes; a classification tree splits the same policies into "lapse / renew" leaves. The first scores splits by squared error, the second by node impurity. Same algorithm, different summary statistics.

Why two families of criteria. A regression tree predicts a number, so squared error is the natural loss. A classification tree predicts a class label, so the criterion must measure how mixed each node is. Pure nodes earn impurity zero; 50/50 mixes earn the maximum.

For a region $R_m$ containing $N_m$ training observations, the fitted value is the regional mean ${\hat{y}}_{R_m}=\frac{1}{N_m}{\sum }_{i\in R_m}{y}_i$. The node-level RSS is:

$${\text{RSS}}_m=\sum _{i\in R_m}(y_i-{\hat{y}}_{R_m}{)}^2$$

Tree-level RSS sums across all $M$ terminal nodes:

$$\text{RSS}(T)=\sum _{m=1}^M\sum _{i\in R_m}(y_i-{\hat{y}}_{R_m}{)}^2$$

Choosing a split. For predictor $X_j$ and cutpoint $s$, define half-planes $R_1(j,s)=\{X|X_j\le s\}$ and $R_2(j,s)=\{X|X_j>s\}$. Pick $(j,s)$ to minimize:

$$\sum _{i\in R_1}(y_i-{\hat{y}}_{R_1}{)}^2+\sum _{i\in R_2}(y_i-{\hat{y}}_{R_2}{)}^2$$

KEY: The reduction $\Delta ={\text{RSS}}_{\text{parent}}-({\text{RSS}}_{R_1}+{\text{RSS}}_{R_2})$ is always nonnegative under greedy splitting. A useless split gives $\Delta =0$.

Common mistakes

• Using the parent proportion to compute child impurity. Each child has its own ${\hat{p}}_{mk}$. Reusing parent $p=0.6$ in Example 2 produces a phantom Gini of 0.48 in both children rather than the correct 0.18 and 0.42.
• Forgetting the weight $N_m/N$ when summing child impurities. Unequal child sizes get equal billing under a plain average. A 90/10 split treated as 50/50 distorts the reduction and can flip the ranking of candidate splits.
• Comparing Gini in proportions to entropy in nats directly. Their scales differ (max 0.5 vs 0.693 for two classes). Compare reductions, not raw levels, and never mix criteria within a single tree.

Bottom line

• Regression trees score splits by residual sum of squares (RSS), minimizing ${\sum }_m{\sum }_{i\in R_m}(y_i-{\hat{y}}_{R_m}{)}^2$ where ${\hat{y}}_{R_m}$ is the regional mean (not the global mean).
• Classification trees score splits by node impurity: Gini $G={\sum }_k{\hat{p}}_{mk}(1-{\hat{p}}_{mk})$, entropy $D=-{\sum }_k{\hat{p}}_{mk}\log {\hat{p}}_{mk}$, or classification error $E=1-{\max }_k{\hat{p}}_{mk}$.
• For two classes, Gini peaks at 0.5 when $p=0.5$ and entropy peaks at $\log 2\approx 0.693$ nats; their scales differ, so compare reductions, never raw levels, and never mix criteria.
• Gini and entropy are differentiable in the proportions and sensitive to node purity; classification error is not. Grow the tree with Gini or entropy, prune it with error.

Exam shortcut

If the response is numeric, the criterion is RSS and the regional fitted value is the regional mean; compute parent RSS, child RSS, and subtract. If the response is categorical and the problem asks for "impurity" without naming a measure, default to Gini unless it explicitly says "cross-entropy" or "deviance".

The full lesson (about 1,848 words, 12 min read) adds 2 worked examples, all 6 common mistakes, a self-check, free in the app.

Learning objectives

• C3

Browse all free MAS-II lessons or jump into free MAS-II practice questions.