Sample Questions
Question 1
Easy
What is the purpose of pruning a decision tree?
Solution
A is correct.
Pruning removes branches from a fully grown tree that contribute little to predictive accuracy on new data. A large, deep tree fits the training data very well (possibly perfectly) but tends to overfit — capturing noise rather than signal. Pruning simplifies the tree, improving generalization performance.
Pruning removes branches from a fully grown tree that contribute little to predictive accuracy on new data. A large, deep tree fits the training data very well (possibly perfectly) but tends to overfit — capturing noise rather than signal. Pruning simplifies the tree, improving generalization performance.
Question 2
Medium
Random forests improve upon bagging by:
Solution
D is correct.
The key innovation of random forests over bagging is the random selection of a subset of predictors at each split point. This prevents dominant predictors from being used in every tree, thereby reducing the correlation between trees. Since the variance of an average of correlated quantities depends on the correlation, decorrelation leads to greater variance reduction.
The key innovation of random forests over bagging is the random selection of a subset of predictors at each split point. This prevents dominant predictors from being used in every tree, thereby reducing the correlation between trees. Since the variance of an average of correlated quantities depends on the correlation, decorrelation leads to greater variance reduction.
Question 3
Hard
For a classification tree, which impurity measure is NOT differentiable and therefore less suitable for use as a splitting criterion in tree-growing algorithms, even though it directly measures classification error?
Solution
E is correct.
The misclassification rate is a piecewise linear function that is not differentiable at the points where the majority class changes. More importantly, it is not sufficiently sensitive to changes in class probabilities within a node -- two nodes with different class distributions can have the same misclassification rate. For these reasons, the Gini index and entropy are preferred for growing trees, while the misclassification rate is sometimes used for pruning.
The misclassification rate is a piecewise linear function that is not differentiable at the points where the majority class changes. More importantly, it is not sufficiently sensitive to changes in class probabilities within a node -- two nodes with different class distributions can have the same misclassification rate. For these reasons, the Gini index and entropy are preferred for growing trees, while the misclassification rate is sometimes used for pruning.
