Decision Trees vs Linear Models

Two candidates fit the same insurance dataset. One draws a straight hyperplane through the predictor space; the other carves it into rectangles. Knowing when each wins is a recurring SRM exam beat.

A linear regression model writes

$$\hat{Y} = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p$$

A decision tree writes

$$\hat{Y} = \sum_{m=1}^{M} c_m \cdot \mathbf{1}\{X \in R_m\}$$

The linear model fits one global surface. The tree fits a piecewise-constant surface across M disjoint rectangles (leaves), each with constant prediction $c_m$ equal to the leaf-mean for regression trees or the majority class for classification trees.

KEY: Linear models predict with a formula; trees predict with a lookup. Same data, two completely different representations of $f$.

If the true relationship between X and Y is approximately linear and additive, ordinary least squares (OLS) estimates $\beta$ with low variance and the model generalizes well even on modest samples. A single tree must approximate that smooth surface with a staircase of rectangles, which wastes splits and adds variance.

Common mistakes

• Claiming trees always beat linear models on real data. When the true $f$ is smooth and additive, OLS or GLM has lower test error than any single tree and often beats ensembles too.
• Forgetting that trees cannot extrapolate. Predicting Y for X values past the training maximum pins a tree at the top-leaf mean (e.g., a single value like $45,000), while OLS keeps projecting.
• Treating linear-model coefficients as causal in a tree comparison. Coefficients are conditional associations, not causal effects, in either framework. Don't sell a linear coefficient as something a tree "cannot estimate."

Bottom line

• Linear models fit a global additive form $\hat{Y} = X\hat\beta$; trees fit a piecewise-constant $\hat{Y} = \sum c_m \mathbf{1}\{X \in R_m\}$, partitioning predictors into rectangles. The form drives every strength and weakness.
• Trees handle non-linearities, interactions, and mixed predictor types automatically; linear models need explicit transformations, dummy coding, and interaction terms.
• Trees process categorical predictors and missing values natively (via surrogate splits); linear models require dummies and imputation.
• Linear regression supplies standard errors, p-values, and confidence intervals; single trees supply none.

Exam shortcut

If the vignette mentions extrapolation, smooth dose-response, or required confidence intervals, pick a linear or generalized linear model. If the vignette stresses sharp thresholds, many categorical predictors, missing values, or "non-statistician audience," pick a tree. When the prompt asks for the BEST of both, the answer is almost always a tree-based ensemble (random forest or boosting) with a linear / GLM baseline reported alongside.

The full lesson (about 2,461 words, 16 min read) adds 3 worked examples, all 6 common mistakes, a self-check, free in the app.

Learning objectives

• 4d

Browse all free Exam SRM lessons or jump into free Exam SRM practice questions.