Decision Trees vs Linear Models

Two candidates fit the same insurance dataset. One draws a straight hyperplane through the predictor space; the other carves it into rectangles. Knowing when each wins is a recurring SRM exam beat.

A linear regression model writes

$$\hat{Y}={\beta }_0+{\beta }_1{X}_1+{\beta }_2{X}_2+\cdots +{\beta }_p{X}_p$$

A decision tree writes

$$\hat{Y}=\sum _{m=1}^M{c}_m\cdot 1\{X\in R_m\}$$

The linear model fits one global surface. The tree fits a piecewise-constant surface across M disjoint rectangles (leaves), each with constant prediction $c_m$ equal to the leaf-mean for regression trees or the majority class for classification trees.

KEY: Linear models predict with a formula; trees predict with a lookup. Same data, two completely different representations of $f$.

If the true relationship between X and Y is approximately linear and additive, ordinary least squares (OLS) estimates $\beta$ with low variance and the model generalizes well even on modest samples. A single tree must approximate that smooth surface with a staircase of rectangles, which wastes splits and adds variance.

Common mistakes

• Claiming trees always beat linear models on real data. When the true $f$ is smooth and additive, OLS or GLM has lower test error than any single tree and often beats ensembles too.
• Forgetting that trees cannot extrapolate. Predicting Y for X values past the training maximum pins a tree at the top-leaf mean (e.g., a single value like $45,000), while OLS keeps projecting.
• Treating linear-model coefficients as causal in a tree comparison. Coefficients are conditional associations, not causal effects, in either framework. Don't sell a linear coefficient as something a tree "cannot estimate."

Bottom line

• Linear models fit a global additive form $\hat{Y}=X\hat{\beta }$; trees fit a piecewise-constant $\hat{Y}=\sum c_{m}1\{X\in R_m\}$, partitioning predictors into rectangles. The form drives every strength and weakness.
• Trees handle non-linearities, interactions, and mixed predictor types automatically; linear models need explicit transformations, dummy coding, and interaction terms.
• Trees process categorical predictors and missing values natively (via surrogate splits); linear models require dummies and imputation.
• Linear regression supplies standard errors, p-values, and confidence intervals; single trees supply none.

Exam shortcut

If the vignette mentions extrapolation, smooth dose-response, or required confidence intervals, pick a linear or generalized linear model. If the vignette stresses sharp thresholds, many categorical predictors, missing values, or "non-statistician audience," pick a tree. When the prompt asks for the BEST of both, the answer is almost always a tree-based ensemble (random forest or boosting) with a linear / GLM baseline reported alongside.

The full lesson (about 2,131 words, 14 min read) adds 2 worked examples, all 6 common mistakes, a self-check, free in the app.

Learning objectives

• 4d

Browse all free Exam SRM lessons or jump into free Exam SRM practice questions.