Free SOA Exam SRM (Statistics for Risk Modeling) Basics of Statistical Learning Practice Questions

Bias measures the systematic error introduced by the modeling assumptions. It is the difference between the expected prediction $E[\hat{f}(x_0)]$ (averaged over many training sets) and the true function value $f(x_0)$. A high-bias model makes strong assumptions that may not match the true relationship (underfitting).

Why each other option is incorrect:
- (E) This describes variance, not bias.
- (C) This describes the irreducible error $\text{Var}(\varepsilon)$.
- (D) The total prediction error includes bias, variance, and irreducible error — not just bias.
- (B) Correlation between predictions and true values is related to model accuracy but is not the definition of bias.

The Bayes classifier assigns each observation to the most probable class given the observed features: $\hat{y}(x_0) = \arg\max_j P(Y = j | X = x_0)$. This is the theoretically optimal classifier that minimizes the overall misclassification rate.

Why each other option is incorrect:
- (E) The Bayes classifier requires knowledge of the true conditional class probabilities, which are generally unknown. Even with large samples, we can only estimate these probabilities, not compute them exactly.
- (B) The Bayes classifier achieves the lowest possible error rate (Bayes error rate), but this is generally nonzero because of overlapping class distributions.
- (C) The regression function $f(x) = E(Y|X=x)$ is relevant for regression, not classification. The Bayes classifier uses conditional class probabilities.
- (D) KNN with $K = 1$ is an approximation, not equal to the Bayes classifier. As $n \to \infty$ and $K$ grows appropriately, KNN can approach the Bayes classifier, but $K = 1$ specifically has high variance.

The ranking from highest to lowest variance is: I (validation set) > III (LOOCV) > II (10-fold CV).

- **Validation set approach (I)**: Uses only 50% of data for training, and the estimate depends entirely on one random split. This produces the highest variance because a single split can be very unrepresentative.
- **LOOCV (III)**: Uses $n-1$ observations for training in each fold. While it averages over $n$ folds, the training sets are nearly identical (differ by only 1 observation), producing highly correlated fold estimates. Averaging correlated estimates does not reduce variance as effectively, so LOOCV has moderate-to-high variance.
- **10-fold CV (II)**: Uses 90% of data for training and averages over 10 less-correlated fold estimates. The lower correlation between folds means averaging is more effective at reducing variance.

Why each other option is incorrect:
- (A) This places LOOCV as having the lowest variance, which contradicts the fact that its fold estimates are highly correlated.
- (B) This places the validation set approach as having the lowest variance, but it actually has the highest due to complete dependence on one split.
- (C) This places 10-fold CV as having the highest variance, which is incorrect.
- (E) This places 10-fold CV as having higher variance than the validation set approach, which is incorrect.