Machine Learning Methods and Prediction

Your firm hands you a credit-risk model with 800 features and 50,000 loans. Classical econometrics wants clean parameters. The trader paying your bonus wants the lowest out-of-sample loss. Machine learning bridges the two. Risk managers need to know exactly what trade is being made.

Classical econometrics asks "what is the true coefficient on X-1, and is it significant?" Machine learning asks "what function minimizes prediction error on data I have not seen?" Different questions, different methods.

Econometrics favors linear models, few parameters, and inference about a data-generating process. ML favors highly nonlinear models, large feature spaces, and out-of-sample performance over significance.

KEY: A statistically significant coefficient may have no predictive value out of sample. A model with low in-sample R-squared may have superior predictive accuracy. Pick the method that matches the question. Capital regulation needs inference, fraud detection needs accuracy.

The target determines the regression's role. Linear regression predicts a continuous quantity, such as next-quarter loss or a bond spread, using $\hat y=\beta_0+\sum_j\beta_jx_j$ . Ordinary least squares fits coefficients by minimizing squared prediction errors.

Read the full lesson, free →

Worked examples and practice. Free with a free account, no card.

Common mistakes

Tuning hyperparameters on the test set. Hyperparameters belong on validation. Touching test more than once inflates reported accuracy and you can no longer trust the headline number. Trap: a question describes "the analyst tries 20 ridge alpha values, picks the one with best test accuracy". That's a discipline violation, not the right answer.
Forgetting to rescale features for distance-based methods. KNN, K-means, SVM, and ridge are all sensitive to scale. A feature in dollars dominates a feature in fractions if both are passed raw. Always standardize or min-max scale first. Trap: K-means clusters appear driven entirely by one large-scale variable. The fix is rescaling, not more clusters.
Confusing PCA components with regression coefficients. PCA is unsupervised. The direction of maximum variance in X may have nothing to do with Y. Using the first PC as a "best feature" can throw away information. Trap: a question pairs PCA with a regression of Y on the first PC.

Bottom line

ML targets out-of-sample prediction; classical econometrics targets parameter inference. Different goals, different methods.
Train / validation / test split: train fits parameters, validation tunes hyperparameters, test reports honest out-of-sample performance, and you touch test only once.
Overfitting (large train-to-validation error gap) is fought with regularization, cross-validation, and more data; underfitting (both errors high) is fought with more model flexibility.
Ridge (L2) shrinks coefficients smoothly; LASSO (L1) zeroes irrelevant ones, performing feature selection that ridge cannot.

Exam shortcut

When a question gives a model with high training accuracy and low validation accuracy, the answer is overfit and the fix is regularization, more data, or a simpler model. When it asks LASSO vs. ridge with many irrelevant features, LASSO. When it asks accuracy on a 1%-positive problem, the trap is "high accuracy means good model". Wrong unless precision and recall are also strong.

The full lesson (about 5,524 words, 37 min read) adds 2 worked examples, all 6 common mistakes, a self-check, free in the app.

Learning objectives

Browse all free FRM Part I lessons or jump into free FRM Part I practice questions.