Machine Learning

A model that classifies every training observation perfectly is usually the worst model in the room. That single fact organizes most of what follows.

Supervised learning infers a pattern between inputs and a known output, then applies that pattern to new inputs. It requires labeled data: matched sets of features and targets. In machine learning (ML) vocabulary, the dependent variable Y is the target and the independent variables X are the features. Training means fitting the algorithm on labeled data; the fit is judged on predictions for data the algorithm never saw.

Supervised problems split by the nature of the target. A continuous target is a regression problem (forecasting returns). A categorical or ordinal target is a classification problem (fraudulent versus legitimate, or credit rating buckets).

Unsupervised learning uses inputs with no target supplied. The algorithm discovers structure in the data itself. Its two problem families are dimension reduction (fewer features, variation preserved) and clustering (grouping similar observations).

Deep learning applies neural networks with many hidden layers to complex tasks: image classification, speech recognition, natural language processing.

Common mistakes

• Calling overfitting a bias problem. Overfitting is variance error. Underfitting is bias error, seen as high error in both samples.
• Adding data to fix underfitting. More training samples never lift a high-bias model to the desired accuracy; only reduced bias does.
• Treating LASSO's $\lambda$ as estimated. It is a hyperparameter set by the researcher and tuned on the validation sample.

Bottom line

• Supervised uses labeled data (target plus features); unsupervised has no target; deep learning uses multi-layer neural networks
• Continuous target = regression problem; categorical or ordinal target = classification problem
• Overfitting = high variance error, low in-sample error, high out-of-sample error; underfitting = high bias error in both
• Fixes: complexity reduction/regularization and k-fold cross-validation (k typically 5 or 10), not usable on time series

Exam shortcut

Read the target first. Target given means supervised; no target means unsupervised. Continuous means regression; categories mean classification. Then read the error pattern: good in-sample and bad out-of-sample is always overfitting/variance, and the answer choice naming bias is the trap. Both errors high is bias/underfitting, and "collect more data" is the trap there.

The full lesson (about 2,216 words, 15 min read) adds 2 worked examples, all 6 common mistakes, a self-check, free in the app.

Learning objectives

• machine learning

Browse all free CFA Level II lessons or jump into free CFA Level II practice questions.