Compute K-nearest neighbors (KNN).

A policy comes in with a driver age of 28 and a credit score of 705. Five similar policies in your training set had losses of $0, $0, $1,200, $0, and $3,400. KNN turns that neighborhood into a prediction with one line of arithmetic.

Why KNN exists. Parametric models impose a functional form (linear, logistic, GLM). KNN imposes none. The training data IS the model, and prediction is a local average over the closest observed cases. That makes KNN powerful when the true decision surface is highly non-linear, and dangerous when features are noisy or unscaled.

Given training set $\{(x_i,y_i){\}}_{i=1}^n$, a query point $x_0$, and an integer $K$:

• Compute the distance $d(x_0,x_i)$ for every training point $i$.
• Sort and identify the $K$ smallest. Call this neighborhood ${\mathcal{N}}_K(x_0)$.
• Classification: assign the plurality class. Regression: assign the mean response.
• Break ties (equal distances or equal class counts) by a stated rule.

$$\hat{P}(Y=j\mid X=x_0)=\frac{1}{K}\sum _{i\in {\mathcal{N}}_K(x_0)}1\{y_i=j\}$$

Common mistakes

• Skipping standardization. Leaving mileage (range 0 to 50,000) on its raw scale next to age (range 16 to 80) makes the distance essentially a function of mileage alone. A query with mileage 12,000 will have all five neighbors at similar mileages regardless of age.
• Tuning K on the training set. $K=1$ gives zero training error but typically the worst test error. Reporting training error as model performance is a textbook trap; the canonical wrong answer is $K=1$ "because it had perfect training accuracy."
• Using an even $K$ in binary classification. $K=4$ can produce a 2-2 vote. Either set $K$ odd or define a tiebreaker (e.g., smaller total distance wins, or the prior majority class).

Bottom line

• KNN is non-parametric and memory-based. Training cost is zero; prediction cost is $O(n)$ per query because every test point scans the entire training set, where $n$ is the training set size.
• Classifier: predict the majority (plurality) class among the $K$ nearest training points. Regressor: predict the mean (or weighted mean) of the $K$ nearest training responses.
• Distance metric matters. Default is Euclidean $d(x,y)=\sqrt{{\sum }_j(x_j-y_j{)}^2}$. Always standardize features first, applying the training mean and standard deviation, or the largest-scale variable dominates.
• K controls bias-variance. $K=1$ is zero bias, maximum variance (jagged boundary); large $K$ is high bias, low variance (smooth boundary). Tune $K$ by cross-validation, never training error.

Exam shortcut

If the problem gives features on wildly different scales and does not mention standardization, assume the examiner wants you to standardize first and recompute; the unstandardized answer is the trap. If the problem asks for the prediction at a single query point with $K$ named explicitly, list distances, sort, take the top $K$, and apply majority vote (classification) or mean (regression) in that order; do not skip the sort.

The full lesson (about 2,049 words, 14 min read) adds 2 worked examples, all 6 common mistakes, a self-check, free in the app.

Learning objectives

• C1

Browse all free MAS-II lessons or jump into free MAS-II practice questions.