Choosing the Number of Clusters

Free SOA Exam SRM (Statistics for Risk Modeling) lesson in Unsupervised Learning Techniques. 10 min read, ~1,439 words.

Elbow method: plot within-cluster sum of squares W(k) against k; pick the kink where marginal gain flattens. Silhouette: average a per-point score in [-1, 1] measuring fit to own cluster vs. nearest neighbor; choose k with highest mean silhouette. Gap statistic: compare log W(k) to a uniform-reference benchmark; pick smallest...