Choosing the Number of Clusters

Free SOA Exam SRM (Statistics for Risk Modeling) lesson in Unsupervised Learning Techniques. 10 min read, ~1,547 words.

W(k) is non-increasing in k, so minimizing it alone always returns k = n and cannot select a cluster count by itself. Elbow method: plot within-cluster sum of squares W(k) against k; pick the kink where marginal gain flattens, not the largest raw drop. Silhouette: average a per-point score in...