67 Hierarchical Clustering
Both k-means and fuzzy c-means share a requirement that can feel arbitrary: the number of clusters, \(k\), has to be chosen before the algorithm runs. Hierarchical clustering (Joe H. Ward, 1963) sidesteps that decision, at least at first. Rather than partitioning the data into a fixed number of groups directly, it builds a complete nested tree of groupings, starting with every observation as its own cluster and successively merging the two closest clusters together, one pair at a time, until everything has merged into a single cluster containing the whole dataset. The number of clusters to actually use is then chosen afterward, by deciding where to cut the resulting tree.
67.1 Agglomerative Clustering and Linkage
This bottom-up merging process is called agglomerative clustering, the most common form of hierarchical clustering. At each step, the algorithm needs a rule for measuring distance between two clusters, not just two individual observations, called the linkage method:
- Complete linkage: the distance between two clusters is the distance between their two farthest members, tending to produce compact, evenly sized clusters.
- Single linkage: the distance is the distance between their two closest members, which can chain together long, straggly clusters.
- Average linkage: the distance is the average distance across every pair of members between the two clusters.
- Ward’s method: merges whichever pair of clusters increases the total within-cluster variance the least at each step, generally producing well-separated, similarly sized clusters and a common default choice.
The resulting tree is displayed as a dendrogram, a diagram where the height of each merge reflects how dissimilar the two merging clusters were. Cutting the dendrogram horizontally at a chosen height produces a specific number of clusters: a low cut yields many small, tight clusters, and a high cut yields few large, loose ones.
67.2 Worked Example
The same eighteen farms used throughout this topic, described by size, irrigation share, and fertilizer spending per acre.
| Farm | Size (acres) | Irrigation (%) | Fertilizer Spend (₹’000/acre) |
|---|---|---|---|
| 1 | 2.5 | 10 | 3.5 |
| 2 | 3.0 | 15 | 4.0 |
| 3 | 3.5 | 8 | 5.0 |
| 4 | 4.0 | 20 | 4.5 |
| 5 | 2.8 | 12 | 3.8 |
| 6 | 4.5 | 22 | 6.0 |
| 7 | 6.5 | 40 | 9.5 |
| 8 | 7.0 | 45 | 10.0 |
| 9 | 8.0 | 38 | 11.0 |
| 10 | 7.5 | 50 | 9.8 |
| 11 | 9.0 | 42 | 12.5 |
| 12 | 8.5 | 55 | 11.5 |
| 13 | 12.0 | 70 | 16.0 |
| 14 | 14.0 | 75 | 18.0 |
| 15 | 11.5 | 68 | 15.5 |
| 16 | 16.0 | 85 | 20.0 |
| 17 | 13.0 | 72 | 17.0 |
| 18 | 15.5 | 90 | 21.0 |
67.3 Hierarchical Clustering in R
67.4 Reading the Result
The dendrogram itself is often more informative than any single cut of it: the height at which two branches join shows exactly how similar or different those groups are, something a flat method like k-means never reports directly, since it returns only a final partition with no record of how the clusters relate to each other. The cross-tabulation against the earlier k-means result should show strong, close-to-diagonal agreement, since both methods are finding the same underlying three-group structure in the same data, and this kind of cross-check, do multiple clustering methods agree, is one of the few genuinely useful validation tools available when no ground truth label exists to check against. Where the two methods disagree, look at exactly which farms are involved: they are very likely the same borderline farms the fuzzy c-means membership matrix flagged in the previous section, a case of three different techniques converging on the same honest uncertainty rather than each one hiding it in a different place.
Summary
| Concept | Description |
|---|---|
| Foundations | |
| Hierarchical Clustering | Builds a complete nested tree of clusters rather than requiring the number of clusters chosen in advance |
| Agglomerative (Bottom-Up) Merging | Starts with every observation as its own cluster and merges the closest pair repeatedly until one cluster remains |
| Linkage Methods | Complete, single, and average linkage define cluster-to-cluster distance differently, shaping the resulting tree |
| Ward's Method | Merges whichever pair increases within-cluster variance the least at each step; a common default linkage choice |
| Worked Example | |
| The Dendrogram | A tree diagram where merge height reflects dissimilarity; cutting it at a chosen height sets the cluster count |
| Worked Example: The Same Eighteen Farms | The same eighteen farms clustered by size, irrigation share, and fertilizer spend as earlier in this topic |
| Fitting with hclust() and Cutting with cutree() | hclust() with Ward's method builds the tree; cutree() cuts it into a chosen number of flat clusters |
| Cross-Checking Against K-Means | Strong agreement between hierarchical and k-means clusterings is a practical, if informal, validation check |