59 Decision Trees
Every technique covered so far in this topic classifies using a single global rule: one discriminant function, one boundary, one set of class densities applied uniformly across the whole predictor space. A decision tree (Leo Breiman et al., 1984) instead asks a sequence of simple yes/no questions, splitting the data at each step on whichever predictor and threshold best separates the classes, and repeating within each resulting group. The result is a set of nested rules that a person can read directly: “if NDVI is below 0.57, the field is very likely Stressed; if not, check soil moisture next.”
59.1 How a Split Is Chosen
At each node, the tree considers every predictor and every possible threshold, and picks whichever split produces the purest resulting groups, most commonly measured by the Gini impurity:
\[ Gini = 1 - \sum_{k=1}^{K} p_k^2 \]
where \(p_k\) is the proportion of class \(k\) in a node. A node containing only one class has a Gini of 0, the purest possible outcome; a node split evenly between two classes has a Gini of 0.5. The tree keeps splitting, choosing whichever split most reduces the weighted Gini impurity across the resulting child nodes, until a stopping rule is reached, typically a minimum node size or a maximum depth, since a tree grown without limits will keep splitting until every training point sits in its own leaf, fitting noise rather than signal.
59.2 Worked Example
The same twenty fields used throughout this topic.
| Field | NDVI | Soil Moisture (%) | Avg Temp (°C) | Status |
|---|---|---|---|---|
| 1 | 0.42 | 18 | 36 | Stressed |
| 2 | 0.38 | 16 | 37 | Stressed |
| 3 | 0.75 | 38 | 26 | Healthy |
| 4 | 0.68 | 32 | 28 | Healthy |
| 5 | 0.45 | 20 | 34 | Stressed |
| 6 | 0.80 | 40 | 25 | Healthy |
| 7 | 0.55 | 25 | 31 | Stressed |
| 8 | 0.72 | 35 | 27 | Healthy |
| 9 | 0.40 | 17 | 38 | Stressed |
| 10 | 0.78 | 42 | 24 | Healthy |
| 11 | 0.60 | 28 | 30 | Healthy |
| 12 | 0.48 | 22 | 33 | Stressed |
| 13 | 0.82 | 44 | 23 | Healthy |
| 14 | 0.35 | 15 | 38 | Stressed |
| 15 | 0.65 | 30 | 29 | Healthy |
| 16 | 0.50 | 24 | 32 | Stressed |
| 17 | 0.85 | 45 | 22 | Healthy |
| 18 | 0.44 | 19 | 35 | Stressed |
| 19 | 0.58 | 27 | 30 | Healthy |
| 20 | 0.52 | 23 | 33 | Stressed |
59.3 Decision Trees in R
rpart, like MASS, is one of R’s recommended packages bundled with every standard installation.
59.4 Reading the Result
The printed split rules can be read top to bottom as a flowchart: the first line splits on whichever predictor, almost certainly NDVI given how cleanly it separates the two classes, gives the largest reduction in impurity, and each subsequent line refines one of the resulting groups further. Variable importance ranks every predictor by how much it contributed to reducing impurity across the whole tree, not just at the very first split, which is useful when several splits deeper in the tree lean on a predictor that never appears at the top.
The Overfitting Risk
A single decision tree grown without a stopping rule will eventually carve out a leaf for every training point, achieving perfect training accuracy while learning almost nothing that generalizes to a new field. The minsplit and cp (complexity parameter) settings used above are exactly the kind of stopping rule that keeps this in check, and pruning a tree back after growing it deep, guided by cross-validated error rather than training error, is standard practice. This overfitting tendency is also the direct motivation for random forests, covered next.
Summary
| Concept | Description |
|---|---|
| Foundations | |
| Decision Tree | Classifies via a sequence of yes/no questions on the predictors, producing rules a person can read directly |
| How a Split Is Chosen | Every predictor and threshold is considered; the split reducing impurity the most is chosen at each node |
| Gini Impurity | Measures how mixed a node's classes are; zero for a pure node, higher as classes become more evenly mixed |
| Stopping Rules | Minimum node size or maximum depth prevent a tree from splitting until every point has its own leaf |
| Worked Example | |
| Worked Example: Crop Health Classification | The same twenty NDVI, soil moisture, and temperature fields classified as Healthy or Stressed |
| Fitting with rpart() | rpart is a recommended package bundled with every standard R installation, no extra setup required |
| Variable Importance | Ranks predictors by their total contribution to reducing impurity across the whole tree, not just the first split |
| Overfitting Risk in a Single Tree | An unrestricted tree can fit the training data perfectly while generalizing poorly to new observations |