63 Gradient Boosting
AdaBoost, covered in the previous topic, improves each round by increasing the weight on observations the ensemble is currently getting wrong. Gradient boosting (Jerome H. Friedman, 2001) generalizes the same sequential-correction idea a different way: instead of reweighting observations, each new weak learner is trained to predict the residuals, the errors the ensemble has made so far, and its prediction is added to the running total. For a continuous outcome under squared-error loss, this reduces to something intuitive: fit a simple model, see what it still gets wrong, fit another simple model to those specific errors, add it in, and repeat.
\[ F_m(x) = F_{m-1}(x) + \eta \cdot h_m(x) \]
where \(F_{m-1}\) is the ensemble built so far, \(h_m\) is a new weak learner fit to the current residuals \(y_i - F_{m-1}(x_i)\), and \(\eta\) is the learning rate, a shrinkage factor, typically well below 1, that keeps any single round from correcting too aggressively and overshooting.
63.1 Why Fit to Residuals
Fitting each new learner to the previous ensemble’s residuals means every round is aimed precisely at whatever pattern remains unexplained, rather than at the outcome as a whole. Early rounds pick up the dominant, easy-to-find structure in the data; later rounds work on progressively smaller and subtler leftover patterns. A small learning rate means many rounds are needed to reach a given level of fit, but it also means the ensemble builds up its prediction gradually and is less prone to fitting noise in any single round, a direct trade-off between the number of rounds and how aggressively each one is allowed to act.
63.2 Worked Example
Polynomial Regression fit a quadratic curve to a nitrogen response trial where yield rises, peaks, and then declines at the highest doses, exactly the kind of curved, non-monotonic pattern gradient boosting can also learn, without having to decide on a polynomial degree in advance.
| Nitrogen Dose (kg/ha) | Yield (quintals/ha) |
|---|---|
| 0 | 15 |
| 20 | 21 |
| 40 | 27 |
| 60 | 32 |
| 80 | 35 |
| 100 | 38 |
| 120 | 40 |
| 140 | 41 |
| 160 | 40 |
| 180 | 39 |
| 200 | 37 |
63.3 Gradient Boosting in R
Built here from its own definition using rpart regression stumps as the weak learner, the same defensive choice used throughout the previous topic, since dedicated packages such as gbm or xgboost are not part of R’s base or recommended set.
63.4 Reading the Result
Gradient boosting should track the eleven observed yields at least as closely as the quadratic curve, and often more closely, since it is not constrained to any particular functional shape the way a polynomial of a fixed degree is. That flexibility is also its main risk: with enough rounds and too small a shrinkage factor, a gradient boosting model can fit training data almost perfectly while learning very little that generalizes, which is precisely why the cross-validation covered earlier in this topic matters more for a flexible technique like this one than it does for a simple linear fit. In practice, the number of boosting rounds is itself a tuning parameter chosen by cross-validated error, stopping once additional rounds start improving the training fit without improving the held-out performance.
Summary
| Concept | Description |
|---|---|
| Foundations | |
| Gradient Boosting | Adds a sequence of weak learners, each correcting the ensemble's current errors, weighted by a shrinkage factor |
| Fitting to Residuals | Each new learner targets exactly the pattern the ensemble built so far has not yet explained |
| The Learning Rate | A small learning rate spreads the correction across many rounds, reducing the risk of overfitting to noise |
| Gradient Boosting vs AdaBoost | AdaBoost reweights misclassified observations; gradient boosting fits new learners directly to residuals |
| Worked Example | |
| Worked Example: Revisiting the Nitrogen Response Curve | The same eleven-dose nitrogen trial used for polynomial regression, now fit with boosted regression stumps |
| Building a Gradient Booster from rpart Stumps | Twenty rounds of rpart regression stumps fit sequentially to residuals, base R and rpart only |
| Comparing RMSE Against Polynomial Regression | RMSE compares how closely each method's fitted curve matches the eleven observed yields |
| Choosing the Number of Rounds by Cross-Validation | Too many rounds risk fitting noise; the right number is chosen by validating on held-out data, not training fit |