61  Ensemble Methods

Random forest, covered in the previous section, is one specific instance of a broader idea: combining several models into one that predicts better than any of its parts alone. That broader idea is called ensembling, and it splits into two genuinely different strategies. Bagging (bootstrap aggregating), what random forest does, builds many models independently and in parallel, each on its own resampled version of the data, then averages or votes across them, mainly to reduce variance. Boosting builds models sequentially instead: each new model is trained to correct the mistakes of the ensemble built so far, focusing its attention on the observations the previous models got wrong, mainly to reduce bias.

61.1 AdaBoost, in Outline

Adaptive Boosting (AdaBoost) (Yoav Freund & Robert E. Schapire, 1997) is the original and clearest example of the boosting strategy:

  1. Start with every observation weighted equally.
  2. Fit a weak learner, often a decision stump, a tree with a single split, using the current weights.
  3. Compute that learner’s weighted error rate, and give it a vote strength (alpha) based on how much better than chance it did: a more accurate stump earns a louder vote in the final ensemble.
  4. Increase the weight on every observation the stump got wrong, so the next round’s stump is forced to pay more attention to exactly the cases the ensemble is still struggling with.
  5. Repeat for a fixed number of rounds, then combine every stump’s vote, weighted by its own alpha, into one final prediction.

Each individual stump is a weak learner, barely better than a coin flip on its own, but the weighted sequence of them together can classify quite accurately, since each new stump is deliberately built to cover the previous ensemble’s blind spots rather than repeating what already works.

61.2 Worked Example

The same twenty fields used throughout this topic.

Field NDVI Soil Moisture (%) Avg Temp (°C) Status
1 0.42 18 36 Stressed
2 0.38 16 37 Stressed
3 0.75 38 26 Healthy
4 0.68 32 28 Healthy
5 0.45 20 34 Stressed
6 0.80 40 25 Healthy
7 0.55 25 31 Stressed
8 0.72 35 27 Healthy
9 0.40 17 38 Stressed
10 0.78 42 24 Healthy
11 0.60 28 30 Healthy
12 0.48 22 33 Stressed
13 0.82 44 23 Healthy
14 0.35 15 38 Stressed
15 0.65 30 29 Healthy
16 0.50 24 32 Stressed
17 0.85 45 22 Healthy
18 0.44 19 35 Stressed
19 0.58 27 30 Healthy
20 0.52 23 33 Stressed

61.3 AdaBoost in R

61.4 Reading the Result

The alpha values typically shrink across rounds as the easy splits get claimed early and later stumps are left fighting over the genuinely hard, boundary-region fields, a pattern worth noticing in the printed sequence above. Ten single-split stumps voting together, each individually weak, should still reach accuracy comparable to or better than the single full decision tree from the previous section, which is the entire point of boosting: a sequence of simple, cheap models focused on each other’s mistakes can match a much more complex single model, often with less risk of overfitting to noise in any one region of the data.

Beyond AdaBoost: Gradient Boosting

Modern industrial boosting, gradient boosting machines and implementations such as XGBoost and LightGBM, generalizes the same core idea, sequential correction of prior errors, to a much wider range of loss functions and adds substantial engineering for speed and regularization. The reweighting scheme in AdaBoost above is best understood as the historical and conceptual starting point for that whole family, not as a simplified stand-in for it; production forecasting and classification systems built on tabular agricultural data today are more likely to reach for a gradient boosting library than for AdaBoost itself, which is flagged here as a direction worth knowing about rather than built from scratch in this topic.

Summary

Concept Description
Foundations
Ensemble Methods Combines several models into one that predicts better than any single model alone
Bagging vs Boosting Bagging builds models independently to reduce variance; boosting builds them sequentially to reduce bias
AdaBoost Outline Each round reweights observations toward the ensemble's current mistakes and adds a new weak learner
Decision Stumps as Weak Learners A single-split tree, barely better than chance alone, used as the building block boosting combines many of
Worked Example
Worked Example: Crop Health Classification The same twenty NDVI, soil moisture, and temperature fields classified as Healthy or Stressed
Building AdaBoost from rpart Stumps Ten decision stumps fit sequentially with case weights, following the classic AdaBoost.M1 algorithm
Vote Strength (Alpha) Across Rounds A stump's vote strength reflects how much better than chance it performed on the currently weighted data
Gradient Boosting as the Modern Extension XGBoost and LightGBM generalize the same sequential-correction idea with far more engineering behind it