51 Stepwise Regression
Stepwise regression takes a different approach to the same underlying question, which predictors actually belong in the model, by adding or removing predictors one at a time based on a chosen criterion, most commonly the Akaike Information Criterion (AIC), which rewards model fit while penalizing every added predictor:
\[ AIC = 2k - 2\ln(\hat{L}) \]
where \(k\) is the number of parameters and \(\hat{L}\) is the model’s maximized likelihood. A lower AIC is preferred. Three variants are in common use:
- Forward selection: starts with no predictors and adds the one that improves the criterion most at each step, stopping when no addition helps.
- Backward elimination: starts with every candidate predictor and removes the least useful one at each step, stopping when no removal helps.
- Both directions (stepwise): at each step, considers adding a predictor or removing one already in the model, whichever improves the criterion more.
51.1 A Caution Before Using It
Stepwise regression is included here because it remains widely taught and widely used, not because it is the recommended default. Testing many candidate predictors in sequence and keeping only the ones that look significant is a form of the multiple-testing problem: some predictors will appear to matter purely by chance, and the final model’s reported p-values and R-squared no longer mean what they would in a model specified in advance, since the selection process itself was driven by the same data used to evaluate it. Modern practice generally prefers Lasso, covered earlier in this topic, for automatic predictor selection, precisely because its penalty is built into a single optimization rather than a sequence of separate significance tests. Where stepwise selection is used, its output is best treated as a starting point for a model that still needs validating on data it was not built from, not as a final answer.
51.2 Worked Example
The same fifteen fields used earlier in this topic, with one addition: plot elevation, a variable with no real agronomic reason to affect wheat yield, included deliberately to see how stepwise selection handles a predictor that does not actually belong.
| Field | Nitrogen (kg/ha) | Phosphorus (kg/ha) | Potassium (kg/ha) | Rainfall (mm) | Elevation (m) | Yield (quintals/ha) |
|---|---|---|---|---|---|---|
| 1 | 40 | 22 | 14 | 380 | 320 | 30 |
| 2 | 60 | 28 | 16 | 420 | 410 | 33 |
| 3 | 60 | 32 | 20 | 460 | 280 | 35 |
| 4 | 80 | 38 | 22 | 400 | 350 | 35 |
| 5 | 80 | 42 | 26 | 500 | 300 | 38 |
| 6 | 100 | 48 | 28 | 440 | 450 | 38 |
| 7 | 100 | 52 | 32 | 520 | 390 | 41 |
| 8 | 120 | 58 | 34 | 460 | 270 | 40 |
| 9 | 120 | 62 | 38 | 540 | 500 | 43 |
| 10 | 140 | 68 | 40 | 480 | 340 | 43 |
| 11 | 140 | 72 | 44 | 560 | 420 | 45 |
| 12 | 160 | 78 | 46 | 500 | 310 | 45 |
| 13 | 160 | 82 | 50 | 580 | 480 | 48 |
| 14 | 180 | 88 | 52 | 520 | 360 | 48 |
| 15 | 200 | 98 | 58 | 600 | 330 | 51 |
51.3 Stepwise Regression in R
51.4 Reading the Result
Backward elimination should drop elevation, since it carries no real relationship with yield in this dataset, and likely drops one or two of the correlated NPK doses as well, for the same redundancy reason Lasso removed them. Whichever predictors survive, the AIC of the selected model should sit at or below the AIC of the full model, confirming the simpler model is not worse by the criterion that was used to build it. That last phrase is exactly the caution above: the model looks good by the very yardstick used to choose it, which is not the same as it being validated evidence that these particular predictors, and only these, drive yield.
Summary
| Concept | Description |
|---|---|
| Foundations | |
| Stepwise Regression | Adds or removes predictors one at a time based on a chosen fit criterion, most often AIC |
| Akaike Information Criterion (AIC) | Rewards model fit while penalizing the number of parameters; a lower AIC is preferred |
| Forward Selection | Starts with no predictors and adds the most helpful one at each step |
| Backward Elimination | Starts with all candidate predictors and removes the least helpful one at each step |
| Both-Directions Stepwise | Considers both adding and removing a predictor at each step, whichever improves the criterion more |
| Caution | |
| The Multiple-Testing Caution | Testing many predictors in sequence inflates the chance some are kept by chance rather than real effect |
| Worked Example | |
| Worked Example: Adding an Irrelevant Predictor | An elevation variable with no real link to yield was added to test whether selection correctly drops it |
| step() in R | Base R function that automates forward, backward, or both-direction selection from a fitted model |