44 Multiple Regression
Wheat yield does not depend on nitrogen alone. Rainfall, soil quality, sowing date, and pest pressure all play a part, often at the same time. Multiple regression extends the single-predictor model from the previous section to two or more independent variables, so that several factors can be weighed together rather than one at a time:
\[ Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_k X_k + \varepsilon \]
Each \(\beta_j\) represents the change in \(Y\) for a one-unit increase in \(X_j\), holding every other predictor fixed. That “holding fixed” condition is what separates a multiple-regression coefficient from a simple correlation: it isolates each predictor’s own contribution once the others are accounted for.
44.1 Estimating the Coefficients
With one predictor, the slope and intercept can be worked out by hand from the two column sums, as the previous section showed. With two or more predictors, the calculation is normally done by matrix algebra rather than by hand, which is exactly why software takes over from this point on. The coefficients are still chosen the same way: to minimize the total squared distance between observed and predicted values.
44.2 Adjusted R-squared
Plain \(R^2\) has an awkward property: it goes up every time a new predictor is added, even a predictor that has nothing to do with yield. Adjusted \(R^2\) corrects for this by penalizing the model for each additional predictor:
\[ R^2_{adj} = 1 - (1 - R^2)\frac{n-1}{n-k-1} \]
where \(n\) is the sample size and \(k\) is the number of predictors. When comparing two models with a different number of predictors, Adjusted \(R^2\) is the fairer number to look at, not plain \(R^2\).
44.3 Worked Example
Extending the nitrogen example from the previous section, the agronomist also recorded seasonal rainfall for each sub-plot, since rainfall was not held constant across the trial.
| Sub-plot | Nitrogen (kg/ha) | Rainfall (mm) | Yield (quintals/ha) |
|---|---|---|---|
| 1 | 40 | 420 | 24 |
| 2 | 60 | 460 | 27 |
| 3 | 80 | 380 | 28 |
| 4 | 100 | 510 | 35 |
| 5 | 120 | 440 | 33 |
| 6 | 140 | 530 | 40 |
| 7 | 160 | 470 | 38 |
| 8 | 180 | 550 | 43 |
Fitting this model in R (below) gives approximately:
\[ \hat{Y} = -6.9 + 0.083\,X_1 + 0.078\,X_2 \]
where \(X_1\) is nitrogen dose and \(X_2\) is rainfall. Holding rainfall fixed, each additional kilogram of nitrogen per hectare is associated with about 0.083 more quintals of yield. Holding nitrogen fixed, each additional millimeter of rainfall is associated with about 0.078 more quintals of yield. Notice that the nitrogen coefficient dropped from 0.112 in the single-predictor model to 0.083 here: part of what looked like a nitrogen effect earlier was really rainfall moving alongside it, since the two were not independent across sub-plots.
44.4 Multiple Regression in R
44.5 A Caution Worth Repeating
Adding predictors is tempting because it almost always raises plain \(R^2\), but a model with ten weakly justified predictors is usually worse for prediction than one with three well-chosen ones. Two of the checks that matter most, multicollinearity between predictors and the risk of overfitting, are covered properly in Regression Diagnostics and Model Evaluation, the closing section of this topic. Treat every multiple regression result as provisional until those checks have been run.
Summary
| Concept | Description |
|---|---|
| Foundations | |
| Multiple Regression | Extends single-predictor regression to two or more predictors evaluated together |
| Coefficient Interpretation (Holding Others Fixed) | Each coefficient shows the effect of its own predictor with all other predictors held constant |
| Estimating Coefficients with Software | With two or more predictors, coefficients are typically estimated via matrix algebra rather than by hand |
| Model Fit | |
| Adjusted R-squared | R-squared adjusted for the number of predictors, penalizing additions that do not improve the model |
| Why Adjusted R-squared Is Preferred | Plain R-squared always rises with more predictors, even unhelpful ones; Adjusted R-squared corrects this |
| Worked Example | |
| Worked Example: Nitrogen and Rainfall | Fitting nitrogen and rainfall together against wheat yield across 8 sub-plots |
| Coefficients Can Shift When Predictors Are Added | The nitrogen coefficient changed once rainfall was added, since the two predictors moved together |