50 Ridge Regression
Ridge regression (Arthur E. Hoerl & Robert W. Kennard, 1970) solves the same multicollinearity and overfitting problem as Lasso, but with a different penalty on the coefficients, the sum of their squares rather than their absolute values:
\[ \hat{\beta}_{\text{ridge}} = \arg\min_{\beta} \left\{ \sum_{i=1}^{n}(Y_i - X_i\beta)^2 + \lambda \sum_{j=1}^{p} \beta_j^2 \right\} \]
This is the L2 penalty, and it changes the behavior in one important way: it shrinks every coefficient toward zero, but essentially never forces one to land exactly on zero. Ridge keeps every predictor in the model, just with dampened, more stable coefficients, while Lasso will actively drop redundant predictors. Neither is universally better; the choice depends on whether the analyst believes every predictor genuinely contributes something (ridge) or suspects several are redundant and should be eliminated (lasso).
50.1 A Closed-Form Solution
Unlike Lasso, whose absolute-value penalty has no smooth derivative at zero and needs an iterative algorithm such as coordinate descent, ridge regression has an exact matrix solution:
\[ \hat{\beta}_{\text{ridge}} = (X^{\mathsf{T}}X + \lambda I)^{-1} X^{\mathsf{T}} Y \]
where \(X\) holds the standardized predictors, \(Y\) is the centered outcome, and \(I\) is the identity matrix. Adding \(\lambda I\) to \(X^{\mathsf{T}}X\) before inverting is what keeps the solution stable even when predictors are highly correlated. Without it, \(X^{\mathsf{T}}X\) can be nearly singular when predictors overlap heavily, which is exactly the numerical symptom of the multicollinearity problem ridge regression exists to fix.
50.2 Worked Example
The same fifteen fields from the Lasso section, with nitrogen, phosphorus, and potassium doses moving together and rainfall recorded alongside them:
| Field | Nitrogen (kg/ha) | Phosphorus (kg/ha) | Potassium (kg/ha) | Rainfall (mm) | Yield (quintals/ha) |
|---|---|---|---|---|---|
| 1 | 40 | 22 | 14 | 380 | 30 |
| 2 | 60 | 28 | 16 | 420 | 33 |
| 3 | 60 | 32 | 20 | 460 | 35 |
| 4 | 80 | 38 | 22 | 400 | 35 |
| 5 | 80 | 42 | 26 | 500 | 38 |
| 6 | 100 | 48 | 28 | 440 | 38 |
| 7 | 100 | 52 | 32 | 520 | 41 |
| 8 | 120 | 58 | 34 | 460 | 40 |
| 9 | 120 | 62 | 38 | 540 | 43 |
| 10 | 140 | 68 | 40 | 480 | 43 |
| 11 | 140 | 72 | 44 | 560 | 45 |
| 12 | 160 | 78 | 46 | 500 | 45 |
| 13 | 160 | 82 | 50 | 580 | 48 |
| 14 | 180 | 88 | 52 | 520 | 48 |
| 15 | 200 | 98 | 58 | 600 | 51 |
50.3 Ridge Regression in R
50.4 Reading the Result
The unpenalized column shows what plain least squares does with three correlated predictors: it can assign a large positive coefficient to one of nitrogen, phosphorus, or potassium and a large negative coefficient to another, purely to cancel out shared variation, coefficients that would likely flip sign on a slightly different sample of fields. As \(\lambda\) increases, every coefficient shrinks toward zero and, importantly, toward each other, since the correlated predictors end up sharing credit more evenly rather than one dominating arbitrarily. None of them reaches exactly zero, which is the visible difference from the Lasso section: ridge stabilizes the whole set of correlated predictors rather than eliminating any of them.
Ridge vs Lasso: Which to Use
Reach for ridge when there is reason to believe every predictor plays some real role and the goal is simply to stabilize the coefficients, for instance when nitrogen, phosphorus, and potassium are all agronomically meaningful and dropping one outright would remove real information. Reach for Lasso when the goal includes simplifying the model itself, for example building a shorter list of inputs a field advisory app should ask a farmer for. A hybrid, elastic net, blends both penalties and is worth knowing exists even though it is not built from scratch here.
Summary
| Concept | Description |
|---|---|
| Foundations | |
| Ridge Regression | Penalizes the sum of squared coefficients, shrinking them toward zero without eliminating any predictor |
| The L2 Penalty | The sum of squared coefficient values; smooth everywhere, unlike the Lasso's absolute-value penalty |
| Closed-Form Solution | Ridge coefficients can be computed directly via matrix algebra rather than an iterative search |
| Why Adding Lambda-I Stabilizes the Solution | Adding lambda times the identity matrix keeps the matrix invertible even under heavy multicollinearity |
| Worked Example | |
| Worked Example: NPK Doses and Yield | The same fifteen fields used for Lasso, with nitrogen, phosphorus, and potassium moving together |
| Comparing OLS and Ridge Coefficients | Unpenalized coefficients can swing to large, unstable values under multicollinearity; ridge pulls them together |
| Choosing an Approach | |
| Ridge vs Lasso | Ridge keeps every predictor and stabilizes them; Lasso can drop redundant predictors entirely |