49 Lasso Regression
The Regression Diagnostics and Model Evaluation section flagged two risks that plain multiple regression cannot defend itself against: multicollinearity among predictors and overfitting as more predictors are added. Both show up constantly in farm data. Nitrogen, phosphorus, and potassium doses are usually applied together as a balanced NPK blend, so the three doses tend to rise and fall as a set rather than independently, and a model asked to credit yield to each one separately can end up with unstable, hard-to-trust coefficients even when its overall fit looks fine.
Lasso regression (Least Absolute Shrinkage and Selection Operator) addresses this by adding a penalty to the ordinary least-squares objective:
\[ \hat{\beta}_{\text{lasso}} = \arg\min_{\beta} \left\{ \sum_{i=1}^{n}(Y_i - X_i\beta)^2 + \lambda \sum_{j=1}^{p} |\beta_j| \right\} \]
The first term is the familiar sum of squared errors. The second term, the L1 penalty (Robert Tibshirani, 1996), grows with the absolute size of the coefficients and is controlled by \(\lambda\), a tuning parameter chosen by the analyst. A larger \(\lambda\) pushes coefficients harder toward zero, and unlike ridge regression (covered next), Lasso can push a coefficient to exactly zero, which effectively drops that predictor from the model. That is what makes Lasso useful for feature selection as well as shrinkage.
49.1 Why the Penalty Matters
Ordinary least squares only cares about fitting the training data as closely as possible, which is exactly the behavior that goes wrong when predictors overlap heavily or when there are many of them relative to the number of observations. The penalty term trades a small amount of fit on the training data for coefficients that are more stable and that generalize better to a new season’s data. Predictors need to be standardized, mean-centered and scaled to a common variance, before the penalty is applied, since otherwise a predictor measured in large units (rainfall in millimeters) would be penalized differently than one measured in small units (nitrogen in kilograms) purely as an artifact of scale.
49.2 Worked Example
Fifteen fields recorded nitrogen, phosphorus, and potassium fertilizer doses alongside seasonal rainfall and the resulting wheat yield.
| Field | Nitrogen (kg/ha) | Phosphorus (kg/ha) | Potassium (kg/ha) | Rainfall (mm) | Yield (quintals/ha) |
|---|---|---|---|---|---|
| 1 | 40 | 22 | 14 | 380 | 30 |
| 2 | 60 | 28 | 16 | 420 | 33 |
| 3 | 60 | 32 | 20 | 460 | 35 |
| 4 | 80 | 38 | 22 | 400 | 35 |
| 5 | 80 | 42 | 26 | 500 | 38 |
| 6 | 100 | 48 | 28 | 440 | 38 |
| 7 | 100 | 52 | 32 | 520 | 41 |
| 8 | 120 | 58 | 34 | 460 | 40 |
| 9 | 120 | 62 | 38 | 540 | 43 |
| 10 | 140 | 68 | 40 | 480 | 43 |
| 11 | 140 | 72 | 44 | 560 | 45 |
| 12 | 160 | 78 | 46 | 500 | 45 |
| 13 | 160 | 82 | 50 | 580 | 48 |
| 14 | 180 | 88 | 52 | 520 | 48 |
| 15 | 200 | 98 | 58 | 600 | 51 |
Notice how closely nitrogen, phosphorus, and potassium track each other across the fifteen fields: this is the multicollinearity the diagnostics chapter warned about, built into the data by the way farmers actually buy and apply fertilizer.
49.3 Lasso Regression in R
Rather than relying on a package that may not be available in every R environment, the code below fits Lasso from its own definition, using coordinate descent with soft-thresholding (Jerome Friedman et al., 2010), the same algorithm dedicated packages such as glmnet implement internally.
49.4 Reading the Result
At the smaller penalty, most predictors keep a nonzero coefficient. At the larger penalty, one or more of nitrogen, phosphorus, and potassium is typically driven to exactly zero, since the model only needs one of the three correlated fertilizer doses to capture most of their shared signal, and the penalty removes the redundant ones rather than splitting credit unstably across all three. Rainfall tends to survive at both penalty strengths, since it carries information the fertilizer variables do not. This is Lasso doing variable selection: the zeroed-out predictors are not being declared unimportant to yield in some absolute sense, only redundant given the other predictors already in the model.
Choosing Lambda
The examples above fixed \(\lambda\) at two illustrative values so the effect of increasing the penalty could be seen directly. In practice, \(\lambda\) is chosen by cross-validation: fitting the model across a range of \(\lambda\) values and picking the one that minimizes prediction error on data the model was not fitted on, rather than by inspection. Cross-validation itself is covered in Introduction to Supervised Learning.
Summary
| Concept | Description |
|---|---|
| Foundations | |
| Why Regularization Is Needed | Multicollinearity and overfitting are risks plain multiple regression cannot defend against on its own |
| Lasso Regression | Adds a penalty on the size of the coefficients to the least-squares objective, controlled by lambda |
| The L1 Penalty | The sum of absolute coefficient values; unlike a squared penalty, it can force coefficients to exactly zero |
| Standardizing Predictors First | Predictors must be centered and scaled before penalizing, so the penalty is not an artifact of measurement units |
| Worked Example | |
| Worked Example: NPK Doses and Yield | Fifteen fields where nitrogen, phosphorus, and potassium doses move together, creating real multicollinearity |
| Coordinate Descent and Soft-Thresholding | The algorithm Lasso is fit with in practice, implemented here directly rather than via a package |
| Interpretation | |
| Lasso as Variable Selection | As lambda grows, redundant correlated predictors are dropped from the model entirely |
| Choosing Lambda by Cross-Validation | The penalty strength is normally chosen by minimizing error on held-out data, not fixed by inspection |