79 L1 and L2 Regularization
Introduction to Deep Learning flagged overfitting as a particular risk for neural networks, given how many weights even a modest network has to fit. Regularization is the standard defense: adding a penalty term to the loss function that discourages weights from growing large, trading a small amount of training-set fit for a model less likely to have simply memorized noise. Two penalty forms, L1 and L2, appeared already in a different context, Advanced Regression Techniques, as Lasso and Ridge regression. The same two penalties apply directly to a neural network’s weights, where the practice is usually called weight decay (Anders Krogh & John A. Hertz, 1991) rather than Lasso or Ridge, but the underlying mathematics is unchanged.
79.1 How Each Penalty Shapes the Weights
L2 regularization adds a penalty proportional to the sum of squared weights, \(\lambda \sum w_i^2\), to the loss. Its gradient contribution, \(2\lambda w_i\), is proportional to each weight’s current size: a large weight gets pulled back hard, a small weight barely at all. This produces a smooth shrinkage across every weight, useful when most features are expected to contribute at least a little, but it essentially never drives a weight to exactly zero.
L1 regularization adds a penalty proportional to the sum of absolute weights, \(\lambda \sum |w_i|\), to the loss (Robert Tibshirani, 1996). Its gradient contribution is \(\lambda \cdot \text{sign}(w_i)\), a constant push toward zero regardless of how large the weight currently is. That constant push, unlike L2’s size-proportional pull, is what allows L1 to drive some weights to exactly zero, effectively performing automatic feature selection: a weight L1 pushes all the way to zero means the network has decided that input contributes nothing useful.
The choice between them mirrors the Lasso-versus-Ridge choice from Topic 11 exactly: L2 when most inputs are believed to matter somewhat, L1 when some inputs are suspected to be irrelevant and a sparse, more interpretable set of active weights is preferred.
79.2 Worked Example
The same twenty-field crop health dataset used throughout Topic 19, now with two additional columns of pure random noise, unrelated to crop status by construction, added alongside NDVI, soil moisture, and temperature. A well-regularized network should learn to ignore those two irrelevant columns; an unregularized one has no mechanism forcing it to.
79.3 Regularization in R
79.4 Reading the Result
The unregularized weights for noise1 and noise2 should be noticeably nonzero, evidence the network is fitting to patterns in pure random noise simply because nothing in its training objective discourages that. The L2 column should show every weight shrunk toward zero relative to the unregularized fit, including the genuinely predictive NDVI, moisture, and temperature weights, but the noise weights rarely reach exactly zero. The L1 column should tell a sharper story: the noise weights should land at or very close to zero, while ndvi, the strongest predictor throughout this book’s worked examples, should retain a clearly nonzero weight, evidence that L1’s constant per-step push toward zero was enough to eliminate the irrelevant features entirely while leaving the genuinely useful signal intact. All three versions should reach comparable training accuracy on a dataset this small and cleanly separated, which is exactly the point: regularization is not primarily about the current training accuracy, it is about which of these three networks is more likely to still work correctly on a new field’s data next season.
Summary
| Concept | Description |
|---|---|
| Foundations | |
| Regularization | Adds a penalty on weight size to the loss, trading a little training fit for better generalization |
| L2 Regularization (Weight Decay) | Penalizes the sum of squared weights; shrinks every weight proportionally to its size, rarely to exactly zero |
| L1 Regularization | Penalizes the sum of absolute weights; applies a constant push toward zero regardless of weight size |
| Why L1 Produces Sparsity and L2 Does Not | L2's size-proportional gradient slows near zero; L1's constant gradient keeps pushing until a weight reaches zero |
| Choosing Between L1 and L2 | L2 when most inputs are believed relevant; L1 when some inputs are suspected irrelevant and sparsity is useful |
| Worked Example | |
| Worked Example: Two Irrelevant Noise Features | The same twenty-field crop health dataset with two added columns of pure random noise |
| Implementing Both Penalties in a Manual Gradient Update | A single added term, 2*lambda*w for L2 or lambda*sign(w) for L1, layered onto the ordinary gradient update |
| Comparing Learned Weights Across Regularization Types | Unregularized noise weights stay nonzero; L2 shrinks them; L1 drives them to or near exactly zero |