78  Gradient Descent

Every neural network trained across the previous two topics leaned on the same underlying mechanism without examining it directly: after backpropagation computes how the loss would change for a small change in each weight, something has to actually decide how far, and in what direction, to move that weight. Gradient descent (Sebastian Ruder, 2016) is that something. It is not specific to neural networks at all: the same update rule that trained a feedforward network’s weights is exactly the rule that could have fit the closed-form regression models back in Regression Models, an alternative path to the same answer lm() reaches directly.

78.1 The Gradient Descent Update Rule

For a loss function \(L\) and a parameter \(\theta\), gradient descent repeatedly applies:

\[ \theta \leftarrow \theta - \eta \frac{\partial L}{\partial \theta} \]

where \(\eta\) is the learning rate and \(\partial L / \partial \theta\) is the gradient, the direction of steepest increase in the loss. Moving against that direction is what decreases the loss. Three variants of this rule differ only in how much data each update looks at:

  • Batch gradient descent: computes the gradient using the entire training dataset at every single update. This produces a smooth, stable path toward the minimum, but each update is expensive when the dataset is large, since every one of them requires a full pass through the data.
  • Stochastic gradient descent (SGD): computes the gradient using just one randomly chosen training example per update. Each individual update is nearly free to compute, and the extra noise this introduces can actually help the optimizer escape shallow local minima, but the path to the minimum is visibly noisier, bouncing around rather than descending smoothly.
  • Mini-batch gradient descent: a middle ground, computing the gradient from a small random subset (a mini-batch, commonly 16 to 256 examples in practice) at each update. This is the default choice for training most neural networks, since it balances update speed against the stability of the resulting descent path.

78.2 Worked Example

Thirty farms with fertilizer expenditure (₹’000/acre) and observed yield (quintals/acre), following a genuinely linear relationship with noise, chosen specifically so the closed-form ordinary least squares solution from lm() provides an exact answer to check all three gradient descent variants against.

78.3 Gradient Descent in R

78.4 Reading the Result

All three gradient descent variants should land close to the same intercept and slope lm() reaches directly, confirming that the iterative update rule and the closed-form OLS formula are solving the exact same optimization problem, minimizing squared error, by two different routes. The differences that remain are about the path each variant took to get there rather than the destination: batch gradient descent, updating from the full dataset every time, should produce the most stable, lowest final loss of the three at a matched number of epochs, while stochastic gradient descent’s single-point updates introduce noise that typically leaves it slightly further from the exact optimum after the same number of passes through the data, even though each of its individual updates was far cheaper to compute. This is precisely the trade-off that makes mini-batch descent the practical default for training real neural networks on datasets far too large for a full batch update to be affordable at every step.

Summary

Concept Description
Foundations
Gradient Descent Iteratively moves each parameter against its gradient to reduce a loss function, one small step at a time
The Update Rule and the Learning Rate The learning rate scales each step; every parameter update follows the same subtract-the-gradient rule
Batch Gradient Descent Uses the entire training dataset for every update, producing a smooth but computationally expensive descent
Stochastic Gradient Descent (SGD) Uses one random example per update, cheap and fast but noisy, and sometimes helpful for escaping local minima
Mini-Batch Gradient Descent Uses a small random subset per update, the practical default balancing update cost against descent stability
Worked Example
Worked Example: Fertilizer Spend and Yield Thirty farms with a genuinely linear fertilizer-spend-to-yield relationship, checked against lm()'s exact answer
Comparing Three Variants Against Closed-Form OLS All three variants converge to nearly the same coefficients lm() reaches directly, by different iterative paths
Convergence Path vs Final Destination The variants differ mainly in how they get to the minimum, not in what minimum they ultimately reach