71  Feedforward Networks and Backpropagation

Introduction to Deep Learning described the forward pass: input values flow through a network’s layers, each neuron computing a weighted sum and applying an activation function, until an output emerges. A feedforward network, also called a multilayer perceptron, is the architecture built directly from that description, with no loops, no memory, and no shortcuts: every connection points forward, from input toward output.

What was left unanswered is how a network’s weights and biases, initially set to small random values that produce essentially useless predictions, end up as values that produce accurate ones. That is the job of backpropagation (David E. Rumelhart et al., 1986), the algorithm that trains a feedforward network by working backward from its prediction error to figure out exactly how much each individual weight contributed to that error, then nudging every weight in the direction that reduces it.

71.1 How Backpropagation Works

Training proceeds in repeated cycles, each called an epoch, and each epoch has two passes:

  1. Forward pass: input data flows through the network exactly as described in the previous topic, producing a prediction at the output layer.
  2. Loss calculation: the prediction is compared against the known label, using a loss function, squared error for a continuous target, cross-entropy for a classification target, that quantifies how wrong the prediction was.
  3. Backward pass: this is where backpropagation does its work. Using the chain rule from calculus, the algorithm computes how much the loss would change for a small change in each weight, starting at the output layer and working backward, layer by layer, toward the input. Each layer’s error signal is computed from the layer after it, which is what makes the process efficient: no weight’s gradient has to be computed from scratch independently.
  4. Weight update: every weight is nudged slightly in the direction that reduces the loss, scaled by a learning rate that controls how large each step is. This update rule, and the family of algorithms built around it, is the subject of Optimization Techniques, the topic that follows this unit’s coverage of specific architectures.

Repeating this cycle over many epochs, and over many training examples each epoch, is what turns a randomly initialized network into one that makes useful predictions.

71.2 Worked Example

The same twenty-field NDVI, soil moisture, and average temperature dataset used to classify crop status as Healthy or Stressed in Classification Models and again in Advanced Techniques in Supervised Learning, now classified by a small feedforward network instead.

Field NDVI Soil Moisture (%) Avg Temp (°C) Status
1 0.42 18 36 Stressed
2 0.38 16 37 Stressed
3 0.75 38 26 Healthy
4 0.68 32 28 Healthy
5 0.45 20 34 Stressed
6 0.80 40 25 Healthy
7 0.55 25 31 Stressed
8 0.72 35 27 Healthy
9 0.40 17 38 Stressed
10 0.78 42 24 Healthy
11 0.60 28 30 Healthy
12 0.48 22 33 Stressed
13 0.82 44 23 Healthy
14 0.35 15 38 Stressed
15 0.65 30 29 Healthy
16 0.50 24 32 Stressed
17 0.85 45 22 Healthy
18 0.44 19 35 Stressed
19 0.58 27 30 Healthy
20 0.52 23 33 Stressed

71.3 Feedforward Networks in R

The first block below builds a tiny one-hidden-layer network entirely from its own definition, implementing the forward pass and backpropagation manually with base R matrix operations, so the mechanics above are visible in code rather than hidden inside a function call. The second block fits the same data with nnet(), from R’s recommended nnet package, which performs the same forward-pass-and-backpropagation training internally in a single function call, the way a network would actually be trained in practice.

71.4 Reading the Result

The printed loss values should show a clear downward trend across the 2000 epochs, direct evidence that backpropagation is doing its job: each epoch’s weight update is measurably reducing the squared error between predictions and true labels. Both the manual network and nnet() should reach comparable, high training accuracy on a dataset this cleanly separated, since they are fitting the same kind of model using the same underlying algorithm, one written out explicitly, the other running inside a single function call. The gap between them, if there is one, usually comes down to nnet()’s more sophisticated optimizer (it does not use plain gradient descent internally) and its random initialization, not a difference in what each network is fundamentally doing.

Summary

Concept Description
Foundations
Feedforward Network (Multilayer Perceptron) An architecture with no loops or memory, every connection pointing forward from input toward output
Backpropagation The algorithm that trains a network by propagating prediction error backward to compute each weight's gradient
The Epoch: Forward Pass, Loss, Backward Pass, Update One full training cycle: compute a prediction, measure its error, propagate that error backward, update every weight
The Chain Rule and Layer-by-Layer Error Signals Each layer's error signal is computed from the layer after it, making gradient computation efficient across depth
The Learning Rate Scales how large each weight update step is; too large risks instability, too small trains slowly
Worked Example
Worked Example: Crop Health Classification The same twenty NDVI, soil moisture, and temperature fields classified as Healthy or Stressed as in earlier topics
Building a Network from Scratch with Base R A one-hidden-layer network with manual forward and backward passes, trained by plain gradient descent
Fitting the Same Data with nnet() nnet(), from R's recommended nnet package, trains the same kind of network in a single function call