72 Recurrent Neural Networks
A feedforward network has no sense of order: feed it this week’s readings, then next week’s, and it treats the two calls as completely unrelated, with no memory carried between them. That is a real limitation for sequential data, a field’s week-by-week soil moisture history, a season’s daily rainfall, a multi-year price series, where what happened earlier genuinely shapes what is likely to happen next. A Recurrent Neural Network (RNN) fixes this by adding a loop: at each time step, the network combines the current input with a hidden state carried over from the previous time step, and that hidden state acts as the network’s evolving memory of everything it has seen in the sequence so far.
72.1 How Recurrence Works
At each time step \(t\), an RNN computes a new hidden state from the current input and the previous hidden state:
\[ h_t = \tanh(W_{xh} x_t + W_{hh} h_{t-1} + b_h) \]
where \(x_t\) is the input at step \(t\), \(h_{t-1}\) is the hidden state carried over from the step before, and \(W_{xh}\), \(W_{hh}\), \(b_h\) are the same weights and bias reused at every time step, this weight-sharing across time is what lets the network handle sequences of any length with a fixed number of parameters. The hidden state at the final time step, \(h_T\), has in principle been shaped by every input the sequence contained, which is what makes it useful as a compressed summary for a prediction.
Training an RNN uses the same backpropagation principle from the previous section, extended across time steps and called backpropagation through time (BPTT): the network is conceptually unrolled into one copy per time step, and the error at the final output is propagated backward through every one of those unrolled steps, accumulating a gradient for the shared weights at each one along the way.
A well-known practical issue with simple RNNs is that gradients propagated across many time steps can shrink toward zero (the vanishing gradient problem), making it hard for a plain RNN to learn dependencies that span a long sequence. Long Short-Term Memory (LSTM) networks (Sepp Hochreiter & Jürgen Schmidhuber, 1997) were developed specifically to address this, using a more elaborate internal gating mechanism to control what the hidden state remembers and forgets, and are the default choice in practice for anything beyond short sequences.
72.2 Worked Example
Fifteen fields, each with seven consecutive weeks of soil moisture readings that trend toward a field-specific baseline level with some autocorrelation and noise, a realistic simplification of how moisture readings actually behave week to week. The task: given a field’s first seven weekly readings as an input sequence, predict its eighth-week reading.
72.3 Recurrent Networks in R
No package in R’s base or recommended set implements RNNs directly (they generally require a deep learning framework such as keras or torch, unavailable here). The scalar RNN below, with a single hidden unit, is built directly from the recurrence and BPTT equations above using nothing but base R, enough to demonstrate how the mechanism actually works.
72.4 Reading the Result
The RNN’s RMSE should beat the naive persistence baseline, evidence that the recurrent hidden state is picking up genuine autocorrelation structure across the full seven-week input rather than the network learning nothing beyond what the most recent reading alone would already suggest. That comparison against a naive baseline matters generally for any sequence model: a forecasting method is only earning its complexity if it clears a bar this simple to compute, exactly the same principle the AR, MA, ARMA and ARIMA Forecasting topic applied against a held-out test set. A single scalar hidden unit is a genuine simplification, a production RNN would use a hidden state with many units and almost always the LSTM gating mechanism described above, but the recurrence and BPTT mechanics scale up unchanged as the hidden state grows.
Summary
| Concept | Description |
|---|---|
| Foundations | |
| Recurrent Neural Network (RNN) | Adds a loop to a feedforward network, combining the current input with memory carried from prior time steps |
| The Hidden State as Memory | The hidden state at each step is shaped by every input the sequence has contained up to that point |
| Weight Sharing Across Time Steps | The same weights are reused at every time step, letting one fixed-size network handle any sequence length |
| Backpropagation Through Time (BPTT) | Extends backpropagation across an unrolled sequence, propagating output error back through every time step |
| The Vanishing Gradient Problem and LSTM | Gradients can shrink across many time steps in a plain RNN; LSTM's gating mechanism addresses this directly |
| Worked Example | |
| Worked Example: Forecasting an Eighth Week from Seven | Fifteen fields' seven-week soil moisture sequences used to predict each field's eighth-week reading |
| Building a Scalar RNN from Scratch | A one-hidden-unit RNN trained with manually implemented BPTT, using only base R |
| Benchmarking Against a Naive Persistence Baseline | RMSE compared against simply predicting that the next reading equals the most recent one |