74  Reinforcement Learning

Every technique covered so far in this book, including the feedforward, recurrent, and convolutional networks earlier in this topic, learns from a fixed dataset of examples with known correct answers. Reinforcement learning (Richard S. Sutton & Andrew G. Barto, 2018) is a genuinely different training paradigm: there is no labeled dataset at all. Instead, an agent takes actions inside an environment, receives a reward (or penalty) after each action, and gradually learns a policy, a rule for choosing actions, that maximizes the total reward it accumulates over time. Deciding each week whether to irrigate a field is exactly this kind of problem: the right decision depends on the field’s current condition, the consequences of the decision only become clear afterward, and the goal spans an entire season, not any single week in isolation.

74.1 The Reinforcement Learning Framework

  • State: the situation the agent currently finds itself in, such as a field’s current soil moisture level.
  • Action: a choice the agent can make, such as irrigating today or not.
  • Reward: the immediate numeric feedback received after taking an action in a given state, positive for a good outcome, negative for a costly or harmful one.
  • Policy: the strategy the agent is learning, a mapping from states to the action it should take in each one.
  • The exploration-exploitation trade-off: while learning, an agent has to balance exploiting the best action it currently believes it knows against exploring other actions that might turn out to be even better, since acting only on current, possibly incomplete knowledge risks settling for a mediocre policy.

Q-learning is one of the most widely used reinforcement learning algorithms, and it works by learning a Q-table, one entry for every state-action pair, estimating the total future reward the agent can expect if it takes that action in that state and then behaves optimally afterward. After each action, the Q-table is updated by the rule:

\[ Q(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \right] \]

where \(\alpha\) is the learning rate, \(\gamma\) (the discount factor) controls how much future reward matters relative to immediate reward, \(r\) is the reward just received, and \(s'\) is the state the agent landed in after acting. Once training is complete, the learned policy simply picks, for every state, whichever action has the highest Q-value.

74.2 Worked Example

A simplified irrigation-scheduling problem with five soil moisture states, Very Dry, Dry, Optimal, Wet, and Very Wet, and two possible actions each week, Irrigate or Skip. Irrigating tends to push the moisture state up by one level; skipping tends to let it drift down by one level as the crop uses water and moisture evaporates, both with some randomness. The reward is highest at Optimal, penalized at either extreme, and irrigating carries a small water cost on top of that.

74.3 Q-Learning in R

74.4 Reading the Result

The learned policy should recommend Irrigate for the Very Dry and Dry states, Skip for the Wet and Very Wet states, and, at the Optimal state, whichever action the small irrigation cost tips the balance toward, since staying at Optimal by skipping avoids that cost entirely while irrigating from Optimal risks overshooting into Wet. Nobody hand-coded that policy anywhere in this script: it emerged entirely from the agent trying actions, observing rewards, and updating its Q-table across two thousand simulated episodes, exactly the property that makes reinforcement learning suited to sequential decision problems where the right rule of thumb is not obvious in advance and the consequences of a decision only unfold over time. A real irrigation-scheduling system would need a far richer state description (multiple sensors, weather forecasts, crop growth stage) and a more sophisticated algorithm than a plain lookup table, but the underlying loop, act, observe a reward, update an estimate of long-run value, repeat, is unchanged.

Summary

Concept Description
Foundations
Reinforcement Learning Learns a policy from rewards received while acting in an environment, not from a fixed labeled dataset
State, Action, Reward, Policy The core vocabulary: a situation, a choice, feedback on that choice, and the learned rule mapping one to the other
The Exploration-Exploitation Trade-off Balances acting on the best currently known action against trying other actions that might turn out better
Q-Learning and the Q-Table Estimates the expected long-run value of every state-action pair, updated after every action taken
The Q-Learning Update Rule Blends the reward just received with the best estimated future value, moving the Q-table toward accuracy
Worked Example
Worked Example: Irrigation Scheduling Five soil moisture states and two actions, with a reward highest at the ideal moisture level
Training a Q-Table in R A Q-table trained over 2000 simulated episodes using an epsilon-greedy exploration strategy, base R only
Reading the Learned Policy The policy emerges from training rather than being hand-coded, recommending irrigation only when the field needs it