75 Concurrent (Multi-Branch) Networks
Every network built so far in this topic takes one kind of input: a single feature vector, a single sequence, a single image. Real agricultural decisions often draw on several genuinely different kinds of data at once, satellite-derived vegetation indices, ground sensor readings, weather records, each with its own scale, structure, and reliability. A concurrent network (also called a multi-branch or multimodal architecture) (Tadas Baltrušaitis et al., 2019) handles this directly: each input stream is processed by its own dedicated sub-network, called a branch, and the branches’ outputs are combined, or fused, before a final shared layer produces one prediction.
75.1 Why Separate Branches Before Merging
Concatenating every feature from every data source into one long input vector and feeding it to a single ordinary feedforward network, the approach used everywhere earlier in this book, is a workable baseline, but it forces one network to learn appropriate internal representations for fundamentally different kinds of signals all at once. Giving each data source its own branch lets that branch specialize: an imagery branch can learn representations suited to vegetation indices, a sensor branch can learn representations suited to soil and weather readings, and each branch can even have a different internal architecture entirely, a convolutional branch for raw imagery patches, a recurrent branch for a sensor’s time series, a plain feedforward branch for single-reading tabular features. Only after each branch has extracted its own useful summary does a merge layer, typically an ordinary dense layer, combine those summaries and make the final call.
75.2 Worked Example
The same twenty fields used throughout this topic, now split across two data sources feeding two separate branches. The imagery branch receives NDVI and canopy cover percentage (derived from satellite or drone imagery). The sensor branch receives soil moisture and average temperature (recorded by ground-based IoT sensors). Both branches feed a shared merge layer that produces the final Healthy or Stressed classification.
| Field | NDVI | Canopy Cover (%) | Soil Moisture (%) | Avg Temp (°C) | Status |
|---|---|---|---|---|---|
| 1 | 0.42 | 64 | 18 | 36 | Stressed |
| 2 | 0.38 | 60 | 16 | 37 | Stressed |
| 3 | 0.75 | 84 | 38 | 26 | Healthy |
| 4 | 0.68 | 80 | 32 | 28 | Healthy |
| 5 | 0.45 | 66 | 20 | 34 | Stressed |
(the remaining fifteen fields follow the same NDVI, moisture, and temperature values used throughout this topic; canopy cover is generated in the code below as a function of NDVI plus noise)
75.3 Concurrent Networks in R
75.4 Reading the Result
Backpropagation trained all three pieces together, both branches and the merge layer, from a single combined loss signal at the output, exactly the same chain-rule mechanics from the first section of this topic, just applied across a branching structure rather than one straight stack of layers. The branch activation summaries reveal how much work each data source is actually doing: if the imagery branch’s mean activation already separates Healthy from Stressed fields almost as cleanly as the full merged network does, NDVI and canopy cover are carrying most of the signal on their own, and the sensor branch is contributing comparatively little beyond that. That kind of diagnostic, checking what each modality contributes individually before trusting the fused result, matters in practice: a concurrent network fed a broken or missing sensor stream should degrade gracefully rather than fail outright, and knowing in advance how much each branch actually contributes is what tells a system designer whether that risk is worth worrying about.
Summary
| Concept | Description |
|---|---|
| Foundations | |
| Concurrent (Multi-Branch) Network | Processes multiple distinct input streams through separate sub-networks, then combines their outputs |
| Branches and Specialization | Each branch can specialize its internal representation to the specific kind of data it receives |
| The Merge (Fusion) Layer | A shared layer, typically dense, that combines every branch's output into one final prediction |
| Why Not Just Concatenate All Features Upfront | Separate branches let each data source's representation be learned appropriately before being combined |
| Worked Example | |
| Worked Example: Fusing Imagery and Sensor Streams | An imagery branch (NDVI, canopy cover) and a sensor branch (moisture, temperature) feeding one classifier |
| Building a Two-Branch Network from Scratch | Two small branch sub-networks and a merge layer, base R only, following the same structure as file 01 |
| Training End-to-End with Backpropagation | The chain rule propagates error back through the merge layer into both branches in a single training loop |
| Diagnosing Each Branch's Individual Contribution | Comparing each branch's hidden activation against the true label shows how much each data source contributes |