57 Support Vector Machines
Where LDA finds a boundary by modeling each class’s distribution, a Support Vector Machine, SVM (Corinna Cortes & Vladimir Vapnik, 1995), finds a boundary by focusing only on the observations closest to it, the support vectors, and choosing the boundary that leaves the widest possible margin between them. For a linearly separable dataset, this means finding the hyperplane \(w \cdot x + b = 0\) that maximizes the distance to the nearest point of either class.
Real data rarely separates cleanly, so the practical version is a soft-margin SVM, which allows some points to sit inside the margin or even on the wrong side of it, penalized by the hinge loss:
\[ \min_{w, b} \; \frac{\lambda}{2}\|w\|^2 + \frac{1}{n}\sum_{i=1}^{n} \max\bigl(0, \, 1 - y_i(w \cdot x_i + b)\bigr) \]
where \(y_i \in \{-1, +1\}\). The first term keeps the margin wide; the second penalizes points that violate it, and \(\lambda\) controls the trade-off between the two, playing a role similar to the penalty strength in Lasso and Ridge from the previous topic.
57.1 Fitting a Linear SVM by Gradient Descent
Because the hinge loss has no smooth derivative at the margin boundary, SVMs are normally fit with specialized quadratic-programming solvers bundled in packages such as e1071, which may not be available in every R environment. The soft-margin objective above can also be minimized directly with subgradient descent, in the same spirit as the coordinate descent used for Lasso: at each step, only the currently misclassified or margin-violating points contribute to the update, since correctly classified points beyond the margin have zero hinge loss and therefore zero gradient from that term.
57.2 Worked Example
The same twenty fields from the LDA section, NDVI, soil moisture, and temperature predicting crop health status.
| Field | NDVI | Soil Moisture (%) | Avg Temp (°C) | Status |
|---|---|---|---|---|
| 1 | 0.42 | 18 | 36 | Stressed |
| 2 | 0.38 | 16 | 37 | Stressed |
| 3 | 0.75 | 38 | 26 | Healthy |
| 4 | 0.68 | 32 | 28 | Healthy |
| 5 | 0.45 | 20 | 34 | Stressed |
| 6 | 0.80 | 40 | 25 | Healthy |
| 7 | 0.55 | 25 | 31 | Stressed |
| 8 | 0.72 | 35 | 27 | Healthy |
| 9 | 0.40 | 17 | 38 | Stressed |
| 10 | 0.78 | 42 | 24 | Healthy |
| 11 | 0.60 | 28 | 30 | Healthy |
| 12 | 0.48 | 22 | 33 | Stressed |
| 13 | 0.82 | 44 | 23 | Healthy |
| 14 | 0.35 | 15 | 38 | Stressed |
| 15 | 0.65 | 30 | 29 | Healthy |
| 16 | 0.50 | 24 | 32 | Stressed |
| 17 | 0.85 | 45 | 22 | Healthy |
| 18 | 0.44 | 19 | 35 | Stressed |
| 19 | 0.58 | 27 | 30 | Healthy |
| 20 | 0.52 | 23 | 33 | Stressed |
57.3 SVM in R
57.4 Reading the Result
The sign and relative size of each weight show which predictors push toward Healthy versus Stressed and by how much, on the standardized scale where the weights are directly comparable. NDVI should carry the largest weight, since it separates the two classes most cleanly in the raw data. Unlike LDA, an SVM makes no assumption that the predictors follow a normal distribution within each class, it only cares about the boundary and the points nearest to it, which is why it tends to hold up better than distribution-based methods when that normality assumption is doubtful.
Beyond a Straight Line: The Kernel Trick
The version fit above draws a straight decision boundary, adequate when the classes really are close to linearly separable, as they are here. When they are not, SVMs can swap the linear dot product \(x_i \cdot x_j\) inside the optimization for a nonlinear kernel function, most commonly the radial basis function (RBF) kernel, which implicitly maps the data into a higher-dimensional space where a linear boundary in that space corresponds to a curved boundary in the original one. Fitting a kernel SVM efficiently needs the quadratic-programming machinery a dedicated package provides rather than the simple gradient descent used here, so it is flagged as an extension worth knowing about rather than built from scratch in this topic.
Summary
| Concept | Description |
|---|---|
| Foundations | |
| Support Vector Machine | Finds the boundary that leaves the widest gap between the two classes |
| Maximum Margin | For separable data, the hyperplane maximizing distance to the nearest point of either class |
| Soft Margin and Hinge Loss | Allows some margin violations, penalized by the hinge loss, controlled by a tuning parameter lambda |
| Support Vectors | The observations closest to the boundary; only these drive where the boundary is placed |
| Fitting by Subgradient Descent | The hinge loss has no smooth derivative at the margin, so it is minimized iteratively rather than in closed form |
| Worked Example | |
| Worked Example: Crop Health Classification | The same twenty NDVI, soil moisture, and temperature fields classified as Healthy or Stressed |
| SVM vs LDA | SVM makes no distributional assumption about the predictors within each class, unlike LDA |
| The Kernel Trick | Swapping a nonlinear kernel for the dot product lets an SVM draw a curved boundary |