58  Naive Bayes

Naive Bayes classifies an observation by applying Bayes’ theorem directly, then making one deliberately simplifying assumption to keep the calculation tractable: that every predictor is conditionally independent of every other predictor, given the class.

\[ P(\text{class} \mid X_1, \ldots, X_p) \propto P(\text{class}) \prod_{j=1}^{p} P(X_j \mid \text{class}) \]

The independence assumption is almost never exactly true, NDVI and soil moisture are certainly related to each other, not just to crop status, which is exactly why the method is called “naive.” In practice it tends to work well anyway, because classification only needs the ranking of the class probabilities to come out right, not their exact values, and the independence assumption’s errors often cancel out across predictors rather than compounding.

58.1 The Gaussian Version

For continuous predictors, the standard approach assumes each predictor follows a normal distribution within each class, with its own class-specific mean and standard deviation:

\[ P(X_j \mid \text{class} = c) = \frac{1}{\sqrt{2\pi\sigma_{jc}^2}} \exp\left(-\frac{(X_j - \mu_{jc})^2}{2\sigma_{jc}^2}\right) \]

Fitting the model means estimating \(\mu_{jc}\) and \(\sigma_{jc}\) for every predictor within every class, nothing more. Classifying a new observation multiplies together the density each predictor’s value gets under a given class, times that class’s prior probability, and picks whichever class produces the largest result.

58.2 Worked Example

The same twenty fields used throughout this topic.

Field NDVI Soil Moisture (%) Avg Temp (°C) Status
1 0.42 18 36 Stressed
2 0.38 16 37 Stressed
3 0.75 38 26 Healthy
4 0.68 32 28 Healthy
5 0.45 20 34 Stressed
6 0.80 40 25 Healthy
7 0.55 25 31 Stressed
8 0.72 35 27 Healthy
9 0.40 17 38 Stressed
10 0.78 42 24 Healthy
11 0.60 28 30 Healthy
12 0.48 22 33 Stressed
13 0.82 44 23 Healthy
14 0.35 15 38 Stressed
15 0.65 30 29 Healthy
16 0.50 24 32 Stressed
17 0.85 45 22 Healthy
18 0.44 19 35 Stressed
19 0.58 27 30 Healthy
20 0.52 23 33 Stressed

58.3 Naive Bayes in R

Fitted here from its own definition using base R’s dnorm(), rather than a package such as e1071 that may not be available in every environment.

58.4 Reading the Result

Naive Bayes should land on the same borderline field as LDA and the SVM did, and the disagreement or agreement between methods on that one ambiguous case is itself informative: when several independently built classifiers agree, the prediction can be trusted with more confidence than when they split. Where Naive Bayes earns its keep is speed and simplicity, fitting the model here is nothing more than computing a handful of means and standard deviations, which makes it a reasonable first classifier to try on a new problem, and a natural baseline the more elaborate techniques in this topic should be expected to beat.

Summary

Concept Description
Foundations
Naive Bayes Classifies using Bayes' theorem with the simplifying assumption that predictors are independent given the class
The Conditional Independence Assumption Assumes each predictor contributes to the class probability independently of every other predictor
Why It Works Despite Being Naive Classification only needs the correct ranking of class probabilities, not exact values, so errors often cancel out
Gaussian Naive Bayes Assumes each continuous predictor follows a normal distribution with its own mean and SD in each class
Fitting the Model Fitting requires only estimating each predictor's class-specific mean and standard deviation
Worked Example
Worked Example: Crop Health Classification The same twenty NDVI, soil moisture, and temperature fields classified as Healthy or Stressed
Classifying a New Observation Multiplies each predictor's density under a class by that class's prior, then picks the largest result
Naive Bayes as a Fast Baseline Fast and simple to fit, making it a reasonable first model and a baseline for comparison