53 Classification Model Evaluation
A logistic regression model outputs a probability, not a decision. Turning that probability into a yes/no call requires a threshold, commonly 0.5 as a starting point, and once a threshold is chosen, the model’s predictions can be checked against what actually happened. The standard tool for that comparison is the confusion matrix, a simple table that sorts every prediction into one of four outcomes:
| Actual: Outbreak | Actual: No Outbreak | |
|---|---|---|
| Predicted: Outbreak | True Positive (TP) | False Positive (FP) |
| Predicted: No Outbreak | False Negative (FN) | True Negative (TN) |
Every classification metric in this section is built from these four counts.
53.1 Accuracy, Precision, Recall, and F1
\[ \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} \]
Accuracy is the share of all predictions that were correct, but it can be a misleading headline number on its own. If outbreaks were rare, say occurring on only 5 percent of fields, a model that always predicts “no outbreak” would be 95 percent accurate while being useless for the one purpose it exists for: catching outbreaks. That is why accuracy is normally reported alongside two more targeted metrics.
\[ \text{Precision} = \frac{TP}{TP + FP} \qquad \qquad \text{Recall (Sensitivity)} = \frac{TP}{TP + FN} \]
Precision answers: of the fields the model flagged as at risk, how many actually had an outbreak? Recall answers: of the fields that actually had an outbreak, how many did the model catch? These two pull against each other. A model can raise recall by flagging more fields as at risk, but that usually drops precision by flagging more fields that turn out fine. The F1-score balances the two into a single number:
\[ F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} \]
Which metric matters most depends on the cost of each error type. Missing a real outbreak (a false negative) is usually more expensive for a farmer than investigating a field that turns out fine (a false positive), which argues for prioritizing recall over precision in this particular application, even at some cost to overall accuracy.
53.2 Classification Model Evaluation in R
53.3 The ROC Curve and AUC
Fixing the threshold at 0.5 is a choice, not a requirement, and the Receiver Operating Characteristic (ROC) curve shows how the model behaves across every possible threshold at once. It plots the true positive rate (recall) against the false positive rate (\(FP / (FP + TN)\)) as the threshold sweeps from 1 down to 0. A model with no predictive value traces a diagonal line from corner to corner; a model that separates the two classes perfectly hugs the top-left corner, catching every true outbreak with no false alarms.
The Area Under the Curve (AUC) condenses the whole curve into one number between 0 and 1. An AUC of 0.5 means the model is no better than a coin flip; an AUC of 1.0 means perfect separation. AUC has a useful interpretation independent of any single threshold: it equals the probability that the model would rank a randomly chosen actual-outbreak field higher in predicted risk than a randomly chosen actual-no-outbreak field, which is exactly what the Mann-Whitney calculation in the code above computes directly, without needing to trace the full curve.
53.4 Best Practices for a Trustworthy Classifier
- Report precision, recall, and F1 alongside accuracy, never accuracy by itself, especially when the outcome is imbalanced, as pest outbreaks or disease incidents usually are.
- Choose the classification threshold deliberately based on the relative cost of a false positive versus a false negative, rather than defaulting to 0.5 without thinking it through.
- Evaluate on a held-out sample the model was not fitted on, or with k-fold cross-validation, the same caution that applies to regression models. A classifier that looks strong on the fields it was trained on can still perform poorly on next season’s fields.
- Treat AUC as a summary of overall ranking quality, not a replacement for checking precision and recall at the threshold actually being used to make decisions.
Summary
| Concept | Description |
|---|---|
| Confusion Matrix and Core Metrics | |
| Confusion Matrix | Sorts predictions into true positives, true negatives, false positives, and false negatives |
| Accuracy | The share of all predictions that were correct; can look strong even when a model is not useful |
| Why Accuracy Alone Can Mislead | A model that always predicts the majority class can post high accuracy while missing every rare case |
| Precision and Recall | Precision measures correctness among flagged cases; recall measures how many real cases were caught |
| F1-Score | The harmonic mean of precision and recall, balancing the two into a single number |
| Threshold and ROC/AUC | |
| Choosing a Threshold | The 0.5 default is a choice; the right threshold depends on the relative cost of each error type |
| ROC Curve | Plots true positive rate against false positive rate across every possible classification threshold |
| AUC | Area under the ROC curve; equals the probability the model ranks a true positive above a true negative |
| Best Practices | |
| Hold-Out and Cross-Validation for Classifiers | Evaluating on data the model was not fitted on is as essential for classifiers as it is for regression models |