48 Regression Diagnostics and Model Evaluation
A regression model that fits the sample data well is not automatically a model worth trusting. Its coefficients and predictions only hold up if the assumptions behind the model are reasonably satisfied, and its usefulness for forecasting depends on how large its errors typically are on data it has not already seen. Regression diagnostics are the checks, most of them built on the model’s residuals (the difference between an observed value and the value the model predicted), that a fitted model should pass before it is used to make a real decision, whether that is recommending a fertilizer dose or estimating a season’s likely yield.
48.1 Residual Analysis
The residual for observation \(i\) is:
\[ e_i = Y_i - \hat{Y}_i \]
Two plots do most of the diagnostic work:
- Residuals vs Fitted: plotting residuals against the model’s predicted values should show a random scatter around zero with no visible pattern. A curved band suggests the true relationship bends and a straight line is the wrong model. A funnel shape, residuals fanning out as fitted values grow, suggests the variance of the errors is not constant.
- Normal Q-Q Plot: plots the residuals against the values a perfectly normal distribution would produce. Points sitting close to the diagonal line support the normality assumption; a strong curve or S-shape does not.
48.2 Checking the Assumptions
Linearity
Checked with the residuals-vs-fitted plot. A systematic curve there means the model form itself, not just the data, needs revisiting, which is exactly why polynomial and nonlinear alternatives exist alongside plain linear regression.
Homoscedasticity (constant variance)
Checked visually (a funnel-shaped residual spread signals a problem) or formally with a test such as Breusch-Pagan. This is precisely the kind of pattern the rainfall-yield example in the quantile regression section showed: variance that widens as the predictor grows.
Normality of residuals
Checked with the Q-Q plot, or formally with the Shapiro-Wilk test. This assumption matters most for the validity of significance tests and confidence intervals; a model can still produce reasonable point predictions even when residuals are somewhat non-normal.
Independence of errors
Matters most for data collected in a sequence, such as yields recorded season after season on the same field. Checked with the Durbin-Watson test, which detects autocorrelation among residuals.
Multicollinearity
Checked with the Variance Inflation Factor (VIF) for each predictor in a multiple regression model:
\[ VIF_j = \frac{1}{1 - R_j^2} \]
where \(R_j^2\) comes from regressing predictor \(X_j\) on every other predictor in the model. A VIF above roughly 5 to 10 signals that a predictor is substantially explained by the others, which inflates its coefficient’s standard error and makes that coefficient unreliable to interpret on its own.
48.3 Model Evaluation Metrics
R-squared and Adjusted R-squared
Covered in the earlier sections of this topic: the share of variance in the outcome the model explains, with Adjusted \(R^2\) correcting for the number of predictors used.
Root Mean Squared Error (RMSE)
The typical size of a prediction error, expressed in the same units as the outcome:
\[ RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(Y_i - \hat{Y}_i)^2} \]
Because errors are squared before averaging, RMSE penalizes a few large misses more heavily than many small ones.
Mean Absolute Error (MAE)
The average absolute size of the prediction errors:
\[ MAE = \frac{1}{n}\sum_{i=1}^{n}|Y_i - \hat{Y}_i| \]
MAE treats every error in proportion to its size, so it is less sensitive to a single large outlier than RMSE. A lower RMSE or MAE indicates a better-fitting model, and the two are most useful when comparing two or more candidate models on the same dataset rather than read in isolation.
48.4 Regression Diagnostics in R
48.5 Best Practices for a Trustworthy Regression Model
- Look at the residual plots. Never rely on \(R^2\) alone to decide whether a model is adequate.
- Report Adjusted \(R^2\) alongside RMSE or MAE, so the model’s fit and its practical prediction error are both visible.
- Check VIF for every predictor in a multiple regression model before interpreting individual coefficients.
- Remember that a statistically significant coefficient shows association captured in this dataset, not proof that the predictor causes the outcome, unless the data came from a controlled experiment with random assignment.
- Whenever the goal is prediction rather than explanation, validate the model on a hold-out sample it was not fitted on. A model that fits the training sub-plots closely can still perform poorly on next season’s fields if it has picked up quirks specific to the plots it was trained on.
Summary
| Concept | Description |
|---|---|
| Residual Diagnostics | |
| Residual | The difference between an observed value and the value predicted by the regression model |
| Residuals vs Fitted Plot | Plots residuals against fitted values to check for non-linearity or non-constant variance |
| Normal Q-Q Plot | Plots residuals against theoretical normal quantiles to check the normality assumption |
| Assumption Checks | |
| Linearity Check | Verified visually with the residuals-vs-fitted plot for systematic curvature |
| Homoscedasticity Check | Verified visually, or formally with a test such as Breusch-Pagan, for a funnel-shaped residual spread |
| Normality Check | Verified with a Q-Q plot or formally with the Shapiro-Wilk test |
| Independence Check (Durbin-Watson) | Verified with the Durbin-Watson test, especially important for sequential or seasonal data |
| Multicollinearity and VIF | Variance Inflation Factor measures how much a predictor's variance is inflated by correlation with the others |
| Evaluation Metrics | |
| R-squared / Adjusted R-squared | The proportion of variance explained by the model, with the adjusted version penalizing extra predictors |
| RMSE | Root Mean Squared Error, the typical size of a prediction error in the outcome's own units |
| MAE | Mean Absolute Error, the average absolute prediction error, less sensitive to outliers than RMSE |
| Best Practices | |
| Correlation vs Causation Caveat | A significant coefficient shows association, not proof of causation, outside a controlled experiment |
| Hold-out Validation | Evaluating a model's predictive accuracy on data it was not fitted on |