| Concept | Description |
|---|---|
| Errors and Power | |
| Type I Error | Rejecting a true null hypothesis (false positive); its rate is set directly by the significance level alpha |
| Type II Error | Failing to reject a false null hypothesis (false negative); its rate is called beta |
| Power of the Test | The probability of correctly rejecting a false null hypothesis, equal to 1 minus beta |
| Ways to Increase Power | Larger sample size, larger effect size, higher alpha (at the cost of more Type I errors), and lower measurement noise |
| Common P-Value Misreading | A p-value is not the probability the null hypothesis is true; it is the probability of data this extreme if the null hypothesis were true |
35 Errors and Power
Every hypothesis test risks two kinds of mistakes. Understanding them — and what “power” means in relation to them — matters for judging whether a test’s conclusion is trustworthy, not just whether its p-value cleared 0.05.
35.1 Errors in Hypothesis Testing
- Type I error (false positive): rejecting the null hypothesis when it’s actually true. Concluding a new bio-fertilizer boosts yield, when in reality it makes no difference — leading a cooperative to recommend (and farmers to pay for) an input that doesn’t help.
- Type II error (false negative): failing to reject the null hypothesis when the alternative is actually true. Concluding a new drought-resistant seed variety performs no differently from the standard one, when it actually does perform better — a missed opportunity the cooperative never adopts.
The significance level \(\alpha\) directly controls the Type I error rate: setting \(\alpha = 0.05\) means accepting a 5% chance of a false positive whenever the null hypothesis is actually true. Lowering \(\alpha\) (say, to 0.01) reduces the Type I error rate, but — all else equal — increases the Type II error rate, since it becomes harder to reject \(H_0\) even when it’s false. Reducing one type of error, without changing anything else about the study, comes at the cost of the other.
35.2 Power of the Test
The power of a hypothesis test is the probability that it correctly rejects a false null hypothesis — in other words, the probability of correctly detecting a real effect when one exists. Power is defined as:
\[ \text{Power} = 1 - \beta \]
where \(\beta\) is the probability of a Type II error. A high-power test is one unlikely to miss a real effect.
Power can be increased by:
- Increasing the sample size — surveying more farms rather than fewer.
- Increasing the effect size — a bio-fertilizer with a large true effect on yield is easier to detect than one with a marginal effect.
- Raising the significance level \(\alpha\) — but this comes at a direct cost: a higher \(\alpha\) raises the Type I error rate, so this trade-off should be made deliberately, not as a shortcut to a “significant” result.
- Reducing measurement noise — more consistent data collection (calibrated instruments, standardized survey protocols) lowers variability and makes a real effect easier to detect.
In practice, sample size is usually the lever a researcher controls most directly — which is why a pilot study or a power analysis, done before data collection, is standard practice for deciding how many farms, plots, or seasons a study needs.
35.2.1 A Common Misreading of the P-Value
A p-value is not the probability that the null hypothesis is true, and it is not the probability that the observed result happened by chance alone. It is the probability of observing data this extreme (or more extreme) if the null hypothesis were true. A p-value of 0.03 does not mean there’s a 97% chance the bio-fertilizer works — it means that, if the fertilizer truly had no effect, data this striking would show up about 3% of the time by sampling variation alone.
This distinction matters in practice: a small p-value from a huge sample can flag a statistically significant but practically trivial effect (a bio-fertilizer that raises yield by 0.5%, “significant” only because 10,000 farms were sampled), while a study too small to have adequate power can fail to detect an effect that’s real and large. Statistical significance and practical significance are different questions — a result worth acting on needs both.