Hypothesis testing

Hypothesis testing formalizes the question "could this result have happened by chance?" You begin with a null hypothesis (H₀) stating that there is no effect or no difference — a drug works no better than placebo, two population means are equal — and an alternative hypothesis (H₁ or Hₐ) stating what you suspect is true instead. The test never proves the null; it either rejects it or fails to reject it.

The mechanics run in a fixed order. Choose a significance level α, commonly 0.05, before looking at the data. Collect the sample and compute a test statistic — a z-score, t-statistic, or chi-square value depending on the design and whether the population standard deviation is known. Convert that statistic to a p-value, the probability of observing a result at least as extreme as yours if the null were true. If p ≤ α, reject the null and call the result statistically significant; if p > α, fail to reject it.

Two errors are possible. A Type I error rejects a true null — a false positive — and occurs with probability α. A Type II error fails to reject a false null, a false negative, occurring with probability β; the test's power is 1 − β. Lowering α reduces false positives but increases false negatives, and increasing sample size is the main way to improve power without trading one error for the other. Statistical significance is also not the same as practical importance: a large enough sample can make a trivial difference significant.

Hypothesis testing is core content across several exams. AP Statistics devotes whole units to significance testing and to the error types, USMLE Step 1 biostatistics tests p-values and error interpretation in clinical study scenarios, and the FE Mechanical exam covers z-tests and t-tests within probability and statistics. Learn the logic once and the same framework carries across all three.