Hypothesis Testing

"To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of." - Sir Ronald A. Fisher, Presidential Address to the First Indian Statistical Congress (1938)

Overview

Hypothesis testing is the art of making principled decisions from data under uncertainty. Where estimation theory (Section02) asks "what is the value of this parameter?", hypothesis testing asks a sharper question: "is this parameter consistent with a specific claim?" This inversion - from continuous estimation to binary decision - is the formal machinery behind every A/B experiment that ships a product feature, every clinical trial that approves a drug, and every benchmark comparison that claims one model outperforms another.

The discipline has two intertwined origins. Ronald Fisher developed the p-value framework in the 1920s: compute how surprising the data would be if the null hypothesis were true, and report that probability as evidence. Jerzy Neyman and Egon Pearson developed the decision-theoretic framework in 1933: pre-commit to a decision rule with controlled error rates before seeing the data. Modern practice blends both - using p-values as a continuous measure of evidence while respecting the Neyman-Pearson discipline of pre-specified significance levels, power analysis, and sample size planning.

For AI and ML, hypothesis testing has never been more important. Every benchmark leaderboard is an implicit multiple-comparison experiment susceptible to false-discovery inflation. Every online A/B test deployed at scale faces the sequential testing problem. Every data pipeline needs distributional drift detection. This section builds the complete framework: from the formal definition of a test statistic through the Neyman-Pearson lemma, classical t/\chi^2/F tests, likelihood ratio tests, multiple testing correction, nonparametric methods, and the sequential A/B testing infrastructure that powers modern ML deployment.

Prerequisites

• Confidence intervals and asymptotic normality of MLE - Section02 Estimation Theory
• Sampling distributions (t, \chi^2, F distributions) - Ch6 Section02 Common Distributions
• Law of large numbers and CLT - Ch6 Section05 Limit Theorems
• Expectation and variance - Ch6 Section04 Expectation and Moments
• Likelihood functions (log-likelihood, score function) - Section02 Section4-Section5

Learning Objectives

After completing this section, you will:

• State the formal definition of a statistical hypothesis and distinguish simple from composite hypotheses
• Define a test statistic, rejection region, and p-value and explain what each does and does not mean
• Quantify Type I error (\alpha), Type II error (\beta), and power (1-\beta), and explain why they cannot all be minimised simultaneously
• Derive sample size requirements from desired \alpha, \beta, and effect size (Cohen's d)
• Derive and apply one-sample t-test, two-sample Welch t-test, chi-squared goodness-of-fit, and one-way ANOVA
• State and prove the Neyman-Pearson lemma and identify when a UMP test exists
• Apply Wilks' theorem to construct generalized likelihood ratio tests for composite hypotheses
• Explain the multiple testing problem and apply Bonferroni, Holm, and Benjamini-Hochberg corrections
• Implement permutation tests, Wilcoxon rank tests, and KS tests for nonparametric inference
• Design a sequential A/B test using SPRT and explain why it avoids the peeking problem
• Identify statistical pitfalls in NLP benchmark comparisons and LLM evaluation leaderboards

1. Intuition

1.1 The Core Question: Evidence or Noise?

Imagine you flip a coin 100 times and observe 63 heads. Is the coin biased, or is 63 just a chance fluctuation from a fair coin? You cannot answer this by staring at the number 63. You need a framework that asks: how often would a fair coin produce 63 or more heads in 100 flips? If the answer is "1 in 1,000 times", you have strong evidence for bias. If the answer is "1 in 5 times", the result is easily explained by chance.

This is the essence of hypothesis testing: quantify how surprising the data would be if the "nothing interesting happened" explanation were true. The "nothing interesting happened" explanation is the null hypothesis $H_0$. The alternative - something systematic is going on - is the alternative hypothesis $H_1$.

The court-room analogy is exact and instructive. In criminal law, the null hypothesis is innocence ($H_0$: defendant is innocent). The prosecution must present evidence so overwhelming that innocence becomes implausible. The defendant is never "proven innocent" - the court simply fails to accumulate enough evidence to reject $H_0$. Similarly, in statistics, we never "prove" the null hypothesis true; we can only fail to reject it. The asymmetry is deliberate: falsely convicting an innocent person (Type I error) is considered worse than failing to convict a guilty one (Type II error), so we set the bar for conviction (rejection) very high.

For AI: Every time you report "model A achieves 87.3% accuracy vs. model B's 86.1% - a statistically significant improvement at p < 0.05", you are running a hypothesis test. The null hypothesis is $H_0: \mu_A = \mu_B$ (no real difference). The question is whether the 1.2% gap is real signal or sampling noise from a finite test set.

1.2 Two Schools of Thought

Modern statistical testing is a marriage of two incompatible philosophies that practitioners blend without always realising it.

Fisher's approach (1925): Compute the p-value - the probability of observing data at least as extreme as what was obtained, under $H_0$. Report it as a continuous measure of evidence against $H_0$. Never pre-specify $H_1$. Never pre-specify a decision threshold. The p-value is just one piece of evidence to weigh alongside domain knowledge and replication. Fisher rejected the idea of a fixed significance threshold as "absurdly academic".

Neyman-Pearson approach (1933): Pre-specify both $H_0$ and $H_1$, a significance level $\alpha$ (Type I error rate), and a desired power $1 - \beta$ (sensitivity to $H_1$). Compute the most powerful test for those hypotheses. Make a binary decision: reject or do not reject. The p-value is irrelevant - what matters is whether $T > c_\alpha$. This framework optimises long-run decision quality across many repeated experiments.

What practitioners actually do: Use the Neyman-Pearson machinery (pre-specify $\alpha$, compute a test statistic, check whether $p < \alpha$) while interpreting the p-value in Fisher's spirit (as a continuous measure of evidence). This hybrid is coherent enough for most purposes but creates confusions - particularly the widespread misinterpretation of p-values as "the probability that $H_0$ is true" (which is Bayesian thinking, belonging to neither school).

FISHER vs. NEYMAN-PEARSON COMPARISON
========================================================================

  Property           | Fisher                  | Neyman-Pearson
  -------------------+-------------------------+----------------------
  Goal               | Measure evidence        | Make optimal decision
  Pre-specify H_1?    | No                      | Yes
  Pre-specify \\alpha?     | No                      | Yes (before seeing data)
  Output             | p-value (continuous)    | Reject / Do not reject
  Power              | Not part of framework   | Central to design
  Philosophical base | Inductive reasoning     | Long-run frequency
  Use case           | Exploratory science     | Industrial quality control

========================================================================

1.3 Historical Timeline

1.4 Why Hypothesis Testing Matters for AI

Model evaluation: Reporting a test accuracy without a confidence interval or significance test is meaningless for model comparison. Is 87.3% vs. 86.1% real? On 1000 test examples at 0.5 base rate, a 1.2% difference corresponds to a two-proportion z-test with $p \approx 0.15$ - not significant. On 10,000 examples, the same gap gives $p \approx 0.002$. Sample size is everything.

A/B testing at scale: Tech companies run thousands of simultaneous A/B experiments. Each one is a two-sample hypothesis test. The infrastructure problem is: how do you test without pre-committing to a fixed sample size (you want to stop early if the effect is clear), while controlling false discovery rate across simultaneous tests?

Data drift detection: Production ML systems degrade when the input distribution changes. Detecting this is a two-sample test: is the distribution of today's features statistically different from training data? The Kolmogorov-Smirnov test, Maximum Mean Discrepancy, and Population Stability Index are all hypothesis tests under the hood.

LLM benchmark evaluation: The 2024-2026 era of LLM leaderboards (MMLU, HumanEval, BIG-Bench, LMSYS Arena) suffers from massive multiple-comparison inflation. If you test 100 models on 50 benchmarks, you expect 250 false discoveries at $\alpha = 0.05$ even if no model truly differs. Proper evaluation requires FDR correction and bootstrap confidence intervals.

Causal inference for RLHF: When measuring whether RLHF improves output quality, you need a randomised controlled design and a proper two-sample test. Confounded comparisons (different prompts, different raters) can produce entirely spurious "improvements".

2. The Formal Framework

2.1 Hypotheses: Null and Alternative

Definition (Statistical Hypothesis). A statistical hypothesis is a claim about the parameter $\theta$ of a probability model $\{P_\theta : \theta \in \Omega\}$. Formally, a hypothesis specifies a subset $\Theta_0 \subseteq \Omega$:

Simple vs. composite hypotheses: - A simple hypothesis pins $\theta$ to a single value: $H_0: \theta = \theta_0$. - A composite hypothesis specifies a range: $H_0: \theta \leq \theta_0$ or $H_0: \theta \neq \theta_0$.

One-sided vs. two-sided tests: - One-sided (directional): $H_1: \theta > \theta_0$ or $H_1: \theta < \theta_0$. Use when the direction of the effect is theoretically specified in advance. - Two-sided (non-directional): $H_1: \theta \neq \theta_0$. Use when any deviation from $\theta_0$ is of interest, or when the direction is unknown.

The asymmetry between $H_0$ and $H_1$: The null hypothesis is the "default" - the claim we assume true unless data provide sufficient evidence against it. This asymmetry has important consequences: - We control the probability of falsely rejecting $H_0$ (Type I error). - We do not automatically control the probability of falsely accepting $H_0$ (Type II error) - that requires separate power analysis. - "Fail to reject $H_0$" is NOT the same as "accept $H_0$". Absence of evidence is not evidence of absence.

Standard examples:

2.2 Test Statistics and Sampling Distributions

Definition (Test Statistic). A test statistic $T = T(X_1, \ldots, X_n)$ is a function of the data that summarises the evidence against $H_0$. A good test statistic: 1. Has a known distribution under $H_0$ (enables exact p-value computation). 2. Takes extreme values when $H_1$ is true (enables detection).

The sampling distribution of $T$ under $H_0$ is central. For example: - If $X_1, \ldots, X_n \overset{iid}{\sim} \mathcal{N}(\mu, \sigma^2)$ with $\sigma$ known, then under $H_0: \mu = \mu_0$:

• If $\sigma$ is unknown, replacing it with the sample standard deviation $S$ introduces extra variability:

The shift from $\mathcal{N}(0,1)$ to $t_{n-1}$ is not a minor detail - for small $n$, the t-distribution has much heavier tails, making it much harder to reject $H_0$ unless the evidence is very strong.

Standardisation principle: Most test statistics are of the form:

This form ensures $T$ is dimensionless and has a tractable distribution under $H_0$.

2.3 Rejection Regions and Critical Values

Definition (Rejection Region). For a test of size $\alpha$, the rejection region $\mathcal{R}_\alpha$ is a subset of the sample space such that:

For a two-sided test of a Gaussian mean with known $\sigma$:

where $z_{\alpha/2}$ is the $(1 - \alpha/2)$ quantile of $\mathcal{N}(0, 1)$. At $\alpha = 0.05$, $z_{0.025} = 1.96$.

Critical value: The boundary $c_\alpha$ such that $P_{H_0}(T > c_\alpha) = \alpha$ (one-sided) or $P_{H_0}(\lvert T \rvert > c_\alpha) = \alpha$ (two-sided). The test rejects $H_0$ iff $T > c_\alpha$ (or $\lvert T \rvert > c_\alpha$).

Exact vs. approximate rejection regions: - For normal populations with known $\sigma$: exact (z-test). - For normal populations with unknown $\sigma$: exact (t-test, using t distribution). - For non-normal populations, large $n$: approximate (CLT makes $Z$ approximately standard normal). - For small $n$, non-normal: nonparametric tests (Section 7).

2.4 The p-Value

Definition (p-Value). The p-value of a test with statistic $T = t_{\text{obs}}$ is:

for a one-sided test, or

for a two-sided test. Equivalently, $p$ is the smallest significance level $\alpha$ at which the observed data would lead to rejection of $H_0$.

Key properties of p-values:

1. Under $H_0$, $p \sim \mathcal{U}(0,1)$. This is a fundamental result: if $H_0$ is true and the test is exact, the p-value is uniformly distributed. This enables calibration checks.
2. Under $H_1$, $p$ is stochastically smaller - it tends toward 0 as sample size grows or effect size increases.
3. $p$ is a random variable. Running the same experiment twice will give different p-values. The p-value quantifies how surprising the specific data are, not how true or false $H_0$ is.

The six most important p-value misinterpretations:

2.5 Duality: Tests and Confidence Intervals

There is an exact correspondence between hypothesis tests and confidence intervals - a fact that is both theoretically beautiful and practically useful.

The Inversion Principle: Given a size-$\alpha$ test for $H_0: \theta = \theta_0$, the $(1-\alpha)$ confidence interval for $\theta$ is:

Conversely, the size-$\alpha$ test rejects $H_0: \theta = \theta_0$ if and only if $\theta_0 \notin \text{CI}_{1-\alpha}$.

Concrete example: The 95% CI for a Gaussian mean with known $\sigma$ is:

The corresponding z-test rejects $H_0: \mu = \mu_0$ at $\alpha = 0.05$ iff $\mu_0$ falls outside this interval - exactly the inversion principle.

Recall: Confidence intervals were derived in Section02 Estimation Theory. The CI for $\mu$ was constructed by pivoting on the standard normal. Here, we see that same CI is the set of null values we would fail to reject.

Practical implication: Reporting a CI is strictly more informative than reporting a p-value. The CI tells you the effect size and uncertainty; the p-value alone only tells you whether a point null is rejected. Always prefer CIs over p-values where possible.

3. Errors, Power, and Sample Size

3.1 Type I and Type II Errors

Any binary decision procedure applied to random data will sometimes make mistakes. There are exactly two ways to err:

Definition (Type I Error). Rejecting $H_0$ when $H_0$ is true. Also called a false positive. Probability = $\alpha$ (the significance level).

Definition (Type II Error). Failing to reject $H_0$ when $H_1$ is true. Also called a false negative. Probability = $\beta$ (depends on the specific alternative $\theta \in \Theta_1$).

ERROR TYPE TABLE
========================================================================

                      |  H_0 True             |  H_1 True
  --------------------+----------------------+----------------------
  Reject H_0           |  Type I Error (\\alpha)    |  CORRECT (Power 1-\\beta)
  Do not reject H_0    |  CORRECT (1-\\alpha)       |  Type II Error (\\beta)

  Analogy:            |  Convict innocent    |  Free the guilty
  Medical test:       |  False positive      |  False negative

========================================================================

The fundamental trade-off: For a fixed sample size $n$, decreasing $\alpha$ (requiring stronger evidence to reject) increases $\beta$ (making it harder to detect real effects). To decrease both simultaneously, you must increase $n$.

Conventional thresholds (and their limitations): - $\alpha = 0.05$: Fisher's suggestion from 1925, now a near-universal convention despite having no theoretical justification. - $\alpha = 0.01$: More stringent; used in physics and genomics. - $\alpha = 0.005$: Proposed by Benjamin et al. (2018) as a new standard to reduce false discoveries. - For AI deployment decisions: The appropriate $\alpha$ depends on the cost of each error type. Deploying a harmful model is a Type II error; rejecting a good model is a Type I error. These costs are application-specific.

3.2 The Power Function

Definition (Power Function). The power function of a test is:

evaluated at every $\theta \in \Omega$.

Key properties of a well-designed power function: - $\pi(\theta) = \alpha$ for $\theta \in \Theta_0$ (the test has correct size). - $\pi(\theta) \to 1$ as $\theta$ moves far from $\Theta_0$ (the test is consistent). - $\pi(\theta)$ is large for $\theta \in \Theta_1$ of practical interest (the test is powerful).

Power at a specific alternative: For the one-sample z-test with $H_0: \mu = \mu_0$ vs. $H_1: \mu = \mu_1 > \mu_0$:

This formula reveals exactly how power depends on: - Effect size $\delta = (\mu_1 - \mu_0)/\sigma$: larger effect -> higher power. - Sample size $n$: larger $n$ -> higher power (as $\sqrt{n}$). - Significance level $\alpha$: larger $\alpha$ -> higher power (but more Type I errors).

Minimum detectable effect (MDE): The smallest $\delta$ such that $\pi(\mu_0 + \delta\sigma) \geq 1 - \beta_{\text{target}}$. Solving for $\delta$:

where $z_\beta = \Phi^{-1}(1-\beta)$. At $\alpha = 0.05$, $\beta = 0.20$: $z_{0.05} + z_{0.20} = 1.645 + 0.842 = 2.487$.

3.3 Effect Size

The problem with raw differences: A mean difference of 2 points on an exam is huge if the standard deviation is 1, but negligible if it is 100. Effect sizes standardise the comparison.

Cohen's d (for means):

Benchmarks (Cohen 1988): $d = 0.2$ small, $d = 0.5$ medium, $d = 0.8$ large.

Cohen's h (for proportions):

The arcsin transform stabilises variance. Benchmarks: $h = 0.2$ small, $h = 0.5$ medium, $h = 0.8$ large.

Cramer's V (for contingency tables with $\chi^2$):

where $r, c$ are the numbers of rows and columns. $V \in [0, 1]$ with 0 = no association.

For AI: When comparing two models' accuracies, report Cohen's h for proportions. A 1% absolute accuracy gain with $h = 0.03$ is small and may not justify the deployment cost; with $h = 0.20$ it is meaningful. Effect size is always reported alongside p-value in rigorous ML papers.

3.4 Sample Size Calculation

Solving the power equation for $n$ gives the required sample size to detect effect size $\delta$ with power $1-\beta$ at level $\alpha$:

(one-sample, one-sided). For two-sided tests replace $z_\alpha$ with $z_{\alpha/2}$.

Two-sample comparison of means (equal group sizes):

where $\delta = (\mu_1 - \mu_2)/\sigma$.

Two-proportion z-test (comparing accuracy rates $p_1$ vs. $p_2$):

where $\bar{p} = (p_1+p_2)/2$.

Worked example: You want to detect a 2% accuracy improvement (from 85% to 87%) with 80% power at $\alpha = 0.05$. - $\bar{p} = 0.86$, $z_{0.025} = 1.96$, $z_{0.20} = 0.842$. - $n \approx \frac{(1.96\sqrt{2(0.86)(0.14)} + 0.842\sqrt{0.85(0.15)+0.87(0.13)})^2}{(0.02)^2} \approx 3{,}600$ per group.

This reveals why benchmark comparisons on small test sets are inconclusive: on 1,000 examples per group, the same test has power $\approx 30\%$.

3.5 ROC Analogy

The error rate trade-off in hypothesis testing is structurally identical to the ROC curve in binary classification:

In both settings, you trace out a curve by varying the threshold (critical value / classification threshold), and the curve represents the complete trade-off between sensitivity and specificity. The AUC of a classifier measures the same thing as the integrated power function of a test: how well the score separates the two classes.

For AI: The ROC analogy makes hypothesis testing intuitive for ML practitioners. Choosing $\alpha = 0.05$ is exactly like choosing a classification threshold to achieve 5% FPR. Power analysis is like calculating recall at that threshold.

4. Classical Parametric Tests

4.1 The Z-Test

Setting: $X_1, \ldots, X_n \overset{iid}{\sim} \mathcal{N}(\mu, \sigma^2)$ with $\sigma^2$ known. Test $H_0: \mu = \mu_0$.

Test statistic:

Rejection regions and p-values: - Two-sided ($H_1: \mu \neq \mu_0$): reject iff $\lvert Z \rvert > z_{\alpha/2}$; $p = 2(1 - \Phi(\lvert Z \rvert))$. - Upper-tailed ($H_1: \mu > \mu_0$): reject iff $Z > z_\alpha$; $p = 1 - \Phi(Z)$. - Lower-tailed ($H_1: \mu < \mu_0$): reject iff $Z < -z_\alpha$; $p = \Phi(Z)$.

Two-sample z-test for proportions: Compare $p_1$ (proportion in group 1) vs. $p_2$ (group 2).

where $\hat{p} = (n_1\hat{p}_1 + n_2\hat{p}_2)/(n_1+n_2)$ is the pooled proportion. This is the standard test for A/B experiments comparing click-through rates or model accuracy.

Validity: Requires $n_1\hat{p}(1-\hat{p}) \geq 5$ and $n_2\hat{p}(1-\hat{p}) \geq 5$. For rare events or small samples, use Fisher's exact test.

4.2 Student's t-Test

The t-test is the workhorse of applied statistics: it handles the realistic case where $\sigma$ is unknown.

One-sample t-test: $X_1, \ldots, X_n \overset{iid}{\sim} \mathcal{N}(\mu, \sigma^2)$, $\sigma$ unknown. Test $H_0: \mu = \mu_0$.

where $S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar{X})^2$ is the sample variance. Reject at level $\alpha$ iff $\lvert T \rvert > t_{n-1, \alpha/2}$.

Gosset's insight: Why $t_{n-1}$ and not $\mathcal{N}(0,1)$? Because $S$ is estimated from data, not known. Substituting $S$ for $\sigma$ introduces additional randomness. The t-distribution has heavier tails to account for this extra uncertainty. As $n \to \infty$, $t_{n-1} \to \mathcal{N}(0,1)$.

Paired t-test: When observations come in natural pairs (before/after measurements, matched subjects), compute differences $D_i = X_i - Y_i$ and apply the one-sample t-test to $D_1, \ldots, D_n$. This removes between-pair variability and dramatically increases power.

Two-sample Welch t-test: Compare means from two independent groups with possibly unequal variances ($\sigma_1^2 \neq \sigma_2^2$):

where the Welch-Satterthwaite degrees of freedom are:

Always use the Welch t-test (not the pooled t-test) unless you have strong prior evidence that $\sigma_1 = \sigma_2$. The pooled t-test's assumption of equal variances is rarely justified and can inflate Type I error badly.

Robustness: The t-test is remarkably robust to non-normality for $n \geq 30$ by the CLT. For small $n$ with strongly skewed or heavy-tailed distributions, use nonparametric alternatives (Wilcoxon signed-rank or Mann-Whitney).

For AI: Use the paired t-test when comparing two models evaluated on the same test examples (paired observations). Use Welch's t-test when comparing models evaluated on different test sets.

4.3 The Chi-Squared Test

Goodness-of-fit test: Observed counts $O_1, \ldots, O_k$ from $n$ observations, expected counts $E_i = np_{i0}$ under $H_0$.

Valid when $E_i \geq 5$ for all $i$. The $\chi^2$ approximation improves with $n$.

Test of independence: A $r \times c$ contingency table with counts $O_{ij}$. Under $H_0$ (row and column variables independent):

Worked example: A model is evaluated on 4 topic categories. Observed errors: [12, 8, 25, 5]. Under $H_0$ (equal error rate): $E_i = 50/4 = 12.5$ each. $\chi^2 = (12-12.5)^2/12.5 + \ldots = 16.08$, df = 3, $p = 0.001$. Strong evidence the error rate varies by topic.

For AI: Chi-squared tests are natural for: - Testing whether a language model's errors are uniformly distributed across categories. - Testing whether a tokenizer's vocabulary coverage is uniform across languages. - Detecting systematic biases in model outputs (contingency table: output category vs. demographic group).

4.4 The F-Test and ANOVA

F-test for two variances: $H_0: \sigma_1^2 = \sigma_2^2$.

Rarely used directly; arises naturally inside ANOVA.

One-way ANOVA: Compare means across $k$ groups with $n_j$ observations in group $j$. Total $N = \sum n_j$.

Decompose total variance into between-group and within-group components:

Test statistic:

Reject $H_0: \mu_1 = \cdots = \mu_k$ when $F > F_{k-1, N-k, \alpha}$.

Post-hoc tests: A significant ANOVA F-test says "at least one mean differs" but not which ones. Post-hoc comparisons (Tukey HSD, Bonferroni-corrected t-tests) identify the specific differences while controlling FWER.

ANOVA assumptions: Normality within groups, equal variances (homoscedasticity), independence. Use Welch's ANOVA or Kruskal-Wallis test when homoscedasticity fails.

4.5 Which Test When

TEST SELECTION FLOWCHART
========================================================================

  How many groups?
  +-- One group
  |   +-- Normal / large n -> One-sample t-test or z-test
  |   +-- Non-normal, small n -> Wilcoxon signed-rank
  +-- Two groups
  |   +-- Paired observations?
  |   |   +-- Yes: Normal -> Paired t-test
  |   |   +-- Yes: Non-normal -> Wilcoxon signed-rank
  |   +-- Independent observations?
  |       +-- Normal: Welch two-sample t-test
  |       +-- Non-normal: Mann-Whitney U test
  +-- Three or more groups
      +-- Normal, equal variances -> One-way ANOVA
      +-- Normal, unequal variances -> Welch's ANOVA
      +-- Non-normal -> Kruskal-Wallis test

  Categorical / count data?
  +-- One sample, counts vs. expected -> Chi-squared GoF
  +-- Two categorical variables -> Chi-squared independence
  +-- Small expected counts (< 5) -> Fisher's exact test

========================================================================

5. Likelihood Ratio Tests and UMP Tests

5.1 The Neyman-Pearson Lemma

The Neyman-Pearson lemma answers a fundamental question: among all tests with size at most $\alpha$, which one maximises power at a specific alternative $\theta_1$?

Theorem (Neyman-Pearson, 1933). Consider testing $H_0: \theta = \theta_0$ vs. $H_1: \theta = \theta_1$ (both simple). The most powerful size-$\alpha$ test rejects $H_0$ when:

where the constant $k_\alpha$ is chosen so that $P_{\theta_0}(\Lambda > k_\alpha) = \alpha$.

Proof sketch: Let $\phi^*$ be the likelihood ratio test and $\phi$ any other test with $\mathbb{E}_{\theta_0}[\phi] \leq \alpha$. We want to show $\mathbb{E}_{\theta_1}[\phi^*] \geq \mathbb{E}_{\theta_1}[\phi]$.

By construction of $\phi^*$: $(\phi^*(\mathbf{x}) - \phi(\mathbf{x}))(p(\mathbf{x}|\theta_1) - k_\alpha p(\mathbf{x}|\theta_0)) \geq 0$ for all $\mathbf{x}$ (both factors have the same sign). Therefore:

since $\mathbb{E}_{\theta_0}[\phi^*] = \alpha \geq \mathbb{E}_{\theta_0}[\phi]$. $\square$

Intuition: The likelihood ratio $\Lambda(\mathbf{x})$ ranks data points by how much more likely they are under $H_1$ than $H_0$. Including the most $H_1$-likely data points in the rejection region maximises power. No other region of the same size can do better.

Example - Gaussian mean: Testing $H_0: \mu = 0$ vs. $H_1: \mu = 1$ with known $\sigma = 1$, $n$ observations.

Rejecting when $\Lambda > k$ is equivalent to rejecting when $\bar{X} > c$ for some threshold $c$. The NP lemma tells us the z-test is the most powerful test for this specific $H_1$.

5.2 Uniformly Most Powerful Tests

The NP lemma gives the most powerful test against a single specific alternative. Can we find a test that is simultaneously most powerful against all alternatives in $\Theta_1$?

Definition (UMP Test). A size-$\alpha$ test $\phi^*$ is uniformly most powerful (UMP) if for every other size-$\alpha$ test $\phi$ and every $\theta \in \Theta_1$:

Monotone Likelihood Ratio (MLR): A family $\{p(\mathbf{x}|\theta)\}$ has the MLR property in statistic $T(\mathbf{x})$ if for $\theta_1 > \theta_2$, the ratio $p(\mathbf{x}|\theta_1)/p(\mathbf{x}|\theta_2)$ is a non-decreasing function of $T(\mathbf{x})$.

Theorem (Karlin-Rubin). If the family has MLR in $T$, then for $H_0: \theta \leq \theta_0$ vs. $H_1: \theta > \theta_0$, the test that rejects when $T > c_\alpha$ is UMP.

Exponential families have MLR: The natural exponential family $p(\mathbf{x}|\eta) \propto \exp(\eta T(\mathbf{x}) - A(\eta))$ has MLR in the sufficient statistic $T(\mathbf{x})$. This means UMP tests exist for one-sided hypotheses about natural parameters of: Gaussian (mean), Bernoulli (logit), Poisson (log-rate), Exponential (rate), Gamma.

When UMP tests do NOT exist: For two-sided alternatives $H_1: \theta \neq \theta_0$, UMP tests generally do not exist. The best we can do is a UMP unbiased test (UMPU), which has power $\geq \alpha$ everywhere in $\Theta_1$.

5.3 The Generalized Likelihood Ratio Test

For composite hypotheses involving multiple parameters, the Neyman-Pearson approach does not directly apply. The GLRT provides a general-purpose solution.

Definition (GLRT). The generalised likelihood ratio is:

where $\hat{\theta}_0$ is the restricted MLE (constrained to $\Theta_0$) and $\hat{\theta}_{\text{MLE}}$ is the unrestricted MLE.

Note $0 \leq \Lambda \leq 1$. Small $\Lambda$ means the constrained model fits much worse than the unconstrained model - evidence against $H_0$.

Wilks' Theorem. Under $H_0$ and regularity conditions, as $n \to \infty$:

where $k = \dim(\Omega) - \dim(\Theta_0)$ is the number of constraints imposed by $H_0$.

Proof idea: Taylor-expand $\log \mathcal{L}(\hat{\theta}_0)$ around $\hat{\theta}_{\text{MLE}}$. The second-order term yields a quadratic form in $(\hat{\theta}_0 - \hat{\theta}_{\text{MLE}})$ scaled by the Fisher information. By asymptotic normality of MLE (Section02), this quadratic form is $\chi^2_k$. $\square$

Example: Testing $H_0: \mu = 0, \sigma^2 = 1$ in a Gaussian model - 2 constraints, so $-2\log\Lambda \sim \chi^2_2$.

For AI: Wilks' theorem underlies model comparison via likelihood. Any time you compare a restricted neural architecture (fewer parameters) to a full model using their log-likelihoods, you are implicitly using a GLRT. The $\chi^2$ approximation provides a principled p-value.

5.4 Score and Wald Tests

The GLRT requires fitting both the restricted and unrestricted models. Two alternatives - the score test and the Wald test - each require fitting only one model. Together with the GLRT, they form the trinity of asymptotic tests, all asymptotically equivalent under $H_0$ and local alternatives.

Wald Test: Fit the unrestricted MLE $\hat{\theta}$ and check if it is far from $\Theta_0$.

where $\hat{I}(\hat{\theta})$ is the observed Fisher information at the MLE. For scalar $\theta$: $W = (\hat{\theta} - \theta_0)^2 / \widehat{\operatorname{Var}}(\hat{\theta})$, which is the square of a z-score.

Score (Rao) Test: Fit only the restricted MLE $\hat{\theta}_0$ and check if the score function (gradient of log-likelihood) is far from zero there.

where $\mathbf{s}(\theta) = \nabla_\theta \log \mathcal{L}(\theta)$ is the score. Under $H_0$, the score should be near zero; a large score indicates the null constraint is straining the model.

TRINITY OF ASYMPTOTIC TESTS
========================================================================

  Test       | Fits model under | Statistic              | Geometric intuition
  -----------+------------------+------------------------+--------------------
  Wald       | H_1 (unrestr.)    | Distance from \\thetahat to \\Theta_0  | How far is MLE from H_0?
  Score      | H_0 (restr.)      | Gradient at \\thetahat_0         | Is restricted fit stable?
  LRT (GLRT) | Both             | Ratio of likelihoods    | How much does H_0 cost?

  All three -> \\chi^2_k under H_0, with same asymptotic power under H_1

========================================================================

When they differ: For small $n$, the three tests can give different p-values. The LRT is generally most accurate; the Wald test can be anti-conservative (over-rejects) for parameters near boundaries. The score test is preferred when fitting the unconstrained model is computationally expensive.

For AI: The Wald test is used to test whether individual neural network weights are significantly different from zero (a form of pruning criterion). The score test is used in online learning to detect if the current gradient is significantly non-zero (an adaptive stopping criterion).

6. Multiple Testing

6.1 The Multiple Testing Problem

Conduct $m$ independent hypothesis tests, each at level $\alpha$. If all $m$ null hypotheses are true, what is the probability of making at least one false rejection?

For $m = 20$ tests at $\alpha = 0.05$: $1 - 0.95^{20} \approx 0.64$. You expect about one false discovery just by chance, even if nothing is real. This is the multiple testing problem - the fundamental challenge underlying the replication crisis in science and the benchmark arms race in ML.

The error metrics:

m_0 and m_1: Of $m$ tests, let $m_0$ be the number of true nulls and $m_1 = m - m_0$ be the number of true alternatives. Define $V$ = false positives, $S$ = true positives, $R = V + S$ = total rejections. Then FWER $= P(V \geq 1)$ and FDR $= \mathbb{E}[V/R]$ (with $V/R = 0$ if $R = 0$).

6.2 Bonferroni and Holm Corrections

Bonferroni correction: Test each hypothesis at level $\alpha/m$. By union bound:

This guarantees FWER $\leq \alpha$ regardless of the dependency structure between tests.

Procedure: Compute p-values $p_1, \ldots, p_m$. Reject $H_{0i}$ if $p_i < \alpha/m$.

Conservative when tests are positively correlated: If tests share the same data, the union bound is loose. Bonferroni wastes power in such settings.

Holm-Bonferroni step-down procedure (1979): 1. Order p-values: $p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(m)}$. 2. Find the smallest $j$ such that $p_{(j)} > \alpha / (m - j + 1)$. 3. Reject $H_{0,(1)}, \ldots, H_{0,(j-1)}$.

Claim: Holm controls FWER at level $\alpha$ and is uniformly more powerful than Bonferroni - it never rejects fewer hypotheses.

Proof sketch: The key step: if $H_{0,(1)}, \ldots, H_{0,(k)}$ are all true nulls (worst case for false positives), Holm's threshold for the $i$-th ordered p-value is $\alpha/(m-i+1) \geq \alpha/m$, which controls each step-wise rejection probability by the Bonferroni argument. $\square$

Sidak correction: For independent tests, the exact threshold is $1 - (1-\alpha)^{1/m}$ (slightly larger than $\alpha/m$, hence slightly more powerful).

6.3 False Discovery Rate

The Benjamini-Hochberg (BH) procedure (1995):

Given p-values $p_{(1)} \leq \cdots \leq p_{(m)}$ ordered from smallest to largest:

1. Find $k = \max\!\left\{i : p_{(i)} \leq \frac{i}{m} \alpha\right\}$.
2. Reject $H_{0,(1)}, \ldots, H_{0,(k)}$.

If no such $k$ exists, reject nothing.

Theorem (Benjamini-Hochberg). Under independence (or positive dependence, PRDS), BH controls FDR at level $\frac{m_0}{m}\alpha \leq \alpha$.

Proof idea (Storey, 2002): Write $\mathbb{E}[V/R] = \sum_{i \in \mathcal{H}_0} P(\text{reject } H_i \mid H_i \text{ true}) \cdot \mathbb{E}[1/R \mid \text{reject }H_i]$. Under independence and the BH threshold, each term is bounded by $\alpha/m$, giving $\mathbb{E}[V/R] \leq m_0\alpha/m \leq \alpha$. $\square$

BH vs. Bonferroni comparison:

q-values: For each rejected hypothesis, the q-value $q_i$ is the minimum FDR level at which $H_i$ would be rejected. Analogous to p-value for FDR control. Introduced by Storey (2002).

For AI: In genomics (the original motivation for BH), researchers test 20,000 gene expression differences - Bonferroni would require $p < 0.0000025$. In ML, testing 100 models across 50 benchmarks creates 5,000 comparisons - FDR control via BH is the appropriate framework.

6.4 NLP Benchmark Comparisons

The leaderboard problem (2024-2026): The major LLM leaderboards (MMLU, HellaSwag, HumanEval, GSM8K, LMSYS Arena, LiveBench) face severe multiple testing issues:

1. Model selection bias: Model developers report best results across many runs, architectures, and prompting strategies. This is implicit p-hacking at the model level.
2. Benchmark contamination: Test sets get into training data over time. Reported improvements may reflect memorisation rather than generalisation.
3. Multiple comparisons across benchmarks: A model scoring highest on 3 of 10 benchmarks is not necessarily best - with 100 models and 10 benchmarks, 50 false "wins" are expected by chance at $\alpha = 0.05$.
4. Non-stationary test sets: Rolling evaluation windows mean the effective sample size is unclear.

Rigorous evaluation practices: - Report bootstrap CIs (Section 7.5) on aggregate scores. - Apply BH correction when comparing $m$ models. - Use held-out evaluation sets not seen during model selection. - Report McNemar's test for paired model comparisons on the same instances. - Require pre-registration of evaluation protocols before model training.

Significance thresholds for benchmarks: At $m = 100$ comparisons, BH at $\alpha = 0.05$ requires $p_{(k)} \leq 0.05k/100$. For the top-ranked model to be significantly different from the second, you typically need $n \geq 5,000$ test examples per benchmark.

6.5 Bayesian Alternative Preview

Classical multiple testing corrections (Bonferroni, BH) are explicitly frequentist: they control long-run error rates without asking "what is the probability that $H_i$ is true?" The Bayesian framework offers a fundamentally different approach.

Preview: Bayesian Model Comparison

Given observed data $\mathbf{x}$, the Bayes factor for $H_0$ vs. $H_1$ is: $$B_{01} = \frac{P(\mathbf{x} \mid H_0)}{P(\mathbf{x} \mid H_1)} = \frac{\int p(\mathbf{x}|\theta)p(\theta|H_0)d\theta}{\int p(\mathbf{x}|\theta)p(\theta|H_1)d\theta}$$ The Bayes factor naturally accounts for model complexity (Occam's razor) and provides a direct measure of evidence. In the multiple testing setting, Bayesian methods control the posterior expected FDR by placing a prior on the proportion of true nulls $\pi_0$.

-> Full treatment: Section04 Bayesian Inference

7. Nonparametric Tests

7.1 Why Nonparametric?

Classical tests (t, F, z) assume the data follow a specific parametric family (usually Gaussian). What if: - The data are ordinal (rankings, Likert scales)? - The sample size is small ($n < 15$) and normality is implausible? - The data contain extreme outliers that violate distributional assumptions? - You want an exact test without large-sample approximations?

Nonparametric tests make no (or minimal) distributional assumptions. The trade-off: they are typically less powerful than their parametric counterparts when the parametric assumptions hold, but more robust when those assumptions fail.

Distribution-free vs. nonparametric: A test is distribution-free if its null distribution is the same regardless of the data distribution. Permutation tests and rank tests are distribution-free. A test is nonparametric in the sense that it estimates a non-finite-dimensional quantity. The terms are often used interchangeably.

7.2 Permutation and Randomization Tests

Motivation: If $H_0: F_1 = F_2$ (two groups have the same distribution), then under $H_0$, the group labels are exchangeable. We can compute the null distribution of any test statistic exactly by enumerating all $\binom{n_1+n_2}{n_1}$ label permutations.

Algorithm (two-sample permutation test): 1. Compute the observed test statistic $T_{\text{obs}}$ (e.g., difference in means $\bar{X}_1 - \bar{X}_2$). 2. For $b = 1, \ldots, B$: randomly permute the combined group labels; recompute $T^{(b)}$. 3. Estimate p-value: $p = (\#\{b : T^{(b)} \geq T_{\text{obs}}\} + 1) / (B + 1)$.

Properties: - Exact (not asymptotic) when all permutations are enumerated. - Valid for any test statistic, no distributional assumptions. - Computationally expensive for large samples (use $B \approx 10{,}000$ Monte Carlo permutations). - Any statistic: Unlike t-tests, permutation tests work for medians, trimmed means, Gini coefficients, AUC, or custom ML metrics.

For AI: When comparing two LLMs on a shared benchmark, a permutation test on per-example score differences avoids all distributional assumptions. With $n = 500$ test examples, a permutation test with $B = 10{,}000$ has better calibration than a t-test.

7.3 Rank-Based Tests

Mann-Whitney U test (Wilcoxon rank-sum): Two-sample test. Combine and rank all $n_1 + n_2$ observations. Let $W$ = sum of ranks in group 1. Under $H_0$: $\mathbb{E}[W] = n_1(n_1+n_2+1)/2$.

AUC connection: The Mann-Whitney U statistic has a beautiful probabilistic interpretation:

This is exactly the empirical AUC - the probability that a random draw from group 1 exceeds a random draw from group 2. A Mann-Whitney test is equivalent to testing whether AUC $= 0.5$. This unifies hypothesis testing with classifier evaluation.

Wilcoxon signed-rank test: Paired two-sample test. Compute differences $D_i = X_i - Y_i$. Rank the $\lvert D_i \rvert$. $W^+ = $ sum of ranks of positive differences. Under $H_0: \mathbb{E}[W^+] = n(n+1)/4$.

Kruskal-Wallis test: Extension of Mann-Whitney to $k \geq 2$ groups. Rank all $N$ observations jointly; the test statistic is based on the between-group variability of ranks. Under $H_0$: approximately $\chi^2_{k-1}$.

7.4 Kolmogorov-Smirnov Test

One-sample KS test: Test whether a sample comes from a specified distribution $F_0$.

where $F_n$ is the empirical CDF. Under $H_0$, $D_n$ has a known distribution (Kolmogorov distribution) independent of $F_0$.

Two-sample KS test: Test whether two samples share the same distribution.

Under $H_0$: $\sqrt{\frac{nm}{n+m}} D_{n,m} \overset{d}{\to} K$ where $K$ is the Kolmogorov distribution. Reject for large $D_{n,m}$.

Properties: - Sensitive to differences in location, scale, and shape - not just means. - Consistent against all continuous alternative distributions. - Less powerful than t-test against pure location shifts (it wastes power on shape/scale). - CDF-based, so naturally handles multivariate data via joint ECDFs (though the asymptotic distribution changes).

For AI - Data drift detection: The two-sample KS test is the most widely used drift detector in production ML:

DRIFT DETECTION PIPELINE
========================================================================

  Training data distribution: F_train(x)
  Production batch (daily):   F_prod(x)

  For each feature j:
    Compute D_j = sup_x |F_train(x_j) - F_prod(x_j)|
    Compute p_j = KS test p-value
    Apply BH correction across all features

  Alert if: \\exists j with q_j < 0.05 (BH-adjusted)
  Report: Which features drifted and by how much

========================================================================

Limitations: KS tests features marginally (one at a time). For multivariate drift, use Maximum Mean Discrepancy (MMD) or domain classifier-based tests.

7.5 Bootstrap Hypothesis Tests

The bootstrap (Efron 1979) provides a general method for constructing null distributions without parametric assumptions. Reviewed in Section02 for CIs; here we use it for testing.

Bootstrap test for two-sample means ($H_0: \mu_1 = \mu_2$): 1. Compute $T_{\text{obs}} = \bar{X}_1 - \bar{X}_2$. 2. Shift both samples to have equal means: $\tilde{X}_i = X_i - \bar{X}_1 + \bar{X}$, $\tilde{Y}_j = Y_j - \bar{Y} + \bar{X}$ (where $\bar{X}$ is the pooled mean). Now $H_0$ holds exactly in the shifted data. 3. Draw bootstrap samples from $\tilde{X}$ and $\tilde{Y}$, compute $T^{(b)} = \bar{X}^{(b)} - \bar{Y}^{(b)}$. 4. $p = P(T^{(b)} \geq T_{\text{obs}})$.

Bootstrap for complex statistics: The t-test requires normality for exact validity. Bootstrap tests work for any statistic: median differences, correlation coefficients, AUC, BLEU scores, F1 scores - anything you can compute on resampled data.

For AI: Bootstrap CI and tests are standard for NLP evaluation. When comparing BLEU or ROUGE scores, a paired bootstrap test (sampling test-set instances) is the gold standard, as used by Koehn (2004) and standard in MT evaluation.

8. A/B Testing and ML Evaluation

8.1 The A/B Testing Framework

An A/B test is a randomised controlled experiment comparing two (or more) versions of a system. The framework:

1. Define the primary metric (CTR, revenue per user, model accuracy).
2. Define guardrail metrics (latency, crash rate, user retention) that must not degrade.
3. Pre-specify $\alpha$, $\beta$, and MDE (minimum detectable effect) before data collection.
4. Randomise units (users, sessions, requests) to treatment and control.
5. Run until the pre-specified sample size is reached (or sequential stopping criterion is met).
6. Analyse with the appropriate test and report effect size + CI.

Unit of randomisation: The choice of randomisation unit is critical. - User-level: Each user sees only one variant. Avoids within-user interference. Used for UI changes. - Session-level: Users can see both variants in different sessions. Higher statistical power but potentially biased. - Request-level: Each request is independently assigned. Maximum power; appropriate for stateless ML inference.

The experimental design matters more than the test: Even the perfect hypothesis test cannot salvage a poorly designed experiment. Survivorship bias, Novelty effects, and Simpson's paradox are design problems, not statistical ones.

8.2 Sequential A/B Testing

The peeking problem: If you check p-values daily and stop when $p < 0.05$, you have not run an $\alpha = 0.05$ test. You have run a repeated testing procedure with inflated Type I error. For $\alpha = 0.05$, peeking 5 times inflates the actual error rate to $\approx 0.14$; peeking indefinitely drives Type I error to 1.

Sequential Probability Ratio Test (SPRT, Wald 1943): A test with no fixed sample size that guarantees both $P(\text{false positive}) \leq \alpha$ and $P(\text{false negative}) \leq \beta$, while stopping as early as possible.

Given observations $x_1, x_2, \ldots$, compute the log likelihood ratio:

Decision rule: At each step: - If $\ell_n \geq \log(1-\beta)/\alpha$: reject $H_0$ (accept $H_1$). - If $\ell_n \leq \log\beta/(1-\alpha)$: accept $H_0$. - Otherwise: continue sampling.

Wald's bounds: These thresholds guarantee $P(\text{false positive}) \leq \alpha$ and $P(\text{false negative}) \leq \beta$, with minimal expected sample size compared to fixed-$n$ tests.

Mixture Sequential Ratio Test (mSPRT): An extension by Johari et al. (2022) that uses a mixture distribution over $H_1$, producing "always-valid p-values" - p-values that can be checked at any time without inflating error rates. This is the theoretical foundation for modern continuous A/B testing platforms (Spotify, Netflix, Booking.com).

For AI: The standard "wait N days, then look at p-value" A/B protocol is inefficient. Sequential testing with mSPRT or anytime-valid confidence sequences allows early stopping when effects are clear, reducing the cost of failed experiments by 30-50%.

8.3 Model Comparison Tests

Paired t-test on accuracy: Compare model A and B on the same $n$ test examples. For each example $i$, record whether model A was correct ($a_i \in \{0,1\}$) and model B correct ($b_i \in \{0,1\}$). Compute differences $D_i = a_i - b_i$ and apply a one-sample t-test to $D_1, \ldots, D_n$.

McNemar's test: More appropriate for binary outcomes. Contingency table of (correct/incorrect) pairs:

The test statistic $\chi^2 = (n_{10} - n_{01})^2 / (n_{10} + n_{01}) \sim \chi^2_1$ under $H_0$ (models have equal accuracy). Only discordant pairs ($n_{10}$, $n_{01}$) contribute - concordant pairs carry no information about which model is better.

Diebold-Mariano test: For comparing two forecasters. Test $H_0: \mathbb{E}[d_t] = 0$ where $d_t = L(e_{1t}) - L(e_{2t})$ is the loss differential at time $t$. Uses a HAC-robust variance estimator to handle serial correlation in $d_t$.

8.4 Data Drift Detection

Covariate shift: The input distribution $P(X)$ changes between training and deployment, but the conditional $P(Y|X)$ remains stable. This is the most common drift type in production ML.

Concept drift: The relationship $P(Y|X)$ changes. Harder to detect without labels.

Statistical tests for drift:

Population Stability Index (PSI): A practitioner-favourite drift metric:

where $b$ indexes histogram bins. PSI < 0.1: no drift; 0.1-0.25: moderate drift; > 0.25: significant drift requiring retraining. Structurally equivalent to a symmetrised KL divergence.

8.5 LLM Evaluation and Leaderboards

Current best practices (2025-2026) for rigorous LLM evaluation:

1. Bootstrap confidence intervals on aggregate scores: Sample test instances with replacement $B = 1{,}000$ times, compute benchmark score each time. Report median +/- 95% CI.

2. McNemar's test for pairwise comparisons: For two LLMs on the same benchmark, use McNemar's test (paired binary outcomes) rather than an unpaired proportion test.

3. BH-corrected comparisons across benchmarks: When reporting "Model X outperforms Model Y on $k$ benchmarks", apply BH at $\alpha = 0.05$ and report the q-values.

4. Effect sizes, not just p-values: Report Cohen's h (for accuracy differences), or normalised score differences, alongside p-values.

5. Power analysis for benchmark design: A new benchmark should be designed with sufficient items ($n$) to detect a 1% accuracy difference with 80% power. For $\alpha = 0.05$, $\beta = 0.20$, this requires $n \approx 7{,}200$ per model comparison.

6. Chatbot Arena / ELO ratings: LMSYS Arena uses pairwise preference data to estimate ELO ratings. The uncertainty in ELO estimates should be reported as CIs derived from bootstrap resampling of preference pairs.

9. Common Mistakes

10. Exercises

Exercise 1 * - One-Sample t-Test from Scratch

A language model's token latency (ms) is measured on 20 requests: mean = 47.3 ms, sample std = 8.1 ms. The SLA requires mean latency \leq 45 ms.

(a) State $H_0$ and $H_1$ precisely. Is this one-sided or two-sided? (b) Compute the t-statistic and degrees of freedom. (c) Find the critical value at $\alpha = 0.05$. (d) Compute the exact p-value. (e) State your conclusion in plain English.

Exercise 2 * - Chi-Squared Goodness-of-Fit

A text classifier should distribute predictions uniformly across 5 categories. On 500 test examples, observed counts are [87, 113, 95, 102, 103].

(a) State $H_0$ and compute expected counts. (b) Compute the $\chi^2$ statistic. (c) Find the p-value (df = 4, $\chi^2_{4, 0.05} = 9.49$). (d) Is the distribution significantly non-uniform at $\alpha = 0.05$? (e) Compute Cramer's V and interpret the effect size.

Exercise 3 * - Power and Sample Size

You want to detect that model A has a higher accuracy than model B ($p_A = 0.88$, $p_B = 0.85$) with 80% power at $\alpha = 0.05$.

(a) Compute Cohen's h for this effect. (b) Derive the required sample size per group for a two-proportion z-test. (c) What is the power if you can only collect $n = 2{,}000$ per group? (d) Plot the power curve as a function of $n$ (from 500 to 5000). (e) What sample size gives 95% power?

Exercise 4 ** - Neyman-Pearson Lemma for Exponential Distribution

Let $X_1, \ldots, X_n \overset{iid}{\sim} \text{Exp}(\lambda)$. Test $H_0: \lambda = \lambda_0$ vs. $H_1: \lambda = \lambda_1 > \lambda_0$.

(a) Write the likelihood ratio $\Lambda(\mathbf{x}) = \mathcal{L}(\lambda_1)/\mathcal{L}(\lambda_0)$. (b) Show that rejecting when $\Lambda > k$ is equivalent to rejecting when $\bar{X} < c$ for some $c$. (c) Find $c$ in terms of $\alpha$, $\lambda_0$, and $n$ using the fact that $2\lambda_0 n\bar{X} \sim \chi^2_{2n}$. (d) Verify that this test has the correct size $\alpha = 0.05$ for $\lambda_0 = 1$, $n = 10$. (e) Is this test UMP for all $\lambda_1 > \lambda_0$? Justify using the MLR property.

Exercise 5 ** - Multiple Testing Correction

In an NLP evaluation, 50 hypothesis tests are conducted (comparing a new model to baseline on 50 benchmarks). The raw p-values are generated synthetically.

(a) Simulate 45 true nulls (p-values ~ Uniform[0,1]) and 5 true alternatives (p-values ~ Beta(0.2, 1)). (b) Count discoveries with no correction at $\alpha = 0.05$. (c) Apply Bonferroni correction and count discoveries. (d) Apply BH correction and count discoveries. (e) Across 1000 simulation replications, estimate the empirical FWER and FDR for each method. Plot the results.

Exercise 6 ** - Permutation Test for Two-Sample Means

Two LLMs (A and B) are evaluated on 30 shared test prompts. Model A scores: drawn from $\mathcal{N}(0.72, 0.1^2)$. Model B scores: drawn from $\mathcal{N}(0.68, 0.1^2)$.

(a) Compute the observed mean difference. (b) Implement a permutation test with $B = 10{,}000$ permutations. (c) Compute the permutation p-value. (d) Compare to a Welch t-test p-value on the same data. (e) Repeat 500 times and compare the empirical Type I error rates of both tests under $H_0$.

Exercise 7 *** - Sequential A/B Test with SPRT

Two model variants A and B are tested on streaming requests. $H_0: p_A = p_B = 0.80$ vs. $H_1: p_A = 0.85, p_B = 0.80$. Set $\alpha = 0.05$, $\beta = 0.20$.

(a) Derive the log-likelihood ratio $\ell_n$ for Bernoulli outcomes. (b) Compute the Wald stopping boundaries $A = (1-\beta)/\alpha$ and $B = \beta/(1-\alpha)$. (c) Simulate the sequential process until stopping or $n = 2{,}000$. Plot $\ell_n$ vs. $n$ with the boundaries. (d) Compare the expected stopping time under $H_0$ and $H_1$. (e) Estimate the empirical Type I error rate over 1000 simulated experiments where $H_0$ is true. Verify it is $\leq \alpha = 0.05$.

Exercise 8 *** - KS-Based Data Drift Detector for LLM Features

Build a drift detection system for an LLM serving system. Reference distribution: sentence embedding norms $\sim \mathcal{N}(10, 1^2)$. Production batches vary.

(a) Simulate a reference dataset of 5,000 embeddings and three daily production batches: no drift, moderate drift ($\mu = 10.5$), severe drift ($\mu = 12$). (b) Apply the two-sample KS test to each batch vs. reference. (c) Apply BH correction across the 3 batch comparisons. (d) Implement a sliding window detector: alert if last 3 consecutive days all have KS $p < 0.1$. (e) Compare KS vs. a t-test drift detector: which is more sensitive to scale changes? Demonstrate with a scenario where the mean is unchanged but $\sigma = 1.5$.

11. Why This Matters for AI (2026 Perspective)

12. Conceptual Bridge

Looking Back: Estimation Theory (Section02)

Hypothesis testing builds directly on estimation theory (Section02). The estimators derived there - sample mean $\bar{X}$, MLE $\hat{\theta}$, sample variance $S^2$ - reappear as the building blocks of every test statistic. The confidence interval duality (Section 2.5) makes this connection explicit: a confidence interval is the set of parameter values we would fail to reject, and the test is an inversion of the CI procedure.

The asymptotic normality of MLE (Section02 Section8) is the theoretical engine behind the Wald test and the asymptotic validity of the z-test for large samples. Fisher information (Section02 Section4) enters hypothesis testing through the score test and through the Cramer-Rao bound's role in characterising optimal tests.

Confidence intervals (Section02 Section7) and hypothesis tests are dual constructions: every confidence interval corresponds to a test, and every test corresponds to a confidence interval. Reporting CIs is strictly more informative, because CIs communicate effect size and precision, not just a binary reject/don't-reject decision.

Looking Forward: Bayesian Inference (Section04)

Section Section04 provides the Bayesian counterpart to every major concept in this section:

The philosophical divide is deep: frequentists refuse to assign probabilities to hypotheses (hypotheses are fixed; data are random). Bayesians treat parameters and hypotheses as random variables with prior distributions. The Bayesian framework provides a natural solution to multiple testing (the prior on $\pi_0$ automatically corrects for multiplicity), but requires specification of that prior - a potential source of subjectivity.

For AI practitioners, the practical choice is often dictated by computational constraints and domain norms. Frequentist tests are fast and require no prior specification; Bayesian methods provide richer inference at the cost of prior elicitation and posterior computation.

Looking Further Forward: Regression (Section06)

The F-test derived in Section 4.4 reappears in Section06 as the overall F-test for regression significance. The t-test for individual regression coefficients ($H_0: \beta_j = 0$) is a direct application of the Wald test from Section 5.4. The multiple testing problem reappears when testing many coefficients simultaneously in high-dimensional regression - LASSO regularisation can be seen as an implicit multiple testing correction that shrinks small coefficients to zero.

POSITION IN CURRICULUM
========================================================================

  Section02 ESTIMATION THEORY
    MLE, Fisher info, CIs, asymptotic normality
           |
           v (test statistics are functions of estimators)
  Section03 HYPOTHESIS TESTING  <-- YOU ARE HERE
    p-values, power, t/\\chi^2/F tests, LRT, multiple testing,
    nonparametric tests, A/B testing, sequential tests
           |                         |
           v                         v
  Section04 BAYESIAN INFERENCE    Section06 REGRESSION ANALYSIS
  (Bayes factors, posterior    (F-test, t-tests on
   probability of hypotheses)   regression coefficients)
           |
           v
  Ch8 OPTIMISATION
  (RLHF experiment design,
   model selection, early stopping)

========================================================================

Hypothesis testing is the formal language of scientific comparison. Every claim that "model A is better than model B", every statement that "this feature is significant", every assertion that "the distribution shifted" - all of these are hypothesis tests, whether or not they are recognised as such. Making these tests explicit, pre-specified, and properly corrected is the difference between rigorous science and post-hoc storytelling.

Appendix A: Key Distributions in Hypothesis Testing

Relationships: - $Z^2 \sim \chi^2_1$ - $t_n^2 \sim F_{1,n}$ (square of t is F with df 1 in numerator) - $\chi^2_k / k \cdot \chi^2_m / m = F_{k,m}$ (ratio of chi-squared variables / their df) - $-2\log\Lambda \overset{d}{\to} \chi^2_k$ (Wilks' theorem)

Appendix B: Critical Values Reference

Appendix C: Statistical Testing in Python

from scipy import stats
import numpy as np

# One-sample t-test
t_stat, p_val = stats.ttest_1samp(data, popmean=mu0)

# Welch two-sample t-test
t_stat, p_val = stats.ttest_ind(group1, group2, equal_var=False)

# Paired t-test
t_stat, p_val = stats.ttest_rel(before, after)

# Chi-squared goodness-of-fit
chi2, p_val = stats.chisquare(observed, expected)

# Chi-squared test of independence
chi2, p_val, dof, expected = stats.chi2_contingency(contingency_table)

# One-way ANOVA
f_stat, p_val = stats.f_oneway(group1, group2, group3)

# Mann-Whitney U test
u_stat, p_val = stats.mannwhitneyu(x, y, alternative='two-sided')

# Kolmogorov-Smirnov two-sample test
ks_stat, p_val = stats.ks_2samp(sample1, sample2)

# Wilcoxon signed-rank test
w_stat, p_val = stats.wilcoxon(differences)

Appendix D: Glossary

Appendix E: Proof of the Bonferroni Inequality

Lemma (Bonferroni). Let $A_1, \ldots, A_m$ be events. Then:

Proof: By inclusion-exclusion and the fact that all higher-order intersection terms are non-negative:

from scipy import stats
import numpy as np

# -- One-sample tests --------------------------------------------------
# Z-test (manually, since scipy has no z-test function)
z = (xbar - mu0) / (sigma / np.sqrt(n))
p_two = 2 * (1 - stats.norm.cdf(abs(z)))

# One-sample t-test
t, p = stats.ttest_1samp(x, popmean=mu0)

# Wilcoxon signed-rank test
w, p = stats.wilcoxon(x - mu0)

# -- Two-sample tests --------------------------------------------------
# Welch's t-test (ALWAYS use equal_var=False unless you have strong reason)
t, p = stats.ttest_ind(x, y, equal_var=False)

# Paired t-test
t, p = stats.ttest_rel(x, y)

# Mann-Whitney U
u, p = stats.mannwhitneyu(x, y, alternative='two-sided')

# Two-sample KS test
d, p = stats.ks_2samp(x, y)

# Permutation test (scipy >= 1.8)
result = stats.permutation_test((x, y),
    statistic=lambda a, b: a.mean() - b.mean(),
    n_resamples=10_000, alternative='two-sided')
p = result.pvalue

# -- Multi-sample tests ------------------------------------------------
# One-way ANOVA
f, p = stats.f_oneway(group1, group2, group3)

# Kruskal-Wallis
h, p = stats.kruskal(group1, group2, group3)

# -- Categorical tests -------------------------------------------------
# Chi-squared goodness-of-fit
chi2, p = stats.chisquare(observed, f_exp=expected)

# Chi-squared test of independence
chi2, p, dof, expected = stats.chi2_contingency(table)

# McNemar's test (statsmodels)
from statsmodels.stats.contingency_tables import mcnemar
result = mcnemar([[n11, n10], [n01, n00]])
p = result.pvalue

from statsmodels.stats.multitest import multipletests

# Bonferroni, Holm, BH corrections
reject, p_adj, _, _ = multipletests(p_values, alpha=0.05, method='bonferroni')
reject, p_adj, _, _ = multipletests(p_values, alpha=0.05, method='holm')
reject, p_adj, _, _ = multipletests(p_values, alpha=0.05, method='fdr_bh')

# method options: 'bonferroni', 'holm', 'fdr_bh', 'fdr_by', 'sidak',
#                 'holm-sidak', 'simes-hochberg', 'hommel'

from statsmodels.stats.power import (
    TTestIndPower, TTestOneSamplePower, NormalIndPower
)

# Required sample size for two-sample t-test
analysis = TTestIndPower()
n = analysis.solve_power(effect_size=0.5, alpha=0.05, power=0.80,
                          ratio=1.0, alternative='two-sided')

# Power at fixed n
power = analysis.power(effect_size=0.5, nobs1=64, alpha=0.05,
                        ratio=1.0, alternative='two-sided')

# For proportions
from statsmodels.stats.proportion import proportion_effectsize, zt_ind_solve_power
h = proportion_effectsize(0.87, 0.85)  # Cohen's h
n = zt_ind_solve_power(effect_size=h, alpha=0.05, power=0.80)