Chapter 6 — Probability Theory

"Probability is not about chance — it is about the precise language for reasoning under uncertainty. That language is the foundation of every machine learning model."

Overview

Probability theory is the mathematical framework for reasoning about uncertainty, and every component of modern machine learning is probabilistic at its core. Training data is a sample from an unknown distribution; model parameters are drawn randomly at initialisation; predictions are distributions over outcomes; generative models explicitly model data-generating processes; language model outputs are probability distributions over tokens.

This chapter builds the formal theory from sample spaces and probability axioms through random variables, common distributions, joint and conditional probability, expectation and moments, concentration inequalities, stochastic processes, and Markov chains. The goal is not abstract measure theory for its own sake — every concept is grounded in its ML applications, from cross-entropy loss to the central limit theorem underlying batch statistics to Markov chain Monte Carlo in Bayesian inference.

The conceptual arc: probability spaces (§01) → named distributions and their properties (§02) → joint and conditional probability, independence (§03) → expectation, variance, and moments (§04) → concentration inequalities and tail bounds (§05) → stochastic processes and martingales (§06) → Markov chains and steady-state theory (§07).

---

Subsection Map

---

Reading Order and Dependencies

01-Introduction-and-Random-Variables   (foundation: probability spaces, random variables)
        ↓
02-Common-Distributions                (vocabulary: named distributions and their properties)
        ↓
03-Joint-Distributions                 (multivariate: joint, conditional, Bayes' theorem)
        ↓
04-Expectation-and-Moments             (summary statistics: E[X], Var[X], covariance)
        ↓
05-Concentration-Inequalities          (bounds: how far can a random variable stray from its mean?)
        ↓
06-Stochastic-Processes                (sequences: LLN, CLT, Gaussian processes)
        ↓
07-Markov-Chains                       (structured processes: MCMC, Bayesian inference machinery)
        ↓
07-Statistics (Chapter 7)             (estimation: MLE, MAP, confidence intervals, hypothesis tests)

---

What Belongs Where — Canonical Homes

This table is the authoritative scoping guide. If a topic has a canonical home, every other section must give at most a 1–2 paragraph preview with a forward/backward reference — never a full treatment.

---

Overlap Danger Zones

1. Probability Axioms ↔ Common Distributions

• §01 introduces probability through axioms and defines the simplest distributions (Bernoulli, uniform) as concrete examples of the formalism.
• §02 gives the complete treatment of each named distribution — parameters, PDF/PMF, CDF, mean, variance, MGF, relationships. §02 must not re-derive the axioms; §01 must not provide the full moments and relationships of Binomial, Gaussian, etc.

2. Conditional Probability ↔ Joint Distributions ↔ Bayes

• §01 defines conditional probability $P(A|B) = P(A \cap B)/P(B)$ and states Bayes' theorem as a formula.
• §03 develops the full machinery: joint PDF/PMF, marginalisation, conditional distributions for random variables, chain rule of probability, and Bayes' theorem with prior/likelihood/posterior structure.
• §07 applies Bayesian updates in the Markov chain setting. §07 must backward-reference §03 for Bayes' theorem.

3. Expectation ↔ Moments ↔ MGF

• §04 is the exclusive home of the expectation operator, variance, covariance, higher moments, LOTUS, and moment generating functions as analytical tools.
• §02 may state the mean and variance of each named distribution (as facts), but must not derive them from first principles — §04 owns those derivations.

4. Concentration Inequalities ↔ Law of Large Numbers

• §05 proves Markov's and Chebyshev's inequalities and derives Hoeffding/Chernoff bounds, culminating in PAC-learning generalisation bounds.
• §06 proves the weak and strong LLN using the tools from §05. §06 should backward-reference §05 rather than re-proving the concentration results it uses.

5. Markov Chains ↔ Conditional Independence

• §03 defines conditional independence $X \perp Y | Z$.
• §07 uses conditional independence as the definition of the Markov property: $X_{t+1} \perp X_{0:t-1} | X_t$. §07 must backward-reference §03 rather than re-define conditional independence.

6. Gaussian Distribution ↔ Gaussian Process ↔ Multivariate Gaussian

• §02 defines the univariate Gaussian: PDF, CDF (via $\Phi$), moments, MGF, standard normal.
• §03 defines the multivariate Gaussian: joint PDF, covariance matrix, marginals, conditionals, affine transformations.
• §06 defines the Gaussian process as an infinite-dimensional extension: any finite collection of function values follows a multivariate Gaussian. §06 must backward-reference §02 and §03 rather than re-derive Gaussian properties.

---

Forward and Backward Reference Format

Forward reference (full treatment is later):

> **Preview: Bayes' Theorem for Distributions**
> For random variables $X$ and $Y$ with joint density $p(x,y)$, Bayes' theorem becomes
> $p(x|y) = p(y|x)p(x)/p(y)$, giving the posterior over $x$ given observation $y$.
>
> → _Full treatment: [Joint Distributions](../03-Joint-Distributions/notes.md)_

Backward reference (builds on earlier section):

> **Recall:** The expected value $\mathbb{E}[X] = \int x\, p(x)\, dx$ was defined in
> [Expectation and Moments](../04-Expectation-and-Moments/notes.md). The covariance
> $\text{Cov}(X, Y) = \mathbb{E}[(X - \mu_X)(Y - \mu_Y)]$ defined there is the key
> parameter of the multivariate Gaussian developed here.

---

Key Cross-Chapter Dependencies

From Chapter 5 — Multivariate Calculus:

• Integration in multiple variables (§01/§03 use multivariate integrals for marginalisation)
• Lagrange multipliers (§04 in Ch5) → maximum entropy distributions (§02 exponential family derivation)
• Gradients and log-derivatives → score functions, Fisher information matrix (§03, §04)

From Chapter 4 — Calculus Fundamentals:

• Integration (computing CDFs, normalisation constants, moments)
• Series (computing MGFs, Poisson approximation)

Into Chapter 7 — Statistics:

• §02 (distributions) → likelihood functions and MLE
• §03 (Bayes' theorem) → MAP estimation and Bayesian inference
• §04 (expectation, variance) → unbiased estimators, sample statistics
• §05 (concentration) → confidence intervals and hypothesis testing
• §06 (CLT) → asymptotic normality of estimators
• §07 (Markov chains) → MCMC-based posterior sampling

Into Chapter 8 — Optimisation:

• §02 (Gaussian, softmax) → cross-entropy loss derivation
• §03 (conditional distributions) → EM algorithm (expectation step)
• §04 (KL divergence, entropy) → variational inference

Into Chapter 9 — Information Theory:

• §02 (exponential family) → natural exponential family and sufficient statistics
• §04 (entropy = $-\mathbb{E}[\log p(X)]$) → Shannon entropy definition
• §05 (MGF) → Cramér's theorem, large deviations

---

ML Concept Map

---

Prerequisites

Before starting this chapter, ensure you are comfortable with:

• Integration (single and multi-variable): computing areas under curves, normalisation constants — §03-Integration
• Series: power series, convergence — §04-Series-and-Sequences
• Set theory basics: union, intersection, complement — introduced in §01 of this chapter
• Matrix algebra: matrix–vector products, determinants, positive definite matrices — Chapter 3
• Lagrange multipliers (for exponential family derivation): §04-Optimality-Conditions

---

← Previous Chapter: Multivariate Calculus | Next Chapter: Statistics →