Expectation and Moments

Scaling. For iid $X_1,\ldots,X_n$ with variance $\sigma^2$:

The variance of the sample mean decreases as $1/n$ - the statistical basis for why larger datasets reduce estimation uncertainty.

For AI: Weight initialisation strategies (Xavier/Glorot, He) are derived from variance propagation. If $\mathbf{h}^{[l]} = W^{[l]}\mathbf{h}^{[l-1]}$, then the variance of each output neuron is $n_{\text{in}} \cdot \text{Var}(w_{ij}) \cdot \text{Var}(h^{[l-1]}_j)$. To keep variance constant across layers (preventing vanishing/exploding activations): $\text{Var}(w_{ij}) = 1/n_{\text{in}}$ (Xavier) or $2/n_{\text{in}}$ (He for ReLU). This is the variance-of-sums formula applied to layer activations.

5. Conditional Expectation

5.1 Definition and Basic Properties

The conditional expectation $\mathbb{E}[Y \mid X = x]$ is the expected value of $Y$ given that $X$ takes the specific value $x$. It is a function of $x$, not a number.

Discrete case:

Continuous case:

As a random variable. When we write $\mathbb{E}[Y \mid X]$ (without specifying the value of $X$), we mean the random variable obtained by composing the function $x \mapsto \mathbb{E}[Y \mid X=x]$ with $X$. This is a function of $X$ and is therefore itself a random variable.

Basic properties:

1. $\mathbb{E}[aY + bZ \mid X] = a\mathbb{E}[Y \mid X] + b\mathbb{E}[Z \mid X]$ (linearity)
2. $\mathbb{E}[g(X) \cdot Y \mid X] = g(X) \cdot \mathbb{E}[Y \mid X]$ (taking out known factors)
3. If $X \perp Y$: $\mathbb{E}[Y \mid X] = \mathbb{E}[Y]$ (conditioning on independent variable doesn't help)
4. $\mathbb{E}[\mathbb{E}[Y \mid X]] = \mathbb{E}[Y]$ (tower property - see Section5.2)

Example. Let $(X,Y)$ be jointly uniform on the triangle $\{(x,y): 0 \leq y \leq x \leq 1\}$. From Section03 (Appendix V.1), the conditional density is $f_{Y|X}(y|x) = 1/x$ on $[0,x]$. Therefore:

The conditional mean of $Y$ given $X = x$ is $x/2$ - exactly half of $x$, which makes sense since $Y$ is uniform on $[0,x]$.

5.2 Tower Property (Iterated Expectation)

The tower property (also called the law of iterated expectations or Adam's law) is one of the most useful results in probability:

More generally, for any function $g(X)$:

Proof (continuous case):

Intuition. If you first average $Y$ within groups defined by $X$, then average those group means weighted by group size, you get the overall mean of $Y$.

Example: Expected test score. A class has 40% students from Group A (mean score 75) and 60% from Group B (mean score 85). Overall expected score: $0.4 \times 75 + 0.6 \times 85 = 81$. This is exactly the tower property: $\mathbb{E}[Y] = \mathbb{E}[\mathbb{E}[Y|G]]$ where $G$ is group membership.

For AI: The tower property underlies the EM algorithm. The E-step computes $Q(\theta|\theta^{(t)}) = \mathbb{E}_{Z|X,\theta^{(t)}}[\log p(X,Z|\theta)]$ - the expected complete-data log-likelihood, where the expectation is over the latent variable $Z$ given the observed $X$. The tower property guarantees that maximising $Q$ increases $\log p(X|\theta)$, the marginal likelihood. In policy gradient methods, $\mathbb{E}_\tau[R] = \mathbb{E}_{s_0}[\mathbb{E}_\tau[R | s_0]]$ - decomposing trajectory returns by initial state.

5.3 Conditional Variance and Law of Total Variance

The conditional variance of $Y$ given $X = x$ is:

Law of Total Variance (Eve's Law):

This decomposes total variance into: - Within-group variance: $\mathbb{E}[\text{Var}(Y \mid X)]$ - average variability within each value of $X$. - Between-group variance: $\text{Var}(\mathbb{E}[Y \mid X])$ - variability of the group means.

Proof:

Using the tower property twice and writing $\mu(x) = \mathbb{E}[Y|X=x]$:

And $(\mathbb{E}[Y])^2 = (\mathbb{E}[\mu(X)])^2$. Therefore:

For AI: The law of total variance is the mathematical foundation of the bias-variance decomposition (Section8). For a model trained on dataset $\mathcal{D}$: total error = expected within-dataset error + between-dataset variance. It also explains why mixture models (GMMs) combine within-component variance and between-component variance.

5.4 AI Applications of Conditional Expectation

Optimal predictor. The function $f^*(x) = \mathbb{E}[Y \mid X=x]$ minimises the expected squared error:

Proof: For any $f$,

The cross term vanishes because $\mathbb{E}[Y - f^*(X) \mid X] = \mathbb{E}[Y|X] - f^*(X) = 0$, so by the tower property $\mathbb{E}[(Y-f^*(X))g(X)] = 0$ for any $g$. The last term is non-negative. Therefore $\mathbb{E}[(Y-f(X))^2] \geq \mathbb{E}[(Y-f^*(X))^2]$. $\square$

This is profound: the best possible predictor (in MSE sense) is the conditional expectation. Neural networks are universal approximators trained to approximate $f^* = \mathbb{E}[Y|X]$.

Rao-Blackwell theorem. In estimation theory: if $\hat{\theta}$ is any unbiased estimator of $\theta$ and $T$ is a sufficient statistic, then $\tilde{\theta} = \mathbb{E}[\hat{\theta} \mid T]$ is also unbiased and has variance $\leq \text{Var}(\hat{\theta})$. Conditioning on sufficient statistics never increases variance. This is the variance reduction analogue of why attention over relevant context helps.

Attention as conditional expectation. The output of self-attention at position $i$ is:

This is the conditional expectation of the value vectors under the attention distribution $\alpha_i$ over key positions. High-entropy attention = averaging many values; low-entropy (sharp) attention = conditioning on one specific value.

6. Moment Generating Functions and Characteristic Functions

6.1 MGF Definition and Basic Properties

The moment generating function (MGF) of a random variable $X$ is:

defined for all $t$ in an open interval containing 0. If no such interval exists (the integral diverges), the MGF is said not to exist.

Moments from the MGF. Differentiating $k$ times and evaluating at $t=0$ gives the $k$-th raw moment:

Proof: $M_X(t) = \mathbb{E}[e^{tX}] = \mathbb{E}\left[\sum_{k=0}^\infty \frac{(tX)^k}{k!}\right] = \sum_{k=0}^\infty \frac{\mathbb{E}[X^k]}{k!} t^k$ (when interchange is valid).

Differentiating $k$ times: $M_X^{(k)}(0) = \mathbb{E}[X^k]$. $\square$

Uniqueness theorem. If $M_X(t) = M_Y(t)$ for $t$ in an open neighbourhood of 0, then $X$ and $Y$ have the same distribution. The MGF uniquely determines the distribution (when it exists).

Existence. The MGF exists when $\mathbb{E}[e^{tX}] < \infty$ for $|t| < t_0$ for some $t_0 > 0$. Heavy-tailed distributions (Cauchy, Pareto with small $\alpha$, log-normal) do NOT have finite MGFs. The characteristic function (Section6.4) always exists and is the preferred tool for such distributions.

6.2 MGF of Common Distributions

Bernoulli($p$):

Check: $M'(t) = pe^t$, $M'(0) = p = \mathbb{E}[X]$. [ok]

Binomial($n,p$): (sum of $n$ iid Bernoulli)

First moment: $M'(0) = np$. [ok]

Poisson($\lambda$):

Derivation: $M(t) = \sum_{k=0}^\infty e^{tk} \frac{\lambda^k e^{-\lambda}}{k!} = e^{-\lambda}\sum_k \frac{(\lambda e^t)^k}{k!} = e^{-\lambda} e^{\lambda e^t} = e^{\lambda(e^t-1)}$.

, , so . [ok]

Normal($\mu, \sigma^2$):

Derivation: Complete the square in the exponent of $\int e^{tx} \cdot \frac{1}{\sqrt{2\pi}\sigma} e^{-(x-\mu)^2/(2\sigma^2)} dx$.

, , . [ok]

Exponential($\lambda$):

Derivation: $\int_0^\infty e^{tx} \lambda e^{-\lambda x} dx = \lambda \int_0^\infty e^{-({\lambda-t})x} dx = \frac{\lambda}{\lambda-t}$ (converges iff $t < \lambda$).

, , . [ok]

For AI: MGFs appear in the proof of the Chernoff bound (Section05), which gives exponentially sharp tail bounds. The exponential form $\mathbb{E}[e^{tX}]$ is closely related to the log-partition function in the exponential family: $A(\eta) = \log Z(\eta)$. The moments of the sufficient statistic can be computed by differentiating $A(\eta)$ - the MGF and the log-partition function are connected by a change of variables.

6.3 MGF of Sums and the Reproductive Property

Key theorem. If $X \perp Y$:

Proof: $M_{X+Y}(t) = \mathbb{E}[e^{t(X+Y)}] = \mathbb{E}[e^{tX} e^{tY}] = \mathbb{E}[e^{tX}]\mathbb{E}[e^{tY}] = M_X(t)M_Y(t)$, using independence for the third equality. $\square$

This is the MGF version of the convolution theorem. It provides elegant proofs of reproductive properties:

Gaussian reproductive property. If $X \sim \mathcal{N}(\mu_1,\sigma_1^2)$ and $Y \sim \mathcal{N}(\mu_2,\sigma_2^2)$ independently:

This is the MGF of $\mathcal{N}(\mu_1+\mu_2, \sigma_1^2+\sigma_2^2)$. So $X+Y \sim \mathcal{N}(\mu_1+\mu_2, \sigma_1^2+\sigma_2^2)$.

Poisson reproductive property. If $X \sim \text{Poisson}(\lambda_1)$, $Y \sim \text{Poisson}(\lambda_2)$ independently:

So $X+Y \sim \text{Poisson}(\lambda_1+\lambda_2)$.

6.4 Characteristic Function

The characteristic function of $X$ is:

where $i = \sqrt{-1}$.

Key advantage: $\varphi_X(t)$ exists for ALL distributions (since $|e^{itX}| = 1$ always), unlike the MGF which may not exist for heavy-tailed distributions.

Connection to MGF: $\varphi_X(t) = M_X(it)$ when the MGF exists. But the characteristic function is defined even when the MGF is not.

Moments from $\varphi_X$. When moments exist:

Inversion formula. The distribution is uniquely determined by its characteristic function:

(when $f_X$ is continuous). This is the Fourier inversion formula - the characteristic function IS the Fourier transform of the PDF.

For AI: Characteristic functions appear in the analysis of convergence of distributions (Central Limit Theorem proofs use characteristic functions because they always exist). The score-based diffusion model framework uses the score function $\nabla_x \log p(x)$, which is related to the characteristic function via Fourier analysis. Random Fourier Features (Rahimi & Recht, 2007) approximate kernel functions via random sampling from the characteristic function of the kernel.

6.5 Cumulants and the Cumulant Generating Function

The cumulant generating function (CGF) is the log of the MGF:

Cumulants $\kappa_k$ are the coefficients in the Taylor expansion of $K_X(t)$:

so $\kappa_k = K_X^{(k)}(0)$.

First four cumulants: - $\kappa_1 = \mathbb{E}[X] = \mu$ (mean) - $\kappa_2 = \text{Var}(X) = \sigma^2$ (variance) - $\kappa_3 = \mathbb{E}[(X-\mu)^3] = \mu_3$ (third central moment = $\gamma_1 \sigma^3$) - $\kappa_4 = \mu_4 - 3\sigma^4$ (excess kurtosis $\times \sigma^4$)

Key property. For independent $X \perp Y$:

Cumulants of a sum of independent variables add - this is the additivity property that makes cumulants more natural than moments for sums.

Normal distribution: $K_X(t) = \mu t + \sigma^2 t^2/2$. All cumulants $\kappa_k = 0$ for $k \geq 3$. This characterises the normal distribution - it is the unique distribution with all cumulants beyond the second equal to zero.

7. Jensen's Inequality and Moment Inequalities

7.1 Jensen's Inequality

Definition. A function $f : \mathbb{R} \to \mathbb{R}$ is convex if for all $x, y$ and $\lambda \in [0,1]$:

Geometrically: the chord between any two points lies above the curve. Equivalently (for twice-differentiable $f$): $f''(x) \geq 0$ everywhere.

Theorem (Jensen's Inequality). If $f$ is convex and $\mathbb{E}[|X|] < \infty$:

Proof (via supporting hyperplane). Since $f$ is convex, for any $a$ there exists a supporting hyperplane at $a$: a linear function $\ell(x) = f(a) + c(x-a)$ such that $f(x) \geq \ell(x)$ for all $x$ (where $c$ is the subgradient at $a$). Setting $a = \mathbb{E}[X]$:

Taking expectations of both sides:

Equality condition: $f(\mathbb{E}[X]) = \mathbb{E}[f(X)]$ iff $f$ is linear on the support of $X$, or $X$ is almost surely constant.

For concave $f$: $f(\mathbb{E}[X]) \geq \mathbb{E}[f(X)]$ (Jensen's inequality reverses).

Common applications:

7.2 Applications: KL Divergence and ELBO

KL divergence non-negativity. The Kullback-Leibler divergence is:

Theorem (Gibbs' inequality). $\text{KL}(p \| q) \geq 0$, with equality iff $p = q$ almost everywhere.

Proof via Jensen:

where we applied Jensen's inequality with the concave function $\log$. Therefore $\text{KL}(p\|q) \geq 0$. $\square$

ELBO derivation via Jensen. For a latent variable model $p_\theta(x) = \int p_\theta(x,z) \, dz$:

where the inequality applies Jensen's to the concave $\log$. This lower bound is the ELBO:

The gap between $\log p_\theta(x)$ and the ELBO is exactly $\text{KL}(q_\phi \| p_\theta(z|x)) \geq 0$. Maximising the ELBO tightens this bound.

Log-sum inequality. For non-negative $a_i, b_i$:

This is the discrete version of KL non-negativity via Jensen.

7.3 Cauchy-Schwarz Inequality

Theorem (Cauchy-Schwarz for expectations). For any random variables $X$ and $Y$:

with equality iff $Y = cX$ almost surely for some constant $c$.

Proof. For any $t \in \mathbb{R}$:

This is a quadratic in $t$ that is always non-negative. A quadratic $at^2 + bt + c \geq 0$ for all $t$ iff discriminant $b^2 - 4ac \leq 0$. Here $a = \mathbb{E}[X^2]$, $b = -2\mathbb{E}[XY]$, $c = \mathbb{E}[Y^2]$:

Corollary ($|\rho| \leq 1$). Apply the Cauchy-Schwarz inequality to the centred variables $X - \mu_X$ and $Y - \mu_Y$:

Dividing: $\rho^2 \leq 1$.

For AI: Cauchy-Schwarz appears in attention - the dot product $q^\top k$ is bounded by $\|q\| \|k\|$, motivating the $\sqrt{d_k}$ scaling in scaled dot-product attention ($\text{softmax}(QK^\top/\sqrt{d_k})V$). Without this scaling, the dot products can become large in magnitude for high-dimensional keys/queries, causing the softmax to saturate and gradients to vanish.

7.4 Lyapunov Inequality

Theorem (Lyapunov's inequality). For $0 < r \leq s$:

In words: the $L^r$ norm of $X$ (as a random variable) is non-decreasing in $r$.

Proof. Apply Jensen's inequality with the convex function $f(t) = t^{s/r}$ (convex since $s/r \geq 1$) to the random variable $Y = |X|^r$:

Taking $(1/s)$ powers of both sides: $(\mathbb{E}[|X|^r])^{1/r} \leq (\mathbb{E}[|X|^s])^{1/s}$. $\square$

Consequence. If the $s$-th moment is finite, all lower moments are finite. If $\mathbb{E}[X^4] < \infty$, then $\mathbb{E}[X^3], \mathbb{E}[X^2], \mathbb{E}[|X|] < \infty$ as well.

For AI: Lyapunov's inequality underlies the Lyapunov CLT (a variant of the central limit theorem for non-identically distributed variables). In the theory of stochastic gradient descent, the existence of higher moments of the gradient controls the tightness of convergence bounds.

7.5 Preview: Tail Bounds from Moments

The most direct application of moments to probability is bounding how far a random variable can stray from its mean. These are covered fully in Section05, but we preview the key results here.

Preview: Markov's Inequality

For any non-negative random variable $X$ and $t > 0$: $$P(X \geq t) \leq \frac{\mathbb{E}[X]}{t}$$

Proof: $\mathbb{E}[X] = \int_0^\infty xf(x)\,dx \geq \int_t^\infty xf(x)\,dx \geq t\int_t^\infty f(x)\,dx = t\,P(X\geq t)$. The mean bounds the probability of large values.

-> Full treatment and applications: Section05 Concentration Inequalities

Preview: Chebyshev's Inequality

For any random variable $X$ with mean $\mu$ and variance $\sigma^2$: $$P(|X - \mu| \geq k\sigma) \leq \frac{1}{k^2}$$

Proof: Apply Markov's inequality to $(X-\mu)^2 \geq k^2\sigma^2$: $P((X-\mu)^2 \geq k^2\sigma^2) \leq \mathbb{E}[(X-\mu)^2]/(k^2\sigma^2) = 1/k^2$. The variance bounds how often $X$ deviates from its mean by $k$ standard deviations.

-> Full treatment and sharper bounds: Section05 Concentration Inequalities

8. Bias-Variance Decomposition

8.1 Statistical Risk and Optimal Predictor

Consider predicting $Y$ from $X$ using a function $f : \mathbb{R}^d \to \mathbb{R}$. The statistical risk (expected squared loss) is:

Theorem. The minimiser of $\mathcal{R}(f)$ over all measurable functions is the conditional expectation:

and the minimum risk equals the irreducible noise:

Proof: From Section5.4, for any $f$:

And $\mathcal{R}(f^*) = \mathbb{E}[(Y - \mathbb{E}[Y|X])^2] = \mathbb{E}[\text{Var}(Y|X)]$ by definition. $\square$

The Bayes risk (irreducible error) $\varepsilon^* = \mathbb{E}[\text{Var}(Y|X)]$ cannot be reduced by any model, no matter how complex. It is the noise inherent in the prediction problem.

For regression: With $Y = f^*(X) + \varepsilon$ where $\varepsilon$ is independent noise with variance $\sigma^2$, the Bayes risk is $\sigma^2$.

For AI: The irreducible error is why perfect test accuracy is impossible on noisy datasets. In language modelling, even a perfect model (which exactly learns the true distribution) will have non-zero cross-entropy loss equal to the entropy of the true distribution $H(p^*)$. GPT-4's perplexity on human text is bounded below by the entropy of human language.

8.2 Bias-Variance-Noise Decomposition

In practice, we learn a model $\hat{f}$ from a finite dataset $\mathcal{D}$ of $N$ samples. The model $\hat{f}$ is a random function - it depends on the random training data. We can decompose the expected prediction error at a new test point $x_0$:

Proof. Let $\bar{f} = \mathbb{E}_{\mathcal{D}}[\hat{f}(x_0)]$ (expected prediction) and $y_0 = f^*(x_0) + \varepsilon$ (noisy observation):

The cross term is $2\mathbb{E}[(f^*+\varepsilon-\bar{f})(\bar{f}-\hat{f})] = 2(f^*-\bar{f})\mathbb{E}[\bar{f}-\hat{f}] + 2\mathbb{E}[\varepsilon]\mathbb{E}[\bar{f}-\hat{f}] = 0$ (since $\mathbb{E}[\hat{f}] = \bar{f}$ and $\mathbb{E}[\varepsilon]=0$). Then:

And $\mathbb{E}[(\bar{f}-\hat{f})^2] = \text{Var}(\hat{f}(x_0))$. Combining: Bias^2 + Variance + Noise. $\square$

BIAS-VARIANCE TRADEOFF
========================================================================

  Error                        Total Error
   ^                           /
   |              Variance    /
   |         _______________/
   |        /
   |       /       Bias^2
   |      /  _______________
   |_____/__/________________
   |         Noise
   +-------------------------------- Model Complexity ->

  Classical view: sweet spot minimises total error

========================================================================

Interpretations:

• High bias: The model is too simple to capture $f^*$ (underfitting). Consistent but wrong. Example: linear model for a quadratic $f^*$.
• High variance: The model is too complex - it fits training noise (overfitting). Right on average but variable. Example: high-degree polynomial on limited data.
• Noise: Irreducible. The best we can do.

8.3 Double Descent and Modern Deep Learning

The classical bias-variance picture predicts a unique optimal complexity. Modern deep learning violates this: neural networks trained to zero training loss (interpolating the training data, which classical theory predicts should overfit catastrophically) often generalise well. This is the double descent phenomenon.

DOUBLE DESCENT
========================================================================

  Test Error
   ^
   |        Classical regime        Modern regime
   |              |                      |
   |    +----+    |                      |
   |   ++    ++   |                      |
   |  ++      ++  |                 +----+
   | ++        +--+-----------------+
   |               | Interpolation threshold
   +---------------------------------------- Model Size ->
           ^                   ^
    Underfitting         Overparameterised
    regime               regime (good!)

========================================================================

Why does this happen? In overparameterised models: - Many solutions interpolate the training data. - Gradient descent (especially with weight decay or implicit regularisation from SGD noise) selects the minimum-norm solution among all interpolating solutions. - The minimum-norm interpolant has low variance among interpolants - it is the "flattest" fit.

The bias-variance decomposition still holds in double descent, but the variance first rises (at the interpolation threshold, the model just barely interpolates - highly sensitive to training data) and then falls again (overparameterised models have many interpolating solutions, most with low variance).

For AI (2026 context): Large language models like GPT-4, Claude, and Gemini are massively overparameterised. They interpolate (or nearly interpolate) their training data and yet generalise remarkably well - this is the double descent regime. Understanding why overparameterisation helps generalisation remains one of the central open questions in deep learning theory.

8.4 Regularisation as Bias Injection

L2 regularisation (Ridge regression). The regularised estimator minimises:

For linear regression with design matrix $X$ (overloading notation): $\hat{\theta}_\lambda = (X^\top X + \lambda I)^{-1}X^\top y$.

Bias-variance analysis of Ridge: - Bias: $\hat{\theta}_\lambda$ is a biased estimator of $\theta^*$. Specifically, $\mathbb{E}[\hat{\theta}_\lambda] = (I + \lambda(X^\top X)^{-1})^{-1}\theta^* \neq \theta^*$. Ridge shrinks estimates toward 0, introducing bias. - Variance: $\text{Var}(\hat{\theta}_\lambda) \leq \text{Var}(\hat{\theta}_{\text{OLS}})$ for any $\lambda > 0$. The regularised estimator has smaller variance - it is more stable with respect to perturbations in the training data. - Trade-off: As $\lambda$ increases, bias increases and variance decreases. The optimal $\lambda$ minimises total risk.

For AI: Weight decay in neural network training (the most common form of regularisation) adds $\lambda \|\theta\|^2$ to the loss, equivalent to L2 regularisation. AdamW implements weight decay correctly by decoupling it from the adaptive learning rate. Understanding the bias-variance trade-off tells us why weight decay helps: it reduces model variance (overfitting) at the cost of some bias.

9. Expectation in ML: Core Applications

9.1 Cross-Entropy Loss as an Expectation

For a $K$-class classification problem, the cross-entropy loss for a single example $(x, y)$ with true label $y \in \{1,\ldots,K\}$ and model probabilities $\hat{p} = (\hat{p}_1,\ldots,\hat{p}_K)$ is:

The expected cross-entropy over the data distribution $p(x,y)$:

This equals the cross-entropy $H(p, q_\theta) = -\sum_k p_k \log q_k$ between the true label distribution and the model's predicted distribution, averaged over inputs $X$.

Connection to KL divergence. For each input $x$:

where $H(p) = -\mathbb{E}_p[\log p]$ is the entropy of the true conditional. Since $H(p(\cdot|x))$ doesn't depend on $\theta$, minimising cross-entropy is equivalent to minimising KL divergence - i.e., making the model distribution as close as possible to the true distribution, in the KL sense.

For language models. The cross-entropy loss on a token sequence $(w_1,\ldots,w_T)$ is:

This is the expected negative log-probability per token - an expectation under the empirical distribution of the training corpus. Minimising this drives the model distribution toward the empirical distribution of human text. The perplexity is $\exp(\mathcal{L})$: a model with perplexity 50 is "equally surprised" by text as a uniform distribution over 50 tokens.

9.2 ELBO as an Expected Value

The Variational Autoencoder (VAE) introduces an approximate posterior $q_\phi(z|x)$ (the encoder) to approximate the intractable true posterior $p_\theta(z|x)$. The Evidence Lower BOund (ELBO) is:

Reconstruction term: $\mathbb{E}_{q_\phi}[\log p_\theta(x|z)]$ - the expected log-likelihood of the observed data given a latent code, averaged over the approximate posterior. This rewards the decoder for reconstructing $x$ well from samples of $z \sim q_\phi(z|x)$.

KL regularisation term: $\text{KL}(q_\phi \| p(z))$ - penalises the approximate posterior for straying from the prior $p(z) = \mathcal{N}(0,I)$. For Gaussian encoder $q_\phi = \mathcal{N}(\mu_\phi(x), \text{diag}(\sigma_\phi^2(x)))$:

Gradient computation. The reconstruction term $\mathbb{E}_{q_\phi}[\log p_\theta(x|z)]$ cannot be differentiated through the sampling operation $z \sim q_\phi$ directly. The reparameterisation trick (Section03) writes $z = \mu_\phi + \sigma_\phi \odot \varepsilon$ with $\varepsilon \sim \mathcal{N}(0,I)$, enabling:

This gradient is computable by backpropagation through the deterministic function $z = \mu_\phi + \sigma_\phi \odot \varepsilon$.

9.3 Policy Gradient (REINFORCE)

In reinforcement learning, an agent with policy $\pi_\theta(a|s)$ (parameterised by $\theta$) seeks to maximise the expected return:

where $\tau = (s_0, a_0, r_0, s_1, a_1, \ldots)$ is a trajectory. The REINFORCE gradient is:

Derivation via log-derivative trick (score function):

Since $\log p_\theta(\tau) = \sum_t \log \pi_\theta(a_t|s_t) + \text{const}$ (dynamics don't depend on $\theta$), we get the REINFORCE formula.

The score function $\nabla_\theta \log \pi_\theta(a|s)$ appears because we differentiated the log of the probability. The key identity underlying this:

(differentiate the normalisation condition $\int p_\theta = 1$ to get $\int \nabla_\theta p_\theta = \mathbf{0}$, then note $\nabla_\theta p_\theta = p_\theta \nabla_\theta \log p_\theta$).

For RLHF. Reinforcement learning from human feedback uses policy gradients to fine-tune language models. The policy is the LLM: $\pi_\theta(a|s)$ where $s$ is the context and $a$ is the generated token. The reward $R$ comes from a reward model trained on human preferences. REINFORCE and its variants (PPO, GRPO) compute gradients of $\mathbb{E}[R]$ with respect to the LLM parameters.

9.4 Adam Optimizer as Moment Tracking

The Adam optimizer (Kingma & Ba, 2014) maintains exponential moving averages of the first and second moments of the gradient:

where $g_t = \nabla_\theta \mathcal{L}(\theta_t)$ is the gradient at step $t$.

Bias correction. Since $m_0 = v_0 = 0$, the early estimates are biased toward zero. The debiased estimates are:

Why does this work? $\mathbb{E}[m_t] = (1-\beta_1^t) \cdot \mathbb{E}[g_t]$ (when gradient is stationary), so dividing by $(1-\beta_1^t)$ recovers an unbiased estimate of $\mathbb{E}[g_t]$.

Parameter update:

The update is the first moment (direction of average gradient) scaled by $1/\sqrt{\text{second moment}}$ - normalising by the root mean square of the gradient. This is adaptive learning rate: parameters with large gradient variance get smaller updates (conservative), parameters with small variance get larger updates (aggressive).

Connection to Fisher Information. The diagonal of the Fisher information matrix $\mathcal{I}(\theta)$ is $\text{Var}(g_t) \approx \mathbb{E}[g_t^2]$ (when gradients are zero-mean). Adam's $\hat{v}_t$ approximates the diagonal Fisher, making Adam an approximate natural gradient method.

AdamW. Standard Adam conflates weight decay and L2 regularisation (they are equivalent for SGD but not for adaptive methods). AdamW decouples them:

The last term is direct weight decay - not affected by the adaptive scaling. This is the current default for training large language models.

10. Common Mistakes

11. Exercises

Exercise 1 * - LOTUS: Expected Value of a Transformation

Let $X \sim \text{Exponential}(\lambda=2)$ with PDF $f(x) = 2e^{-2x}$ for $x \geq 0$.

(a) Compute $\mathbb{E}[X]$ directly from the definition.

(b) Using LOTUS, compute $\mathbb{E}[X^2]$.

(c) Compute $\text{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$.

(d) Let $Y = e^X$. Compute $\mathbb{E}[Y]$ using LOTUS. (Hint: $\int_0^\infty e^{x} \cdot 2e^{-2x}dx = \int_0^\infty 2e^{-x}dx$.) What is the domain of $t$ for which $\mathbb{E}[e^{tX}]$ is finite?

Exercise 2 * - Linearity Without Independence

Let $X \sim \text{Uniform}(-1, 1)$ and $Y = X^2$.

(a) Compute $\mathbb{E}[X]$, $\mathbb{E}[Y]$, and $\mathbb{E}[X+Y]$ and verify linearity.

(b) Compute $\text{Cov}(X, Y)$. What does this tell you about the linear relationship?

(c) Are $X$ and $Y$ independent? Justify rigorously.

(d) Compute $\text{Var}(X+Y)$ using the formula $\text{Var}(X) + 2\text{Cov}(X,Y) + \text{Var}(Y)$, and verify numerically.

Exercise 3 * - Moments of the Gaussian

For $X \sim \mathcal{N}(0, 1)$ (standard normal):

(a) Show $\mathbb{E}[X^{2k+1}] = 0$ for all $k \geq 0$ (odd moments vanish).

(b) Using the MGF $M(t) = e^{t^2/2}$, find $\mathbb{E}[X^2]$, $\mathbb{E}[X^4]$, and $\mathbb{E}[X^6]$ by differentiating.

(c) Verify the general formula $\mathbb{E}[X^{2k}] = (2k-1)!! = (2k-1)(2k-3)\cdots 3 \cdot 1$.

(d) For $X \sim \mathcal{N}(\mu, \sigma^2)$, compute the skewness $\gamma_1$ and excess kurtosis $\gamma_2$.

Exercise 4 ** - Tower Property in Action

A fair coin is flipped. If heads (prob 1/2), $X \sim \text{Poisson}(3)$. If tails (prob 1/2), $X \sim \text{Poisson}(7)$.

(a) Using the tower property, compute $\mathbb{E}[X]$ (let $C \in \{H,T\}$ be the coin flip).

(b) Using the law of total variance, compute $\text{Var}(X)$.

(c) Write down the marginal PMF of $X$ (it is a mixture of two Poissons). Verify $\mathbb{E}[X]$ directly.

(d) AI connection: Mixture-of-experts (MoE) models route tokens to different experts. Each expert has its own output distribution. The total output distribution is a mixture; the tower property computes its moments.

Exercise 5 ** - MGF and Moment Extraction

Let $X \sim \text{Gamma}(\alpha, \beta)$ with PDF $f(x) = \frac{\beta^\alpha}{\Gamma(\alpha)} x^{\alpha-1} e^{-\beta x}$ for $x > 0$.

(a) Derive the MGF: show $M_X(t) = \left(\frac{\beta}{\beta-t}\right)^\alpha$ for $t < \beta$.

(b) Use $M_X(t)$ to compute $\mathbb{E}[X]$ and $\text{Var}(X)$ by differentiating.

(c) Show that the sum of $n$ independent $\text{Gamma}(\alpha_i, \beta)$ variables (same rate $\beta$) follows $\text{Gamma}(\sum_i \alpha_i, \beta)$ using the MGF multiplicative property.

(d) The chi-squared distribution $\chi^2_k = \text{Gamma}(k/2, 1/2)$. What is $\mathbb{E}[\chi^2_k]$ and $\text{Var}(\chi^2_k)$?

Exercise 6 ** - Jensen and KL Divergence

(a) Prove that for any probability distribution $p$ over $\{1,\ldots,K\}$:

with equality iff $p$ is uniform. (Hint: Apply Jensen to $f(x) = -\log x$ or use the KL divergence to uniform.)

(b) Show that $\text{KL}(p \| q) \geq 0$ using Jensen's inequality applied to $-\log$.

(c) Compute $\text{KL}(\mathcal{N}(\mu_1, \sigma_1^2) \| \mathcal{N}(\mu_2, \sigma_2^2))$ analytically. Verify it equals zero when $\mu_1=\mu_2$, $\sigma_1=\sigma_2$.

(d) For the ELBO: show that $\log p_\theta(x) \geq \mathcal{L}(\phi,\theta;x)$ with gap $= \text{KL}(q_\phi(z|x) \| p_\theta(z|x))$.

Exercise 7 *** - Bias-Variance Decomposition

Consider estimating $\theta^* = 2$ from $N=10$ iid samples $X_1,\ldots,X_N \sim \mathcal{N}(\theta^*, 1)$.

(a) Show that the sample mean $\hat{\theta}_1 = \bar{X}_N$ is unbiased with variance $\sigma^2/N = 1/10$.

(b) Consider a regularised estimator $\hat{\theta}_\lambda = \frac{N}{N+\lambda}\bar{X}_N$. Compute its bias and variance as functions of $\lambda > 0$.

(c) The MSE of $\hat{\theta}_\lambda$ is Bias^2 + Variance. Find the $\lambda$ that minimises MSE. (Hint: it depends on $\theta^*$.)

(d) Simulate this: draw 10000 datasets of $N=10$ samples, compute $\hat{\theta}_\lambda$ for $\lambda \in \{0, 0.1, 0.5, 1, 2, 5\}$, plot bias^2, variance, and MSE vs $\lambda$.

Exercise 8 *** - Adam as Moment Estimation

(a) Show that the Adam first-moment estimate satisfies $\mathbb{E}[m_t] = (1-\beta_1^t)\mathbb{E}[g]$ when gradients are iid with mean $\mathbb{E}[g]$ (stationary gradient assumption). Therefore $\hat{m}_t = m_t/(1-\beta_1^t)$ is unbiased.

(b) Implement a minimal Adam optimizer in NumPy (no PyTorch) and apply it to minimise $f(\theta) = \theta^4 - 4\theta^2 + \theta$. Track the first and second moment estimates. Plot $m_t$, $v_t$, and $\theta_t$ over 200 steps.

(c) Show that for a quadratic loss $\mathcal{L}(\theta) = \frac{1}{2}(\theta-\theta^*)^2$, Adam converges to $\theta^*$ and the second moment $v_t \to (\theta-\theta^*)^2 = $ squared gradient. In this case, $\hat{m}_t/\sqrt{\hat{v}_t} \to \text{sign}(\theta-\theta^*)$: Adam reduces to steepest descent with unit step size.

(d) Connect to Fisher information: the Fisher information matrix for $p_\theta(x) = \mathcal{N}(\theta, I)$ has diagonal equal to 1 (the variance of the score). Natural gradient descent uses $\mathcal{I}^{-1}\nabla\mathcal{L}$. How does Adam approximate this for diagonal Fisher?

12. Why This Matters for AI (2026 Perspective)

13. Conceptual Bridge

Where We Came From

This section builds directly on the foundations laid in Section01-Section03. The probability spaces and random variable formalism of Section01 provides the objects - PDFs, PMFs, CDFs - over which we compute expectations. The named distributions of Section02 supply the examples whose moments we derive in Section6.2: the mean $1/\lambda$ and variance $1/\lambda^2$ of the exponential, the mean $np$ and variance $np(1-p)$ of the binomial, the mean $\mu$ and variance $\sigma^2$ that define the Gaussian. The joint distribution machinery of Section03 provides the tools for conditional expectation (Section5) and the covariance matrix (Section4): the tower property is essentially iterated marginalisation, and the conditional variance formula is the law of total variance derived via joint distributions.

Where We Are Going

Section Section05 (Concentration Inequalities) takes the moment-bound preview from Section7.5 much further. Markov's inequality (from the mean) and Chebyshev's inequality (from the variance) are the first two results, but Section05 develops exponentially sharper bounds: Hoeffding's inequality (bounded variables), Chernoff bounds (via MGFs), and McDiarmid's inequality (functions of independent variables). These bounds are the mathematical machinery of PAC-learning generalisation theory - they quantify how many training examples are needed to guarantee that the empirical risk is close to the true risk.

Section Section06 (Stochastic Processes) extends expectation to sequences of random variables indexed by time. The Law of Large Numbers proves rigorously that $\bar{X}_N \to \mathbb{E}[X]$ - the sample mean converges to the expectation. The Central Limit Theorem proves that the standardised sample mean converges in distribution to $\mathcal{N}(0,1)$ - the Gaussian emerges as the universal limit of sums, with proof via MGF or characteristic function techniques introduced here.

POSITION IN CHAPTER 6 CURRICULUM
========================================================================

  Section01 Probability Spaces          Section02 Distributions
  +------------------+            +------------------+
  | Kolmogorov axioms|            | Named PDFs/PMFs  |
  | Events, \\sigma-algebra|            | Parameters       |
  | CDF, PDF, PMF    |            | Relationships    |
  +--------+---------+            +--------+---------+
           |                               |
           +--------------+----------------+
                          v
              Section03 Joint Distributions
              +----------------------+
              | Marginals            |
              | Conditionals f(y|x) |
              | MVN, Bayes, Chain   |
              +----------+----------+
                         |
                         v
          +==============================+
          |  Section04 Expectation & Moments  |  <- YOU ARE HERE
          |  E[X], Var, Cov, MGF       |
          |  Jensen, C-S, LOTUS        |
          |  Bias-Variance, Adam       |
          +==============+=============+
                         |
              +----------+----------+
              v                     v
    Section05 Concentration         Section06 Stochastic
    Inequalities              Processes
    +-------------+           +-------------+
    | Markov      |           | LLN: Xbar->E[X]|
    | Chebyshev   |           | CLT: ->N(0,1)|
    | Hoeffding   |           | Gaussian    |
    | PAC bounds  |           | processes   |
    +-------------+           +-------------+
              |                     |
              +----------+----------+
                         v
              Section07 Markov Chains
              +----------------------+
              | Transition matrices  |
              | Steady state         |
              | MCMC (Bayes sampling)|
              +----------------------+

========================================================================

The conceptual arc through Section04 is: we began with probability as a framework for describing uncertainty (Section01), learned the vocabulary of named distributions (Section02), understood how to reason about multiple random variables jointly (Section03), and now in Section04 we have the tools to summarise distributions with numbers - expectations, variances, covariances, moments - and to bound the gap between reality and our estimates. Every subsequent section will use the expectation operator as a core tool: Section05 to prove tail bounds, Section06 to formalise convergence, Section07 to analyse Markov chain stationary distributions via matrix expectations.

Appendix A - Worked Examples: Expectation and Moments

A.1 Expected Value of the Geometric Distribution

The geometric distribution counts the number of trials until the first success in a sequence of iid Bernoulli($p$) trials.

PMF: $P(X=k) = (1-p)^{k-1}p$ for $k = 1, 2, 3, \ldots$

Expected value via the definition:

where $q = 1-p$. Using the identity $\sum_{k=1}^\infty k q^{k-1} = \frac{d}{dq}\sum_{k=0}^\infty q^k = \frac{d}{dq}\frac{1}{1-q} = \frac{1}{(1-q)^2} = \frac{1}{p^2}$:

Variance via the second moment: $\mathbb{E}[X^2] = \sum_{k=1}^\infty k^2(1-p)^{k-1}p$. Using $\sum k^2 q^{k-1} = \frac{d}{dq}[\sum k q^k] = \frac{d}{dq}\frac{q}{(1-q)^2} = \frac{1+q}{(1-q)^3}$:

AI connection: The geometric distribution models the number of tokens until a specific token (e.g., end-of-sequence) appears, assuming iid generation. In practice tokens are not iid, but the geometric provides a baseline model. Expected sequence length under a uniform $p$ = stopping probability is $1/p$.

A.2 Method of Moments Estimation

The method of moments estimates distribution parameters by setting sample moments equal to theoretical moments and solving.

Example: Estimating $(\alpha, \beta)$ for $\text{Gamma}(\alpha, \beta)$.

Theoretical moments: $\mathbb{E}[X] = \alpha/\beta$ and $\text{Var}(X) = \alpha/\beta^2$.

Given $N$ samples $x_1, \ldots, x_N$, compute sample mean $\bar{x} = \frac{1}{N}\sum x_i$ and sample variance $s^2 = \frac{1}{N-1}\sum(x_i-\bar{x})^2$.

Set $\bar{x} = \hat{\alpha}/\hat{\beta}$ and $s^2 = \hat{\alpha}/\hat{\beta}^2$. Solving:

For AI: Many Bayesian models require estimating hyperparameters of prior distributions. Method of moments provides fast, closed-form initial estimates that can be refined by maximum likelihood or MCMC. It is also used in moment matching for knowledge distillation: a student model's distribution moments are matched to the teacher's.

A.3 St. Petersburg Paradox: When Expectation Misleads

The St. Petersburg game: flip a coin repeatedly until the first head. If head appears on flip $k$, win $2^k$ dollars.

Expected winnings:

The expected value is infinite! Yet no rational person would pay more than a few dollars to play this game.

Resolution: Rational agents maximise expected utility, not expected monetary value. For a logarithmic utility function $u(w) = \log(w)$, the expected utility is finite:

This is Bernoulli's resolution (1738): diminishing marginal utility. The paradox reveals that infinite expected value is not sufficient for rational choice - the existence of all moments (or at least bounded utility) is needed.

For AI: Reinforcement learning reward design must account for this. An agent with unbounded reward function may take extremely risky actions that have infinite expected reward but almost surely fail. This motivates bounded reward functions and regularisation in RLHF: keeping reward signals within a bounded range prevents policy collapse toward infinite-expectation strategies.

Appendix B - Proofs of Key Identities

B.1 Cauchy-Schwarz via Inner Product

The expectation $\mathbb{E}[XY]$ can be viewed as an inner product $\langle X, Y \rangle = \mathbb{E}[XY]$ in the $L^2$ space of square-integrable random variables. The Cauchy-Schwarz inequality for inner products states $|\langle X, Y \rangle|^2 \leq \langle X, X \rangle \langle Y, Y \rangle$, which gives $(\mathbb{E}[XY])^2 \leq \mathbb{E}[X^2]\mathbb{E}[Y^2]$ directly.

This inner product interpretation gives $L^2$ convergence its name: $X_n \to X$ in $L^2$ means $\mathbb{E}[(X_n - X)^2] \to 0$ - convergence in the $L^2$ inner product sense.

B.2 Variance as Second Cumulant

The CGF of $X$ is $K_X(t) = \log M_X(t) = \log \mathbb{E}[e^{tX}]$. Computing its second derivative at $t=0$:

At $t=0$: $M_X(0)=1$, $M_X'(0)=\mathbb{E}[X]$, $M_X''(0)=\mathbb{E}[X^2]$, so:

The variance is precisely the second cumulant $\kappa_2$.

B.3 Fisher Information and the Score

The Fisher information matrix is defined as:

The second equality uses the fact that $\mathbb{E}_\theta[\nabla_\theta \log p_\theta(X)] = \mathbf{0}$ (proved by differentiating $\int p_\theta = 1$):

Therefore the Fisher information is the variance (covariance matrix) of the score function $\nabla_\theta \log p_\theta(X)$.

For AI: Fisher information appears in: - Cramer-Rao bound: $\text{Var}(\hat{\theta}) \geq 1/\mathcal{I}(\theta)$ - the variance of any unbiased estimator is at least the reciprocal Fisher information. - Natural gradient: The natural gradient descent update $\mathcal{I}(\theta)^{-1}\nabla\mathcal{L}$ moves in the direction of steepest descent in the space of distributions (KL-divergence geometry), rather than parameter space. Adam approximates this with a diagonal Fisher. - Elastic Weight Consolidation (EWC): Used in continual learning to prevent catastrophic forgetting. The Fisher information diagonal identifies which parameters are important for previous tasks.

B.4 Stein's Lemma

Lemma (Stein, 1972). If $X \sim \mathcal{N}(\mu, \sigma^2)$ and $g$ is differentiable with $\mathbb{E}[|g'(X)|] < \infty$:

Proof: Integration by parts:

Note $(x-\mu)e^{-(x-\mu)^2/(2\sigma^2)} = -\sigma^2 \frac{d}{dx}e^{-(x-\mu)^2/(2\sigma^2)}$. Integrating by parts:

Special cases: - $g(x) = x$: $\mathbb{E}[X(X-\mu)] = \sigma^2 \implies \mathbb{E}[X^2] = \mu^2 + \sigma^2$ [ok] - $g(x) = x^2$: $\mathbb{E}[X^2(X-\mu)] = 2\sigma^2 \mathbb{E}[X] = 2\sigma^2\mu \implies \mathbb{E}[X^3] = \mu^3 + 3\mu\sigma^2$ [ok]

For AI: Stein's lemma is the foundation of Stein's identity used in score matching and denoising diffusion models. The score function $\nabla_x \log p(x)$ satisfies $\mathbb{E}_p[g(X) + \nabla_x \cdot g(X)] = 0$ (Stein's operator), enabling training by minimising a quadratic loss without computing the intractable normalisation constant of $p$.

Appendix C - Moment Computations for Common Distributions

This table gives raw moments, central moments, MGF, skewness, and excess kurtosis for the distributions used throughout the course.

Notes: - Student-$t$ has no MGF (heavy tails cause $\mathbb{E}[e^{tX}] = \infty$ for all $t \neq 0$). - Beta distribution skewness is zero iff $\alpha = \beta$ (symmetric); negative when $\alpha > \beta$, positive when $\alpha < \beta$. - Poisson has equal mean and variance - a property used to test whether count data follows Poisson (overdispersion: $\text{Var} > \mathbb{E}$ means extra variability, e.g., negative binomial is better).

Appendix D - The Exponential Family and Moments

Many common distributions belong to the exponential family, which has a elegant connection between natural parameters and moments.

D.1 Exponential Family Form

A distribution belongs to the exponential family if its density can be written as:

where: - $\eta$ = natural parameter vector - $T(x)$ = sufficient statistic vector - $A(\eta)$ = log-partition function (log-normaliser) - $h(x)$ = base measure

D.2 Moments from the Log-Partition Function

Theorem. For an exponential family distribution:

Proof sketch. Differentiating $\int p(x;\eta)dx = 1$ with respect to $\eta$:

Therefore $\mathbb{E}[T(X)] = \nabla A(\eta)$. Differentiating again gives the covariance formula. $\square$

Examples:

For AI: The log-partition function $A(\eta)$ is the free energy of the exponential family. Its gradient gives the expected sufficient statistics (moments), its Hessian gives the Fisher information matrix. In variational inference, the ELBO is optimised over the natural parameters $\eta$ of the approximate posterior; the optimal $\eta$ satisfies $\nabla A(\eta) = \mathbb{E}_{p}[T(X)]$ (moment matching). This is the connection between maximum entropy, sufficient statistics, and the moments derived in Section6.

Appendix E - Convergence in Probability and L^2 Convergence

The sample mean $\bar{X}_N = \frac{1}{N}\sum_{i=1}^N X_i$ estimates $\mathbb{E}[X]$. In what sense does this estimate converge?

E.1 L^2 Convergence (Mean Square Convergence)

converges to in mean square () sense:

This is an immediate consequence of the variance formula for means (Section4.4). The sample mean's MSE decreases at rate $1/N$.

E.2 Weak Law of Large Numbers (Preview)

The Weak LLN (proved in Section06 using characteristic functions or Chebyshev's inequality) states that for iid $X_i$ with finite mean $\mu$:

i.e., $P(|\bar{X}_N - \mu| > \varepsilon) \to 0$ for any $\varepsilon > 0$.

Proof via Chebyshev (when $\text{Var}(X) < \infty$): $P(|\bar{X}_N - \mu| > \varepsilon) \leq \frac{\text{Var}(\bar{X}_N)}{\varepsilon^2} = \frac{\text{Var}(X)}{N\varepsilon^2} \to 0$.

This is a direct application of Chebyshev's inequality (from Section7.5 preview) to the sample mean. The LLN justifies: if we run a neural network many times on iid batches and average the loss estimates, the average converges to the true expected loss.

-> Full treatment of LLN and CLT: Section06 Stochastic Processes

Appendix F - Conditional Expectation as Projection

The conditional expectation $\mathbb{E}[Y|X]$ can be understood geometrically as an orthogonal projection in the Hilbert space $L^2(\Omega, P)$ of square-integrable random variables.

Inner product: $\langle X, Y \rangle = \mathbb{E}[XY]$

Projection: $\mathbb{E}[Y|\mathcal{G}]$ (conditional expectation on a sub-$\sigma$-algebra $\mathcal{G}$) is the projection of $Y$ onto the closed subspace of $\mathcal{G}$-measurable random variables.

Geometric interpretation: Among all $\mathcal{G}$-measurable (i.e., functions of $X$) approximations to $Y$, the conditional expectation $\mathbb{E}[Y|X]$ minimises the $L^2$ distance $\mathbb{E}[(Y-g(X))^2]$. This is the projection theorem: the best approximation is the projection, and the residual $Y - \mathbb{E}[Y|X]$ is orthogonal to every $\mathcal{G}$-measurable random variable:

Consequence: The tower property $\mathbb{E}[\mathbb{E}[Y|X]] = \mathbb{E}[Y]$ is the projection version of the law of total expectation. In a Hilbert space, projecting $Y$ onto a subspace and then taking the "total length" (expectation) equals the total length of $Y$.

For AI: This projection view clarifies why neural networks trained with MSE loss approximate $\mathbb{E}[Y|X]$: the network is learning the projection of $Y$ onto the subspace of functions representable by the architecture. Deeper networks can represent larger subspaces, hence better approximations to the conditional expectation.

Appendix G - Worked Problems: Moments and Inequalities

G.1 Computing the Moments of the Beta Distribution

The Beta($\alpha, \beta$) distribution has PDF $f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)}$ on $[0,1]$, where $B(\alpha,\beta) = \Gamma(\alpha)\Gamma(\beta)/\Gamma(\alpha+\beta)$.

Raw moment: Using the Beta function definition:

First moment: $\mathbb{E}[X] = \frac{\alpha}{\alpha+\beta}$.

Second moment: $\mathbb{E}[X^2] = \frac{\alpha(\alpha+1)}{(\alpha+\beta)(\alpha+\beta+1)}$.

Variance:

AI connection: The Beta distribution is the conjugate prior for the Bernoulli/Binomial likelihood. After observing $s$ successes and $f$ failures, the posterior is Beta($\alpha+s$, $\beta+f$). The posterior mean is $\frac{\alpha+s}{\alpha+\beta+s+f}$ - a weighted average of the prior mean $\frac{\alpha}{\alpha+\beta}$ and the sample mean $\frac{s}{s+f}$. As $s+f \to \infty$, the posterior converges to the sample mean (data dominates prior).

G.2 Proving $H(p) \leq \log K$ by Jensen

For a probability distribution $p = (p_1,\ldots,p_K)$ with $p_k > 0$ and $\sum_k p_k = 1$, the Shannon entropy is $H(p) = -\sum_k p_k \log p_k$.

Proof that $H(p) \leq \log K$:

Apply Jensen's inequality with the concave function $\log$:

Equality holds iff $1/p_k = $ const for all $k$ (Jensen with equality iff the argument is constant), i.e., $p_k = 1/K$ (uniform). $\square$

Alternative proof via KL divergence:

where $u = (1/K,\ldots,1/K)$ is uniform. KL non-negativity gives $H(p) \leq \log K$.

G.3 Adam Bias Correction: Full Derivation

At step $t$, the Adam first-moment accumulator is:

Unrolling from $m_0 = 0$:

Taking expectations (assuming iid gradients with constant mean $\mathbb{E}[g_i] = \mu_g$):

The bias is $\mathbb{E}[m_t] - \mu_g = -\beta_1^t \mu_g$, which decays to zero geometrically. The bias-corrected estimate $\hat{m}_t = m_t / (1-\beta_1^t)$ satisfies $\mathbb{E}[\hat{m}_t] = \mu_g$ (unbiased).

Similarly for $v_t$, and the debiased $\hat{v}_t = v_t/(1-\beta_2^t)$ estimates $\mathbb{E}[g_t^2]$ unbiasedly.

In early training ($t$ small), $1-\beta_1^t \approx (1-\beta_1)t$ is small, so $\hat{m}_t = m_t/((1-\beta_1)t)$ greatly amplifies $m_t$. Without bias correction, Adam would take tiny steps at the start because $m_t \approx (1-\beta_1)g_1$ after one step. With bias correction: $\hat{m}_1 = g_1$ - the first step uses the actual gradient.

G.4 Conditional Expectation: Gaussian Case

Let $(X,Y) \sim \mathcal{N}(\boldsymbol{\mu}, \Sigma)$ with $\boldsymbol{\mu} = (\mu_X, \mu_Y)$ and $\Sigma = $.

Conditional distribution (from Section03 Schur complement formula):

Therefore:

Verification via law of total variance:

The variance of the conditional mean contributes $\rho^2\sigma_Y^2$ (explained variance), and the within-group variance contributes $(1-\rho^2)\sigma_Y^2$ (unexplained variance). The fraction $\rho^2$ of $\text{Var}(Y)$ explained by $X$ is the coefficient of determination $R^2$ of linear regression.

Appendix H - Notation Summary

Appendix I - Quick Reference: Key Formulas

Expectation:

Variance:

Conditional expectation:

MGF:

Jensen's inequality:

Cauchy-Schwarz:

Bias-Variance:

Adam:

Appendix J - Common Mistakes: Extended Examples

J.1 Jensen's Direction: A Costly Error

One of the most frequent mistakes when applying Jensen's inequality is applying it in the wrong direction. The rule is simple: for convex $f$, $f(\mathbb{E}[X]) \leq \mathbb{E}[f(X)]$; for concave $f$, the inequality reverses.

Common confusion: $\sqrt{\cdot}$ vs. $(\cdot)^2$.

• $f(x) = x^2$ is convex ($f''=2>0$). Jensen: $(\mathbb{E}[X])^2 \leq \mathbb{E}[X^2]$. Equivalently, $\text{Var}(X) = \mathbb{E}[X^2]-(\mathbb{E}[X])^2 \geq 0$. [ok]
• $f(x) = \sqrt{x}$ is concave ($f''=-\frac{1}{4}x^{-3/2}<0$). Jensen: $\sqrt{\mathbb{E}[X]} \geq \mathbb{E}[\sqrt{X}]$ for $X \geq 0$.

A model that minimises $\mathbb{E}[\sqrt{\text{loss}}]$ is NOT the same as minimising $\sqrt{\mathbb{E}[\text{loss}]}$. The former cares about average square-root loss, the latter about the square root of average loss. In practice this distinction matters for robust loss functions.

Common confusion: $\log$ vs. $\exp$.

• $f(x) = \log x$ is concave -> $\log\mathbb{E}[X] \geq \mathbb{E}[\log X]$. This is the key inequality behind the ELBO derivation.
• $f(x) = e^x$ is convex -> $e^{\mathbb{E}[X]} \leq \mathbb{E}[e^X]$. This is the key inequality for bounding the MGF.

The ELBO derivation applies Jensen with $\log$ (concave): $\log \mathbb{E}[Z] \geq \mathbb{E}[\log Z]$ where $Z = p_\theta(x,z)/q_\phi(z|x)$. Students often try to apply it in the wrong direction, getting an upper bound instead of a lower bound.

J.2 The Tower Property Subtlety: Nested Conditioning

The tower property states $\mathbb{E}[\mathbb{E}[Y|X]] = \mathbb{E}[Y]$. A more general version for nested conditioning: for $\sigma$-algebras $\mathcal{G}_1 \subset \mathcal{G}_2$:

Conditioning on less information from already conditioned: you keep the coarser conditioning.

Example: Let $Z$ be a sufficient statistic for $\theta$ given data $X_1,\ldots,X_n$. Then:

where $\tilde{\theta} = \mathbb{E}[\hat{\theta}|Z]$ is the Rao-Blackwellised estimator. The outer expectation equals the original (tower property), but the Rao-Blackwellised estimator has smaller variance. This is NOT circular - it is the statement that smoothing $\hat{\theta}$ via conditioning on $Z$ doesn't change its mean.

J.3 Correlation Does Not Imply Causation - A Statistical View

Zero correlation means $\mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y]$ - no linear relationship. But:

1. There may be nonlinear dependence (Section4.3 example: $Y = X^2$).
2. Even positive correlation may arise from a common cause (confounding): if $Z \to X$ and $Z \to Y$ (fork structure from Section03), then $X$ and $Y$ are correlated even if neither causes the other.

In ML: the correlation between model predictions and labels on the test set measures linear predictive ability, not causal understanding. A model can achieve high correlation while exploiting spurious features (shortcuts). For example, language models correlate "hospital" with "disease" not through causal understanding but through co-occurrence patterns.

Conditional independence as the resolution: If $Z$ is the common cause (confounder), $X$ and $Y$ may be conditionally independent given $Z$: $\text{Cov}(X,Y|Z) = 0$ even though $\text{Cov}(X,Y) \neq 0$. Adjusting for confounders (either by conditioning or instrumental variables) is the statistical approach to causal inference.

Appendix K - Information-Theoretic View of Moments

K.1 Entropy as Negative Expected Log-Probability

The Shannon entropy of a discrete distribution $p$ is:

This is simply minus the expected log-probability. For a continuous distribution with PDF $f$, the differential entropy is:

Gaussian maximises differential entropy. Among all distributions on $\mathbb{R}$ with fixed mean $\mu$ and variance $\sigma^2$, the Gaussian $\mathcal{N}(\mu, \sigma^2)$ maximises differential entropy:

Proof sketch (via KL divergence): For any distribution $f$ with mean $\mu$ and variance $\sigma^2$, let $g = \mathcal{N}(\mu,\sigma^2)$.

Since $\log g(x) = -\frac{1}{2}\log(2\pi\sigma^2) - \frac{(x-\mu)^2}{2\sigma^2}$, and $\int f(x) \cdot \frac{(x-\mu)^2}{2\sigma^2}dx = \frac{\sigma^2}{2\sigma^2} = \frac{1}{2}$ (since $f$ has variance $\sigma^2$):

Therefore $0 \leq -h(f) - h(g)$, i.e., $h(f) \leq h(g)$. Equality iff $f = g$ (since $\text{KL}=0$ iff distributions are equal). $\square$

AI connection: This maximum entropy property explains why the Gaussian prior is so common in Bayesian machine learning: given a known mean and variance (from domain knowledge), the Gaussian is the least informative (maximum entropy) prior consistent with those constraints. It makes the fewest additional assumptions about the distribution.

K.2 Mutual Information as Expected KL Divergence

The mutual information between $X$ and $Y$ is:

This is the KL divergence between the joint distribution and the product of marginals. Since $\text{KL} \geq 0$: $I(X;Y) \geq 0$ with equality iff $X \perp Y$.

Relationship to conditional entropy:

where $H(Y|X) = \mathbb{E}_X[H(Y|X=x)]$ is the conditional entropy (tower property applied to entropy).

For AI: Mutual information is the gold standard for measuring statistical dependence. It captures all forms of dependence (linear and nonlinear), unlike correlation. Contrastive learning methods (SimCLR, CLIP) can be viewed as maximising a lower bound on $I(\text{view}_1; \text{view}_2)$ - encouraging representations to capture information shared between different views/modalities. Information bottleneck methods (used for analysing neural networks) study the trade-off between compressing $X$ (low $I(Z;X)$) and preserving information about $Y$ (high $I(Z;Y)$).

Appendix L - The Reparameterisation Trick: Full Mathematical Treatment

The reparameterisation trick is a technique for computing gradients of expectations when the distribution depends on the parameters we differentiate with respect to.

L.1 The Problem: Gradient Through Sampling

We want $\nabla_\phi \mathbb{E}_{z \sim q_\phi}[f(z)]$ where $q_\phi$ is a distribution parameterised by $\phi$. Naively:

This requires computing $\nabla_\phi q_\phi(z)$ for each $z$, and the expectation is now over the fixed distribution of $z$ values - but the integration measure $q_\phi(z)dz$ changes with $\phi$, making Monte Carlo estimation require evaluating $\nabla_\phi q_\phi$.

L.2 REINFORCE (Score Function Estimator)

The score function trick uses $\nabla_\phi q_\phi = q_\phi \nabla_\phi \log q_\phi$:

This is computable by Monte Carlo: sample $z \sim q_\phi$, compute $f(z)\nabla_\phi \log q_\phi(z)$, average over samples. However, this estimator has high variance because $f(z)$ can be large and $\nabla_\phi \log q_\phi(z)$ can vary greatly.

L.3 Reparameterisation

When $q_\phi$ is a location-scale family (or more generally, when there exists a deterministic transformation $g(\phi, \varepsilon)$):

For Gaussian: $z = \mu_\phi + \sigma_\phi \odot \varepsilon$, $\varepsilon \sim \mathcal{N}(0,I)$.

Then:

and:

The gradient now flows through the deterministic function $g$, enabling automatic differentiation. The variance of this estimator is typically much lower than REINFORCE because $\nabla_z f$ is often smoother than $f \cdot \nabla_\phi \log q_\phi$.

Jacobian of the transform:

So the gradient with respect to $\mu_\phi$ is $\mathbb{E}[\nabla_z f(z)]$ and with respect to $\sigma_\phi$ is $\mathbb{E}[\nabla_z f(z) \odot \varepsilon]$.

L.4 Why Lower Variance?

Consider $f(z) = \sin(z)$ with $z \sim \mathcal{N}(\mu, \sigma^2)$.

REINFORCE gradient: $\mathbb{E}[\sin(z) \cdot (z-\mu)/\sigma^2]$. The product $\sin(z)(z-\mu)$ can be large in magnitude.

Reparameterisation gradient: $\mathbb{E}[\cos(\mu+\sigma\varepsilon)] = \mathbb{E}[\cos(z)]$. The gradient is $\cos(z)$, bounded in $[-1,1]$, much more stable.

In general, reparameterisation produces gradients of $O(1)$ magnitude when $f$ is smooth, while REINFORCE gradients scale with $f$ values which can be large.

Appendix M - Moments in the Context of Score Matching and Diffusion

M.1 Denoising Score Matching

Diffusion models (DDPM, Score SDE) learn the score function $\nabla_x \log p_t(x)$ at each noise level $t$. The training objective is:

where $x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1-\bar{\alpha}_t}\varepsilon$ and $\varepsilon \sim \mathcal{N}(0,I)$.

The conditional score is:

This is an expectation over the noise schedule. The loss simplifies to:

which is an MSE loss - an empirical estimate of $\mathbb{E}[(Y-f(X))^2]$ where $Y$ is the added noise $\varepsilon$ and $f = \varepsilon_\theta$.

M.2 First Moment of the Reverse Process

The reverse diffusion process gives $p_\theta(x_{t-1}|x_t) = \mathcal{N}(\mu_\theta(x_t,t), \sigma_t^2 I)$ where:

The mean of the reverse step is determined by the predicted noise. The conditional expectation $\mathbb{E}[x_{t-1}|x_t]$ is the denoised estimate of $x_0$:

This is LOTUS applied to the reparameterised relationship: the expected clean image given the noisy image is a simple linear function of the predicted noise, evaluated using the noisy image.

Appendix N - Worked Problems: Bias-Variance and Regularisation

N.1 Ridge Regression Bias-Variance Tradeoff

Setup. Data: $y = X\theta^* + \varepsilon$ where $X \in \mathbb{R}^{n\times d}$, $\varepsilon \sim \mathcal{N}(0, \sigma^2 I)$.

Ridge estimator: $\hat{\theta}_\lambda = (X^\top X + \lambda I)^{-1}X^\top y$.

Let $A_\lambda = (X^\top X + \lambda I)^{-1}X^\top$.

Bias: $\text{Bias}(\hat{\theta}_\lambda) = \mathbb{E}[\hat{\theta}_\lambda] - \theta^* = A_\lambda X\theta^* - \theta^* = (A_\lambda X - I)\theta^*$.

Since $A_\lambda X = (X^\top X + \lambda I)^{-1}X^\top X$:

So $\text{Bias}(\hat{\theta}_\lambda) = -\lambda(X^\top X + \lambda I)^{-1}\theta^*$. Bias increases with $\lambda$.

Variance: $\text{Var}(\hat{\theta}_\lambda) = A_\lambda \text{Var}(y) A_\lambda^\top = \sigma^2 A_\lambda A_\lambda^\top = \sigma^2(X^\top X+\lambda I)^{-1}X^\top X(X^\top X+\lambda I)^{-1}$.

As $\lambda \to \infty$: $(X^\top X+\lambda I)^{-1} \to 0$, so variance $\to 0$. As $\lambda \to 0$: recovers OLS variance $\sigma^2(X^\top X)^{-1}$.

Total MSE: $\text{MSE} = \|\text{Bias}\|^2 + \text{tr}(\text{Var}) = \lambda^2 \theta^{*\top}(X^\top X+\lambda I)^{-2}\theta^* + \sigma^2\text{tr}((X^\top X+\lambda I)^{-1}X^\top X(X^\top X+\lambda I)^{-1})$.

The optimal $\lambda$ minimises this sum - bias grows, variance shrinks, and there is a sweet spot.

N.2 Neural Network Initialisation via Variance Propagation

He initialisation (for ReLU networks) is derived from the bias-variance / variance propagation analysis.

For a layer $h_j^{[l]} = \text{ReLU}\!\left(\sum_{i} W_{ji}^{[l]} h_i^{[l-1]}\right)$, with iid inputs $h_i^{[l-1]} \sim (0, v_{l-1})$ (zero mean, variance $v_{l-1}$) and iid weights $W_{ji} \sim (0, \sigma_W^2)$ independent of inputs:

Variance propagation:

where $z \sim \mathcal{N}(0, v_{l-1})$. For ReLU: $\mathbb{E}[\text{ReLU}(z)^2] = \text{Var}(z)/2 = v_{l-1}/2$ (since ReLU zeros half the distribution).

Therefore $v_l = n_{l-1} \cdot \sigma_W^2 \cdot v_{l-1}/2$.

For variance to stay constant across layers: $v_l = v_{l-1}$ requires $\sigma_W^2 = 2/n_{l-1}$.

He initialisation: $W \sim \mathcal{N}(0, 2/n_\text{in})$. This preserves variance through the network during the forward pass, preventing gradients from vanishing or exploding in deep networks.

Appendix O - Self-Assessment Checklist

Use this checklist after studying the section to identify gaps before proceeding to Section05.

Core Mechanics (Should be fluent)

• [ ] I can compute $\mathbb{E}[X]$ from a PMF or PDF using the definition.
• [ ] I can apply LOTUS: $\mathbb{E}[g(X)] = \int g(x)f(x)dx$ without finding the distribution of $g(X)$.
• [ ] I know that linearity $\mathbb{E}[aX+bY]=a\mathbb{E}[X]+b\mathbb{E}[Y]$ holds without independence.
• [ ] I can compute $\text{Var}(X)$ using $\mathbb{E}[X^2]-(\mathbb{E}[X])^2$.
• [ ] I know $\text{Var}(aX+b)=a^2\text{Var}(X)$ (shift doesn't change variance).
• [ ] I can compute skewness $\gamma_1=\mu_3/\sigma^3$ and kurtosis $\gamma_2=\mu_4/\sigma^4-3$.

Intermediate Theory (Should understand proofs)

• [ ] I can state and prove the tower property $\mathbb{E}[\mathbb{E}[Y|X]]=\mathbb{E}[Y]$.
• [ ] I can state and prove the law of total variance.
• [ ] I can derive the MGF of the Gaussian, Exponential, and Poisson distributions.
• [ ] I can state Jensen's inequality and identify when it applies ($f$ convex vs. concave).
• [ ] I can prove $\text{KL}(p\|q)\geq 0$ using Jensen.
• [ ] I can state and prove the Cauchy-Schwarz inequality for expectations.
• [ ] I know that $|\rho|\leq 1$ follows from Cauchy-Schwarz.
• [ ] I understand why zero covariance does NOT imply independence (with a counterexample).

Advanced Applications (Should be able to apply)

• [ ] I can derive the bias-variance decomposition from first principles.
• [ ] I can explain what double descent is and why overparameterised models can generalise.
• [ ] I can derive the ELBO using Jensen's inequality.
• [ ] I can explain the reparameterisation trick and why it reduces gradient variance.
• [ ] I understand Adam as tracking first and second moments with bias correction.
• [ ] I can explain why the score function $\mathbb{E}[\nabla_\theta \log p_\theta(X)]=\mathbf{0}$.

Appendix P - Further Reading and References

Textbooks

1. Probability and Statistics for Engineering and the Sciences - Jay Devore (2015). Accessible introduction to moments, MGFs, and expectation with engineering examples.

2. Probability Theory: The Logic of Science - E.T. Jaynes (2003). Philosophical and technical treatment; excellent on entropy and maximum entropy principle.

3. Pattern Recognition and Machine Learning - Christopher Bishop (2006), Ch. 1-2. The ML-focused treatment of expectations, KL divergence, ELBO, and variational inference.

4. Deep Learning - Goodfellow, Bengio, Courville (2016), Ch. 3. Standard reference for probability in ML context including bias-variance tradeoff.

5. Probabilistic Machine Learning: An Introduction - Kevin Murphy (2022), Ch. 2-4. Modern treatment with extensive ML applications including Adam, VAEs, and diffusion models.

Papers

1. Kingma & Welling (2014). Auto-Encoding Variational Bayes. arXiv:1312.6114. Original VAE paper; derives ELBO via Jensen and introduces reparameterisation trick.

2. Kingma & Ba (2015). Adam: A Method for Stochastic Optimization. arXiv:1412.6980. Original Adam paper; moment tracking interpretation is explicit in the derivation.

3. Williams (1992). Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning. REINFORCE algorithm; score function gradient estimator.

4. Ioffe & Szegedy (2015). Batch Normalization. arXiv:1502.03167. Moment normalisation of activations; analysis of how variance affects training dynamics.

5. Belkin et al. (2019). Reconciling Modern Machine-Learning Practice and the Classical Bias-Variance Trade-Off. PNAS. Double descent phenomenon; formal analysis of why interpolating models can generalise.

6. Song et al. (2020). Score-Based Generative Modeling through Stochastic Differential Equations. arXiv:2011.13456. Diffusion models as score-matching; score function is a moment of the distribution.

Summary of Key Results

This section established the expectation operator as the central tool in probability and machine learning. Starting from the LOTUS definition, we derived linearity (which holds without independence), the tower property (iterated expectation), and the law of total variance (which decomposes uncertainty into within-group and between-group components).

The moment hierarchy captures distribution shape: the first moment (mean) locates it; the second (variance) measures spread; the third (skewness) captures asymmetry; the fourth (kurtosis) captures tail weight. Moment generating functions encode all moments in a single analytic function and enable elegant proofs of the reproductive property (sum of independent Gaussians is Gaussian) and cumulant additivity.

Jensen's inequality - the most important inequality in this section - gives the KL divergence its non-negativity, the ELBO its existence as a lower bound, and the bias-variance decomposition its clean structure. Cauchy-Schwarz gives $|\rho| \leq 1$ and motivates the attention scaling factor $1/\sqrt{d_k}$.

Every major ML method reviewed in Section9 is an application of these results: cross-entropy is expected log-loss, the ELBO is Jensen applied to the marginal likelihood, policy gradient is the score function trick, and Adam is first-and-second-moment tracking with debiasing.

The concepts forward-referenced here - Markov's and Chebyshev's inequalities, the Law of Large Numbers, the Central Limit Theorem - will be fully developed in Section05 and Section06, completing the probabilistic toolkit required for modern ML.

The conceptual unification: expectation is a linear functional on the space of random variables. It maps random variables to real numbers while preserving linear structure. Every computation in this section - variance as $\mathbb{E}[(X-\mu)^2]$, covariance as $\mathbb{E}[(X-\mu_X)(Y-\mu_Y)]$, the MGF as $\mathbb{E}[e^{tX}]$, the characteristic function as $\mathbb{E}[e^{itX}]$, the KL divergence as $\mathbb{E}_p[\log p/q]$, Adam's first moment as $\mathbb{E}[g_t]$ - is an application of this single linear functional applied to different functions of the random variable.

Understanding this unity transforms the apparent complexity of ML training into a coherent framework: we are always estimating expectations from samples, bounding how far sample estimates stray from true expectations, and choosing parameterisations that make those expectations tractable to compute and differentiate.

End of Section06/04 - Expectation and Moments