Lesson overview | Previous part | Lesson overview

Estimation Theory: Appendix M: Glossary of Key Terms to Appendix P: Practice Problems

Appendix M: Glossary of Key Terms

---

Appendix N: Further Reading and References

Primary References

1. Casella, G. & Berger, R.L. (2002). Statistical Inference (2nd ed.). Duxbury. - The definitive graduate textbook on classical estimation theory; covers all topics in this section at full rigor.

2. Lehmann, E.L. & Casella, G. (1998). Theory of Point Estimation (2nd ed.). Springer. - Advanced treatment of UMVUE theory, Rao-Blackwell, and sufficiency.

3. Fisher, R.A. (1922). "On the Mathematical Foundations of Theoretical Statistics." Philosophical Transactions of the Royal Society A, 222, 309-368. - The foundational paper defining sufficiency, efficiency, consistency, and MLE.

4. Cramer, H. (1946). Mathematical Methods of Statistics. Princeton University Press. - Original proof of the CRB.

5. Rao, C.R. (1945). "Information and the Accuracy Attainable in the Estimation of Statistical Parameters." Bulletin of the Calcutta Mathematical Society, 37, 81-91. - Independent CRB proof and Rao-Blackwell theorem.

6. Efron, B. (1979). "Bootstrap Methods: Another Look at the Jackknife." Annals of Statistics, 7(1), 1-26. - Original bootstrap paper.

ML-Specific References

1. Amari, S. (1998). "Natural Gradient Works Efficiently in Learning." Neural Computation, 10(2), 251-276. - Foundation of natural gradient methods.

2. Martens, J. & Grosse, R. (2015). "Optimizing Neural Networks with Kronecker-factored Approximate Curvature." ICML. - K-FAC for tractable FIM approximation in deep networks.

3. Kirkpatrick, J. et al. (2017). "Overcoming Catastrophic Forgetting in Neural Networks." PNAS, 114(13), 3521-3526. - EWC using FIM for continual learning.

4. Hoffmann, J. et al. (2022). "Training Compute-Optimal Large Language Models." NeurIPS. - Chinchilla scaling laws via nonlinear MLE.

5. Hu, E. et al. (2022). "LoRA: Low-Rank Adaptation of Large Language Models." ICLR. - Low-rank MLE for efficient fine-tuning.

6. Guo, C. et al. (2017). "On Calibration of Modern Neural Networks." ICML. - Temperature scaling as MLE on calibration set.

7. Goodfellow, I., Bengio, Y. & Courville, A. (2016). Deep Learning. MIT Press. - Chapter 5 covers MLE in the context of deep learning at the appropriate depth for ML practitioners.

Advanced References

1. van der Vaart, A.W. (1998). Asymptotic Statistics. Cambridge. - Graduate-level asymptotic theory; proofs of asymptotic normality, delta method, semiparametric theory.

2. Wasserman, L. (2004). All of Statistics. Springer. - Excellent graduate-level survey connecting classical statistics to modern methods.

3. Bishop, C.M. (2006). Pattern Recognition and Machine Learning. Springer. - Chapters 1-3 cover MLE, MAP, and Bayesian estimation with ML motivation.

---

This section is part of the Math for LLMs curriculum. For corrections or contributions, see CONTRIBUTING.md.

---

Appendix O: Connections to Information Theory

The Fisher information matrix has a deep connection to Shannon information theory, previewing material from Chapter 9 - Information Theory.

O.1 Fisher Information as Local Curvature of KL Divergence

The KL divergence between distributions at nearby parameter values satisfies:

$$D_{\mathrm{KL}}(p(\cdot;\boldsymbol{\theta}) \| p(\cdot;\boldsymbol{\theta}+\mathrm{d}\boldsymbol{\theta})) = \frac{1}{2}\mathrm{d}\boldsymbol{\theta}^\top \mathcal{I}(\boldsymbol{\theta})\, \mathrm{d}\boldsymbol{\theta} + O(\|\mathrm{d}\boldsymbol{\theta}\|^3)$$

The Fisher information matrix is the Hessian of the KL divergence with respect to the second argument, evaluated at the point where both arguments agree. This identifies $\mathcal{I}$ as the local curvature of the divergence surface.

Consequence: Moving in the Fisher metric direction by $\mathrm{d}\boldsymbol{\theta}$ produces the maximal increase in KL divergence from the current distribution - this is why the natural gradient (moving $\mathcal{I}^{-1}\nabla\ell$ in parameter space) corresponds to moving in the direction of maximal KL divergence gain in distribution space.

O.2 Cramer-Rao and Channel Capacity

Shannon's channel capacity theorem and the Cramer-Rao bound are related through the van Trees inequality (a Bayesian version of the CRB for random $\theta$):

$$\operatorname{MSE}(\hat{\theta}) \geq \frac{1}{n\mathcal{I}(\theta) + \mathcal{I}_{\text{prior}}(\theta)}$$

where $\mathcal{I}_{\text{prior}}$ is the Fisher information of the prior. As the prior becomes uninformative ($\mathcal{I}_{\text{prior}} \to 0$), this recovers the standard CRB. The connection: information accumulates (Fisher information adds) across observations and from the prior, exactly as Shannon information adds in a noisy channel.

O.3 Sufficient Statistics and Data Compression

A sufficient statistic $T$ satisfies the data processing inequality: processing data through $T$ cannot increase Fisher information. In fact, for a sufficient statistic, $\mathcal{I}_T(\theta) = \mathcal{I}_{\mathbf{X}}(\theta)$ - no information is lost. This is the estimation-theory analogue of lossless compression: a sufficient statistic compresses the data without losing any information about $\theta$.

Minimal sufficient statistics provide the maximum compression while retaining all information - analogous to the minimum description length (MDL) principle in information theory.

---

Appendix P: Practice Problems

P.1 Identification Problems

For each of the following, state whether the model is identifiable. If not, identify the unidentifiable combination.

1. $X \sim \mathcal{N}(\mu_1 - \mu_2, 1)$ with parameters $(\mu_1, \mu_2) \in \mathbb{R}^2$.
2. $p(x|\theta) = \theta e^{-\theta x}$ for $x > 0$, with $\theta > 0$. (Identifiable?)
3. A two-layer ReLU network $f(x) = \text{ReLU}(w_1 x) + \text{ReLU}(w_2 x)$ with parameters $(w_1, w_2)$. Is the function class identifiable?

P.2 Computational Problems

1. MLE and invariance. For Poisson data with $n = 20$ and $\sum x_i = 48$: (a) find $\hat{\lambda}$; (b) find the MLE of $e^{-\lambda}$ (the probability of observing zero events); (c) find the MLE of $1/\lambda$ (mean inter-arrival time).

2. CRB for a transformed parameter. For $X \sim \mathcal{N}(\mu, 1)$ with $n$ observations, derive the CRB for estimating $g(\mu) = \mu^2$ using the biased-estimator form of the CRB. Show that the MLE $\hat{\mu}^2 = \bar{x}^2$ does not achieve this bound for finite $n$, but does so asymptotically.

3. Bootstrap vs. asymptotic. For $n = 20$ observations from $\mathcal{N}(\mu, 1)$ with $\bar{x} = 3.2$: (a) compute the exact 95% $t$-CI; (b) simulate the bootstrap CI with $B = 1000$; (c) compare coverage by repeating 1000 times and computing the empirical coverage of each CI type.

P.3 Conceptual Problems

1. Stein paradox. For $d = 1$, the sample mean $\bar{x}$ is the admissible MLE. For $d = 3$, the James-Stein estimator $\tilde{\boldsymbol{\mu}} = (1 - \frac{d-2}{n\lVert\bar{\mathbf{x}}\rVert_2^2})\bar{\mathbf{x}}$ dominates $\bar{\mathbf{x}}$ in MSE. Simulate this for $d = 3$, $n = 1$, true $\boldsymbol{\mu} = (2,2,2)$, $\sigma = 1$, and verify $\text{MSE}(\tilde{\boldsymbol{\mu}}) < \text{MSE}(\bar{\mathbf{x}})$.

2. Model misspecification. A logistic regression model is fitted to data from a probit model ($p(y=1|\mathbf{x}) = \Phi(\mathbf{w}^\top\mathbf{x})$). The MLE converges to what? Explain using the misspecified MLE theorem and the KL divergence interpretation.

3. FIM for temperature scaling. For a $K$-class classifier with logits $\mathbf{z}$ and temperature $T$, the softmax is $p_k = e^{z_k/T}/\sum_j e^{z_j/T}$. Derive the Fisher information $\mathcal{I}(T)$ for a single example and explain why Newton-Raphson temperature calibration converges in 1-2 steps.

---

Section Section02 Estimation Theory - Chapter 7 Statistics - Math for LLMs curriculum

Lines: ~2000 | Theory notebook: 50+ cells | Exercises: 8 graded problems