Chapter 8 - Optimization

"Training a model is optimization under uncertainty, geometry, and compute constraints."

Overview

Optimization is the mathematical engine of machine learning. Every trained model is the result of an algorithm moving through parameter space to reduce an objective while dealing with curvature, noise, constraints, regularization, schedules, and finite compute.

This rewrite treats the chapter as a single arc: convex guarantees first, deterministic first-order methods second, curvature third, constraints fourth, stochastic scale fifth, nonconvex landscapes sixth, adaptive optimizer state seventh, regularization eighth, hyperparameter search ninth, and schedule design tenth.

Subsection Map

#	Subsection	What It Covers	Canonical Topics
## Reading Order and Dependencies

01-Convex-Optimization               (Convex Optimization)
        ↓
02-Gradient-Descent                  (Gradient Descent)
        ↓
03-Second-Order-Methods              (Second-Order Methods)
        ↓
04-Constrained-Optimization          (Constrained Optimization)
        ↓
05-Stochastic-Optimization           (Stochastic Optimization)
        ↓
06-Optimization-Landscape            (Optimization Landscape)
        ↓
07-Adaptive-Learning-Rate            (Adaptive Learning Rate)
        ↓
08-Regularization-Methods            (Regularization Methods)
        ↓
09-Hyperparameter-Optimization       (Hyperparameter Optimization)
        ↓
10-Learning-Rate-Schedules           (Learning Rate Schedules)
        ↓
Chapter 9 - Information Theory

What Belongs Where - Canonical Homes

Topic Family	Canonical Home	Preview Only In
convex sets, convex combinations, convex functions, Jensen inequality, first-order characterization	Convex Optimization	Neighboring sections use brief references only
gradient direction, descent lemma, constant step size, exact line search, backtracking line search	Gradient Descent	Neighboring sections use brief references only
Hessian matrix, quadratic model, Newton step, Newton decrement, damped Newton	Second-Order Methods	Neighboring sections use brief references only
feasible set, active constraint, equality constraints, inequality constraints, Lagrangian	Constrained Optimization	Neighboring sections use brief references only
stochastic objective, empirical risk, population risk, unbiased gradient oracle, gradient variance	Stochastic Optimization	Neighboring sections use brief references only
critical point, local minimum, saddle point, strict saddle, plateau	Optimization Landscape	Neighboring sections use brief references only
effective learning rate, diagonal preconditioner, AdaGrad accumulator, RMSProp exponential averaging, Adam first moment	Adaptive Learning Rate	Neighboring sections use brief references only
explicit penalty, constraint equivalence, L2 penalty, weight decay, AdamW decay	Regularization Methods	Neighboring sections use brief references only
configuration space, conditional parameter, log-uniform sampling, grid search, random search	Hyperparameter Optimization	Neighboring sections use brief references only
schedule function, constant learning rate, step decay, exponential decay, polynomial decay	Learning Rate Schedules	Neighboring sections use brief references only

Overlap Danger Zones

Convexity versus constraints: convex duality is introduced in 01; full KKT machinery belongs to 04.
Deterministic GD versus SGD: 02 owns exact gradients; 05 owns stochastic oracles and gradient noise.
Second-order methods versus adaptive methods: 03 owns curvature matrices; 07 owns diagonal and layerwise adaptive optimizer state.
Regularization versus statistics: 08 owns optimization penalties and constraints; Chapter 7 owns statistical bias-variance and regression interpretation.
Learning-rate choice versus schedule shape: 02 analyzes fixed step sizes; 10 owns time-varying schedules.
Hyperparameter optimization versus schedule formulas: 09 owns search procedures; 10 owns the mathematical shapes being searched.

Key Cross-Chapter Dependencies

From Chapter 3: eigenvalues, PSD matrices, SVD, matrix norms, and condition number.
From Chapters 4-5: gradients, Hessians, Jacobians, Taylor expansions, and backpropagation.
From Chapters 6-7: probability, expectation, empirical risk, MLE, MAP, and validation logic.
Into Chapter 9: cross-entropy, KL divergence, Fisher information, and information geometry.
Into Chapter 10: numerical stability, finite-difference checks, implementation-level line search, and floating-point effects.

2026 Optimization Best Practices For AI/LLMs

Start from AdamW or a well-tested first-order baseline, then justify any more exotic optimizer with ablations.
Separate optimizer choice, regularization, schedule, batch size, and precision; each changes the effective update.
Always log gradient norm, update norm, parameter norm, learning rate, loss, validation metric, and optimizer-state summaries.
Use warmup when early curvature, normalization statistics, or optimizer state makes full-rate updates unstable.
Treat batch size and learning rate as coupled variables, especially under gradient accumulation and data parallelism.
Do not tune on the test set; use validation and nested evaluation for model-selection claims.
Check notebook-scale experiments before claiming a large-run behavior is understood.
Use higher-precision small-batch checks when mixed precision produces suspicious loss spikes.
Prefer explicit scope boundaries over duplicate theory across sections.
Run every notebook top-to-bottom after edits; valid JSON is not the same as a valid learning artifact.

Curated Online Resources

Boyd and Vandenberghe, Convex Optimization: https://web.stanford.edu/~boyd/cvxbook/
Stanford EE364A Convex Optimization: https://web.stanford.edu/class/ee364a/
Nocedal and Wright, Numerical Optimization.
Bottou, Curtis, and Nocedal, Optimization Methods for Large-Scale Machine Learning.
PyTorch optimizer and scheduler docs: https://docs.pytorch.org/docs/stable/optim.html
Optax optimizer transformations: https://optax.readthedocs.io/