Lesson overview | Previous part | Next part

Stochastic Optimization: Part 5: Core Theory III: Practical Variants to 6. Advanced Topics

5. Core Theory III: Practical Variants

This block develops core theory iii: practical variants for Stochastic Optimization. It keeps the scope local to this section while pointing forward when a neighboring topic owns the full treatment.

5.1 Variant built around SGD convergence

In this section, Polyak averaging is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Stochastic Optimization, the phrase "Variant built around SGD convergence" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, Polyak averaging is the part of Stochastic Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where Polyak averaging can be computed directly and compared with theory.
• A logistic-regression or softmax objective where Polyak averaging affects optimization but the model remains interpretable.
• A transformer training diagnostic where Polyak averaging appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating Polyak averaging as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\mathbf{g}_t = \frac{1}{B}\sum_{i \in \mathcal{B}_t}\nabla_{\boldsymbol{\theta}}\ell(\boldsymbol{\theta}_t; \mathbf{x}^{(i)}, y^{(i)})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving Polyak averaging, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes Polyak averaging visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \mathbf{g}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about Polyak averaging is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• minibatch training for deep networks and transformers.
• batch-size and learning-rate coupling in large-scale pretraining.
• distributed gradient averaging under data parallelism.
• variance reduction ideas behind efficient fine-tuning and classical ML solvers.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

5.2 Variant built around strongly convex SGD

In this section, distributed SGD is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Stochastic Optimization, the phrase "Variant built around strongly convex SGD" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, distributed SGD is the part of Stochastic Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where distributed SGD can be computed directly and compared with theory.
• A logistic-regression or softmax objective where distributed SGD affects optimization but the model remains interpretable.
• A transformer training diagnostic where distributed SGD appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating distributed SGD as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\mathbf{g}_t = \frac{1}{B}\sum_{i \in \mathcal{B}_t}\nabla_{\boldsymbol{\theta}}\ell(\boldsymbol{\theta}_t; \mathbf{x}^{(i)}, y^{(i)})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving distributed SGD, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes distributed SGD visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \mathbf{g}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about distributed SGD is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• minibatch training for deep networks and transformers.
• batch-size and learning-rate coupling in large-scale pretraining.
• distributed gradient averaging under data parallelism.
• variance reduction ideas behind efficient fine-tuning and classical ML solvers.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

5.3 Variant built around nonconvex SGD

In this section, gradient accumulation is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Stochastic Optimization, the phrase "Variant built around nonconvex SGD" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, gradient accumulation is the part of Stochastic Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where gradient accumulation can be computed directly and compared with theory.
• A logistic-regression or softmax objective where gradient accumulation affects optimization but the model remains interpretable.
• A transformer training diagnostic where gradient accumulation appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating gradient accumulation as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\mathbf{g}_t = \frac{1}{B}\sum_{i \in \mathcal{B}_t}\nabla_{\boldsymbol{\theta}}\ell(\boldsymbol{\theta}_t; \mathbf{x}^{(i)}, y^{(i)})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving gradient accumulation, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes gradient accumulation visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \mathbf{g}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about gradient accumulation is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• minibatch training for deep networks and transformers.
• batch-size and learning-rate coupling in large-scale pretraining.
• distributed gradient averaging under data parallelism.
• variance reduction ideas behind efficient fine-tuning and classical ML solvers.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

5.4 Implementation constraints and numerical stability

In this section, local SGD is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Stochastic Optimization, the phrase "Implementation constraints and numerical stability" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, local SGD is the part of Stochastic Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where local SGD can be computed directly and compared with theory.
• A logistic-regression or softmax objective where local SGD affects optimization but the model remains interpretable.
• A transformer training diagnostic where local SGD appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating local SGD as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\mathbf{g}_t = \frac{1}{B}\sum_{i \in \mathcal{B}_t}\nabla_{\boldsymbol{\theta}}\ell(\boldsymbol{\theta}_t; \mathbf{x}^{(i)}, y^{(i)})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving local SGD, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes local SGD visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \mathbf{g}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about local SGD is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• minibatch training for deep networks and transformers.
• batch-size and learning-rate coupling in large-scale pretraining.
• distributed gradient averaging under data parallelism.
• variance reduction ideas behind efficient fine-tuning and classical ML solvers.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

5.5 What belongs here versus neighboring sections

In this section, federated averaging is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Stochastic Optimization, the phrase "What belongs here versus neighboring sections" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, federated averaging is the part of Stochastic Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where federated averaging can be computed directly and compared with theory.
• A logistic-regression or softmax objective where federated averaging affects optimization but the model remains interpretable.
• A transformer training diagnostic where federated averaging appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating federated averaging as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\mathbf{g}_t = \frac{1}{B}\sum_{i \in \mathcal{B}_t}\nabla_{\boldsymbol{\theta}}\ell(\boldsymbol{\theta}_t; \mathbf{x}^{(i)}, y^{(i)})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving federated averaging, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes federated averaging visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \mathbf{g}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about federated averaging is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• minibatch training for deep networks and transformers.
• batch-size and learning-rate coupling in large-scale pretraining.
• distributed gradient averaging under data parallelism.
• variance reduction ideas behind efficient fine-tuning and classical ML solvers.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

6. Advanced Topics

This block develops advanced topics for Stochastic Optimization. It keeps the scope local to this section while pointing forward when a neighboring topic owns the full treatment.

6.1 Advanced view of gradient noise scale

In this section, local SGD is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Stochastic Optimization, the phrase "Advanced view of gradient noise scale" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, local SGD is the part of Stochastic Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where local SGD can be computed directly and compared with theory.
• A logistic-regression or softmax objective where local SGD affects optimization but the model remains interpretable.
• A transformer training diagnostic where local SGD appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating local SGD as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\mathbf{g}_t = \frac{1}{B}\sum_{i \in \mathcal{B}_t}\nabla_{\boldsymbol{\theta}}\ell(\boldsymbol{\theta}_t; \mathbf{x}^{(i)}, y^{(i)})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving local SGD, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes local SGD visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \mathbf{g}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about local SGD is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• minibatch training for deep networks and transformers.
• batch-size and learning-rate coupling in large-scale pretraining.
• distributed gradient averaging under data parallelism.
• variance reduction ideas behind efficient fine-tuning and classical ML solvers.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

6.2 Advanced view of SVRG

In this section, federated averaging is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Stochastic Optimization, the phrase "Advanced view of SVRG" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, federated averaging is the part of Stochastic Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where federated averaging can be computed directly and compared with theory.
• A logistic-regression or softmax objective where federated averaging affects optimization but the model remains interpretable.
• A transformer training diagnostic where federated averaging appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating federated averaging as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\mathbf{g}_t = \frac{1}{B}\sum_{i \in \mathcal{B}_t}\nabla_{\boldsymbol{\theta}}\ell(\boldsymbol{\theta}_t; \mathbf{x}^{(i)}, y^{(i)})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving federated averaging, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes federated averaging visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \mathbf{g}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about federated averaging is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• minibatch training for deep networks and transformers.
• batch-size and learning-rate coupling in large-scale pretraining.
• distributed gradient averaging under data parallelism.
• variance reduction ideas behind efficient fine-tuning and classical ML solvers.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

6.3 Advanced view of SAGA

In this section, communication compression is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Stochastic Optimization, the phrase "Advanced view of SAGA" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, communication compression is the part of Stochastic Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where communication compression can be computed directly and compared with theory.
• A logistic-regression or softmax objective where communication compression affects optimization but the model remains interpretable.
• A transformer training diagnostic where communication compression appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating communication compression as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\mathbf{g}_t = \frac{1}{B}\sum_{i \in \mathcal{B}_t}\nabla_{\boldsymbol{\theta}}\ell(\boldsymbol{\theta}_t; \mathbf{x}^{(i)}, y^{(i)})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving communication compression, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes communication compression visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \mathbf{g}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about communication compression is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• minibatch training for deep networks and transformers.
• batch-size and learning-rate coupling in large-scale pretraining.
• distributed gradient averaging under data parallelism.
• variance reduction ideas behind efficient fine-tuning and classical ML solvers.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

6.4 Infinite-dimensional or large-scale interpretation

In this section, LLM pretraining noise is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Stochastic Optimization, the phrase "Infinite-dimensional or large-scale interpretation" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, LLM pretraining noise is the part of Stochastic Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where LLM pretraining noise can be computed directly and compared with theory.
• A logistic-regression or softmax objective where LLM pretraining noise affects optimization but the model remains interpretable.
• A transformer training diagnostic where LLM pretraining noise appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating LLM pretraining noise as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\mathbf{g}_t = \frac{1}{B}\sum_{i \in \mathcal{B}_t}\nabla_{\boldsymbol{\theta}}\ell(\boldsymbol{\theta}_t; \mathbf{x}^{(i)}, y^{(i)})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving LLM pretraining noise, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes LLM pretraining noise visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \mathbf{g}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about LLM pretraining noise is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• minibatch training for deep networks and transformers.
• batch-size and learning-rate coupling in large-scale pretraining.
• distributed gradient averaging under data parallelism.
• variance reduction ideas behind efficient fine-tuning and classical ML solvers.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

6.5 Open questions for frontier model training

In this section, stochastic objective is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Stochastic Optimization, the phrase "Open questions for frontier model training" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, stochastic objective is the part of Stochastic Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where stochastic objective can be computed directly and compared with theory.
• A logistic-regression or softmax objective where stochastic objective affects optimization but the model remains interpretable.
• A transformer training diagnostic where stochastic objective appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating stochastic objective as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\mathbf{g}_t = \frac{1}{B}\sum_{i \in \mathcal{B}_t}\nabla_{\boldsymbol{\theta}}\ell(\boldsymbol{\theta}_t; \mathbf{x}^{(i)}, y^{(i)})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving stochastic objective, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes stochastic objective visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \mathbf{g}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about stochastic objective is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• minibatch training for deep networks and transformers.
• batch-size and learning-rate coupling in large-scale pretraining.
• distributed gradient averaging under data parallelism.
• variance reduction ideas behind efficient fine-tuning and classical ML solvers.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.