Lesson overview | Previous part | Next part

Adaptive Learning Rate: Part 5: Core Theory III: Practical Variants to 6. Advanced Topics

5. Core Theory III: Practical Variants

This block develops core theory iii: practical variants for Adaptive Learning Rate. It keeps the scope local to this section while pointing forward when a neighboring topic owns the full treatment.

5.1 Variant built around AdamW

In this section, trust ratio is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Adaptive Learning Rate, the phrase "Variant built around AdamW" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, trust ratio is the part of Adaptive Learning Rate that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where trust ratio can be computed directly and compared with theory.
• A logistic-regression or softmax objective where trust ratio affects optimization but the model remains interpretable.
• A transformer training diagnostic where trust ratio appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating trust ratio as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t}+\epsilon}$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving trust ratio, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes trust ratio visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = (1-\eta\lambda)\boldsymbol{\theta}_t - \eta \mathbf{u}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about trust ratio is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• AdamW as the default optimizer for transformer pretraining and fine-tuning.
• Adafactor for memory-constrained large models.
• LAMB and LARS for large-batch training.
• optimizer-state diagnostics for training failures and loss spikes.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

5.2 Variant built around coupled L2

In this section, layerwise scaling is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Adaptive Learning Rate, the phrase "Variant built around coupled L2" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, layerwise scaling is the part of Adaptive Learning Rate that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where layerwise scaling can be computed directly and compared with theory.
• A logistic-regression or softmax objective where layerwise scaling affects optimization but the model remains interpretable.
• A transformer training diagnostic where layerwise scaling appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating layerwise scaling as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t}+\epsilon}$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving layerwise scaling, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes layerwise scaling visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = (1-\eta\lambda)\boldsymbol{\theta}_t - \eta \mathbf{u}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about layerwise scaling is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• AdamW as the default optimizer for transformer pretraining and fine-tuning.
• Adafactor for memory-constrained large models.
• LAMB and LARS for large-batch training.
• optimizer-state diagnostics for training failures and loss spikes.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

5.3 Variant built around decoupled weight decay

In this section, Shampoo preview is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Adaptive Learning Rate, the phrase "Variant built around decoupled weight decay" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, Shampoo preview is the part of Adaptive Learning Rate that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where Shampoo preview can be computed directly and compared with theory.
• A logistic-regression or softmax objective where Shampoo preview affects optimization but the model remains interpretable.
• A transformer training diagnostic where Shampoo preview appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating Shampoo preview as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t}+\epsilon}$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving Shampoo preview, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes Shampoo preview visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = (1-\eta\lambda)\boldsymbol{\theta}_t - \eta \mathbf{u}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about Shampoo preview is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• AdamW as the default optimizer for transformer pretraining and fine-tuning.
• Adafactor for memory-constrained large models.
• LAMB and LARS for large-batch training.
• optimizer-state diagnostics for training failures and loss spikes.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

5.4 Implementation constraints and numerical stability

In this section, SOAP preview is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Adaptive Learning Rate, the phrase "Implementation constraints and numerical stability" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, SOAP preview is the part of Adaptive Learning Rate that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where SOAP preview can be computed directly and compared with theory.
• A logistic-regression or softmax objective where SOAP preview affects optimization but the model remains interpretable.
• A transformer training diagnostic where SOAP preview appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating SOAP preview as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t}+\epsilon}$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving SOAP preview, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes SOAP preview visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = (1-\eta\lambda)\boldsymbol{\theta}_t - \eta \mathbf{u}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about SOAP preview is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• AdamW as the default optimizer for transformer pretraining and fine-tuning.
• Adafactor for memory-constrained large models.
• LAMB and LARS for large-batch training.
• optimizer-state diagnostics for training failures and loss spikes.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

5.5 What belongs here versus neighboring sections

In this section, Muon preview is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Adaptive Learning Rate, the phrase "What belongs here versus neighboring sections" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, Muon preview is the part of Adaptive Learning Rate that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where Muon preview can be computed directly and compared with theory.
• A logistic-regression or softmax objective where Muon preview affects optimization but the model remains interpretable.
• A transformer training diagnostic where Muon preview appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating Muon preview as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t}+\epsilon}$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving Muon preview, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes Muon preview visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = (1-\eta\lambda)\boldsymbol{\theta}_t - \eta \mathbf{u}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about Muon preview is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• AdamW as the default optimizer for transformer pretraining and fine-tuning.
• Adafactor for memory-constrained large models.
• LAMB and LARS for large-batch training.
• optimizer-state diagnostics for training failures and loss spikes.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

6. Advanced Topics

This block develops advanced topics for Adaptive Learning Rate. It keeps the scope local to this section while pointing forward when a neighboring topic owns the full treatment.

6.1 Advanced view of Adafactor

In this section, SOAP preview is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Adaptive Learning Rate, the phrase "Advanced view of Adafactor" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, SOAP preview is the part of Adaptive Learning Rate that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where SOAP preview can be computed directly and compared with theory.
• A logistic-regression or softmax objective where SOAP preview affects optimization but the model remains interpretable.
• A transformer training diagnostic where SOAP preview appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating SOAP preview as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t}+\epsilon}$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving SOAP preview, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes SOAP preview visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = (1-\eta\lambda)\boldsymbol{\theta}_t - \eta \mathbf{u}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about SOAP preview is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• AdamW as the default optimizer for transformer pretraining and fine-tuning.
• Adafactor for memory-constrained large models.
• LAMB and LARS for large-batch training.
• optimizer-state diagnostics for training failures and loss spikes.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

6.2 Advanced view of factored second moment

In this section, Muon preview is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Adaptive Learning Rate, the phrase "Advanced view of factored second moment" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, Muon preview is the part of Adaptive Learning Rate that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where Muon preview can be computed directly and compared with theory.
• A logistic-regression or softmax objective where Muon preview affects optimization but the model remains interpretable.
• A transformer training diagnostic where Muon preview appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating Muon preview as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t}+\epsilon}$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving Muon preview, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes Muon preview visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = (1-\eta\lambda)\boldsymbol{\theta}_t - \eta \mathbf{u}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about Muon preview is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• AdamW as the default optimizer for transformer pretraining and fine-tuning.
• Adafactor for memory-constrained large models.
• LAMB and LARS for large-batch training.
• optimizer-state diagnostics for training failures and loss spikes.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

6.3 Advanced view of LARS

In this section, optimizer state diagnostics is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Adaptive Learning Rate, the phrase "Advanced view of LARS" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, optimizer state diagnostics is the part of Adaptive Learning Rate that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where optimizer state diagnostics can be computed directly and compared with theory.
• A logistic-regression or softmax objective where optimizer state diagnostics affects optimization but the model remains interpretable.
• A transformer training diagnostic where optimizer state diagnostics appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating optimizer state diagnostics as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t}+\epsilon}$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving optimizer state diagnostics, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes optimizer state diagnostics visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = (1-\eta\lambda)\boldsymbol{\theta}_t - \eta \mathbf{u}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about optimizer state diagnostics is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• AdamW as the default optimizer for transformer pretraining and fine-tuning.
• Adafactor for memory-constrained large models.
• LAMB and LARS for large-batch training.
• optimizer-state diagnostics for training failures and loss spikes.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

6.4 Infinite-dimensional or large-scale interpretation

In this section, effective learning rate is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Adaptive Learning Rate, the phrase "Infinite-dimensional or large-scale interpretation" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, effective learning rate is the part of Adaptive Learning Rate that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where effective learning rate can be computed directly and compared with theory.
• A logistic-regression or softmax objective where effective learning rate affects optimization but the model remains interpretable.
• A transformer training diagnostic where effective learning rate appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating effective learning rate as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t}+\epsilon}$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving effective learning rate, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes effective learning rate visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = (1-\eta\lambda)\boldsymbol{\theta}_t - \eta \mathbf{u}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about effective learning rate is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• AdamW as the default optimizer for transformer pretraining and fine-tuning.
• Adafactor for memory-constrained large models.
• LAMB and LARS for large-batch training.
• optimizer-state diagnostics for training failures and loss spikes.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

6.5 Open questions for frontier model training

In this section, diagonal preconditioner is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Adaptive Learning Rate, the phrase "Open questions for frontier model training" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, diagonal preconditioner is the part of Adaptive Learning Rate that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where diagonal preconditioner can be computed directly and compared with theory.
• A logistic-regression or softmax objective where diagonal preconditioner affects optimization but the model remains interpretable.
• A transformer training diagnostic where diagonal preconditioner appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating diagonal preconditioner as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t}+\epsilon}$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving diagonal preconditioner, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes diagonal preconditioner visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\theta}_{t+1} = (1-\eta\lambda)\boldsymbol{\theta}_t - \eta \mathbf{u}_t$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about diagonal preconditioner is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• AdamW as the default optimizer for transformer pretraining and fine-tuning.
• Adafactor for memory-constrained large models.
• LAMB and LARS for large-batch training.
• optimizer-state diagnostics for training failures and loss spikes.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.