Lesson overview | Lesson overview | Next part

Hyperparameter Optimization: Part 1: Intuition to 2. Formal Definitions

1. Intuition

This block develops intuition for Hyperparameter Optimization. It keeps the scope local to this section while pointing forward when a neighboring topic owns the full treatment.

1.1 Why Hyperparameter Optimization matters for training systems

In this section, random search is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Hyperparameter Optimization, the phrase "Why Hyperparameter Optimization matters for training systems" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, random search is the part of Hyperparameter Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where random search can be computed directly and compared with theory.
• A logistic-regression or softmax objective where random search affects optimization but the model remains interpretable.
• A transformer training diagnostic where random search appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating random search as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\lambda}^* \in \arg\min_{\boldsymbol{\lambda}\in\Lambda} \mathcal{V}(\mathcal{A}(\boldsymbol{\lambda}), \mathcal{D}_{\mathrm{val}})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving random search, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes random search visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\lambda}_{t+1} = \arg\max_{\boldsymbol{\lambda}\in\Lambda} a_t(\boldsymbol{\lambda})$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about random search is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• learning-rate, weight-decay, batch-size, and schedule tuning for LLM fine-tuning.
• Hyperband and ASHA for neural architecture and training-budget search.
• Bayesian optimization for expensive, low-dimensional continuous tuning.
• validation leakage prevention in model-selection pipelines.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

1.2 The optimization object: parameters, objective, algorithm, and diagnostic

In this section, Sobol initialization is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Hyperparameter Optimization, the phrase "The optimization object: parameters, objective, algorithm, and diagnostic" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, Sobol initialization is the part of Hyperparameter Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where Sobol initialization can be computed directly and compared with theory.
• A logistic-regression or softmax objective where Sobol initialization affects optimization but the model remains interpretable.
• A transformer training diagnostic where Sobol initialization appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating Sobol initialization as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\lambda}^* \in \arg\min_{\boldsymbol{\lambda}\in\Lambda} \mathcal{V}(\mathcal{A}(\boldsymbol{\lambda}), \mathcal{D}_{\mathrm{val}})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving Sobol initialization, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes Sobol initialization visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\lambda}_{t+1} = \arg\max_{\boldsymbol{\lambda}\in\Lambda} a_t(\boldsymbol{\lambda})$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about Sobol initialization is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• learning-rate, weight-decay, batch-size, and schedule tuning for LLM fine-tuning.
• Hyperband and ASHA for neural architecture and training-budget search.
• Bayesian optimization for expensive, low-dimensional continuous tuning.
• validation leakage prevention in model-selection pipelines.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

1.3 Historical arc from classical optimization to modern AI

In this section, surrogate model is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Hyperparameter Optimization, the phrase "Historical arc from classical optimization to modern AI" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, surrogate model is the part of Hyperparameter Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where surrogate model can be computed directly and compared with theory.
• A logistic-regression or softmax objective where surrogate model affects optimization but the model remains interpretable.
• A transformer training diagnostic where surrogate model appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating surrogate model as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\lambda}^* \in \arg\min_{\boldsymbol{\lambda}\in\Lambda} \mathcal{V}(\mathcal{A}(\boldsymbol{\lambda}), \mathcal{D}_{\mathrm{val}})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving surrogate model, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes surrogate model visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\lambda}_{t+1} = \arg\max_{\boldsymbol{\lambda}\in\Lambda} a_t(\boldsymbol{\lambda})$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about surrogate model is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• learning-rate, weight-decay, batch-size, and schedule tuning for LLM fine-tuning.
• Hyperband and ASHA for neural architecture and training-budget search.
• Bayesian optimization for expensive, low-dimensional continuous tuning.
• validation leakage prevention in model-selection pipelines.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

1.4 What this section treats as canonical scope

In this section, Gaussian process is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Hyperparameter Optimization, the phrase "What this section treats as canonical scope" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, Gaussian process is the part of Hyperparameter Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where Gaussian process can be computed directly and compared with theory.
• A logistic-regression or softmax objective where Gaussian process affects optimization but the model remains interpretable.
• A transformer training diagnostic where Gaussian process appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating Gaussian process as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\lambda}^* \in \arg\min_{\boldsymbol{\lambda}\in\Lambda} \mathcal{V}(\mathcal{A}(\boldsymbol{\lambda}), \mathcal{D}_{\mathrm{val}})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving Gaussian process, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes Gaussian process visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\lambda}_{t+1} = \arg\max_{\boldsymbol{\lambda}\in\Lambda} a_t(\boldsymbol{\lambda})$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about Gaussian process is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• learning-rate, weight-decay, batch-size, and schedule tuning for LLM fine-tuning.
• Hyperband and ASHA for neural architecture and training-budget search.
• Bayesian optimization for expensive, low-dimensional continuous tuning.
• validation leakage prevention in model-selection pipelines.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

1.5 A first mental model for LLM training

In this section, expected improvement is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Hyperparameter Optimization, the phrase "A first mental model for LLM training" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, expected improvement is the part of Hyperparameter Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where expected improvement can be computed directly and compared with theory.
• A logistic-regression or softmax objective where expected improvement affects optimization but the model remains interpretable.
• A transformer training diagnostic where expected improvement appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating expected improvement as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\lambda}^* \in \arg\min_{\boldsymbol{\lambda}\in\Lambda} \mathcal{V}(\mathcal{A}(\boldsymbol{\lambda}), \mathcal{D}_{\mathrm{val}})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving expected improvement, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes expected improvement visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\lambda}_{t+1} = \arg\max_{\boldsymbol{\lambda}\in\Lambda} a_t(\boldsymbol{\lambda})$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about expected improvement is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• learning-rate, weight-decay, batch-size, and schedule tuning for LLM fine-tuning.
• Hyperband and ASHA for neural architecture and training-budget search.
• Bayesian optimization for expensive, low-dimensional continuous tuning.
• validation leakage prevention in model-selection pipelines.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

2. Formal Definitions

This block develops formal definitions for Hyperparameter Optimization. It keeps the scope local to this section while pointing forward when a neighboring topic owns the full treatment.

2.1 Primary definition: configuration space

In this section, Gaussian process is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Hyperparameter Optimization, the phrase "Primary definition: configuration space" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, Gaussian process is the part of Hyperparameter Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where Gaussian process can be computed directly and compared with theory.
• A logistic-regression or softmax objective where Gaussian process affects optimization but the model remains interpretable.
• A transformer training diagnostic where Gaussian process appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating Gaussian process as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\lambda}^* \in \arg\min_{\boldsymbol{\lambda}\in\Lambda} \mathcal{V}(\mathcal{A}(\boldsymbol{\lambda}), \mathcal{D}_{\mathrm{val}})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving Gaussian process, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes Gaussian process visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\lambda}_{t+1} = \arg\max_{\boldsymbol{\lambda}\in\Lambda} a_t(\boldsymbol{\lambda})$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about Gaussian process is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• learning-rate, weight-decay, batch-size, and schedule tuning for LLM fine-tuning.
• Hyperband and ASHA for neural architecture and training-budget search.
• Bayesian optimization for expensive, low-dimensional continuous tuning.
• validation leakage prevention in model-selection pipelines.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

2.2 Secondary definition: conditional parameter

In this section, expected improvement is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Hyperparameter Optimization, the phrase "Secondary definition: conditional parameter" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, expected improvement is the part of Hyperparameter Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where expected improvement can be computed directly and compared with theory.
• A logistic-regression or softmax objective where expected improvement affects optimization but the model remains interpretable.
• A transformer training diagnostic where expected improvement appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating expected improvement as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\lambda}^* \in \arg\min_{\boldsymbol{\lambda}\in\Lambda} \mathcal{V}(\mathcal{A}(\boldsymbol{\lambda}), \mathcal{D}_{\mathrm{val}})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving expected improvement, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes expected improvement visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\lambda}_{t+1} = \arg\max_{\boldsymbol{\lambda}\in\Lambda} a_t(\boldsymbol{\lambda})$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about expected improvement is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• learning-rate, weight-decay, batch-size, and schedule tuning for LLM fine-tuning.
• Hyperband and ASHA for neural architecture and training-budget search.
• Bayesian optimization for expensive, low-dimensional continuous tuning.
• validation leakage prevention in model-selection pipelines.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

2.3 Algorithmic object: log-uniform sampling

In this section, upper confidence bound is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Hyperparameter Optimization, the phrase "Algorithmic object: log-uniform sampling" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, upper confidence bound is the part of Hyperparameter Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where upper confidence bound can be computed directly and compared with theory.
• A logistic-regression or softmax objective where upper confidence bound affects optimization but the model remains interpretable.
• A transformer training diagnostic where upper confidence bound appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating upper confidence bound as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\lambda}^* \in \arg\min_{\boldsymbol{\lambda}\in\Lambda} \mathcal{V}(\mathcal{A}(\boldsymbol{\lambda}), \mathcal{D}_{\mathrm{val}})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving upper confidence bound, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes upper confidence bound visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\lambda}_{t+1} = \arg\max_{\boldsymbol{\lambda}\in\Lambda} a_t(\boldsymbol{\lambda})$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about upper confidence bound is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• learning-rate, weight-decay, batch-size, and schedule tuning for LLM fine-tuning.
• Hyperband and ASHA for neural architecture and training-budget search.
• Bayesian optimization for expensive, low-dimensional continuous tuning.
• validation leakage prevention in model-selection pipelines.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

2.4 Examples, non-examples, and boundary cases

In this section, Thompson sampling is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Hyperparameter Optimization, the phrase "Examples, non-examples, and boundary cases" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, Thompson sampling is the part of Hyperparameter Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where Thompson sampling can be computed directly and compared with theory.
• A logistic-regression or softmax objective where Thompson sampling affects optimization but the model remains interpretable.
• A transformer training diagnostic where Thompson sampling appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating Thompson sampling as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\lambda}^* \in \arg\min_{\boldsymbol{\lambda}\in\Lambda} \mathcal{V}(\mathcal{A}(\boldsymbol{\lambda}), \mathcal{D}_{\mathrm{val}})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving Thompson sampling, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes Thompson sampling visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\lambda}_{t+1} = \arg\max_{\boldsymbol{\lambda}\in\Lambda} a_t(\boldsymbol{\lambda})$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about Thompson sampling is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• learning-rate, weight-decay, batch-size, and schedule tuning for LLM fine-tuning.
• Hyperband and ASHA for neural architecture and training-budget search.
• Bayesian optimization for expensive, low-dimensional continuous tuning.
• validation leakage prevention in model-selection pipelines.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.

2.5 Notation, dimensions, and assumptions

In this section, Bayesian optimization is treated as a concrete optimization object rather than a slogan. The goal is to understand how it changes the objective, the update rule, the convergence story, and the diagnostics a practitioner should inspect when training a modern model. For Hyperparameter Optimization, the phrase "Notation, dimensions, and assumptions" means a precise mathematical habit: state the assumptions, write the update, identify what can be measured, and connect the result to a real AI training decision.

Definition.

For this section, Bayesian optimization is the part of Hyperparameter Optimization that controls how the objective, feasible region, or update rule behaves under the assumptions currently in force.

Symbolically, we track it through $f$, $\boldsymbol{\theta}$, $\eta$, $\nabla f(\boldsymbol{\theta})$, and any auxiliary state used by the algorithm.

Examples:

• A small synthetic quadratic where Bayesian optimization can be computed directly and compared with theory.
• A logistic-regression or softmax objective where Bayesian optimization affects optimization but the model remains interpretable.
• A transformer training diagnostic where Bayesian optimization appears through gradient norms, update norms, curvature, or validation loss.

Non-examples:

• Treating Bayesian optimization as a hyperparameter recipe without checking the objective assumptions.
• Inferring global behavior from one noisy minibatch when the section requires a population or full-batch statement.

Useful formula:

$$\boldsymbol{\lambda}^* \in \arg\min_{\boldsymbol{\lambda}\in\Lambda} \mathcal{V}(\mathcal{A}(\boldsymbol{\lambda}), \mathcal{D}_{\mathrm{val}})$$

Proof sketch or reasoning pattern:

Start with the local model around $\boldsymbol{\theta}_t$, isolate the term involving Bayesian optimization, and use the section assumptions to bound the change in objective value. If the assumption is geometric, the proof turns a picture into an inequality. If the assumption is stochastic, the proof takes conditional expectation before applying the bound. If the assumption is algorithmic, the proof checks that the proposed update is a descent, projection, or preconditioning step. This pattern is reusable across optimization theory.

Implementation consequence:

• Log a metric that makes Bayesian optimization visible; otherwise a training run can fail while the scalar loss hides the cause.
• Compare the measured update with the mathematical update below before blaming data or architecture.

$$\boldsymbol{\lambda}_{t+1} = \arg\max_{\boldsymbol{\lambda}\in\Lambda} a_t(\boldsymbol{\lambda})$$

• Keep units straight: parameter norm, gradient norm, update norm, objective value, and validation metric are different objects.

Diagnostic questions:

• Which assumption about Bayesian optimization is most fragile in the current training setup?
• What number would you log to catch the failure one thousand steps before divergence?

AI connection:

• learning-rate, weight-decay, batch-size, and schedule tuning for LLM fine-tuning.
• Hyperband and ASHA for neural architecture and training-budget search.
• Bayesian optimization for expensive, low-dimensional continuous tuning.
• validation leakage prevention in model-selection pipelines.

Local scope boundary: This subsection may reference neighboring material, but the full canonical treatment stays in its own folder. For example, stochastic gradient noise belongs to Stochastic Optimization, external schedule shapes belong to Learning Rate Schedules, and cross-entropy as an information measure belongs to Cross-Entropy.