Lesson overview | Previous part | Next part

Preference Optimization RLHF and DPO: Part 5: DPO Derivation

5. DPO Derivation

DPO Derivation develops the part of preference optimization rlhf and dpo that the approved TOC assigns to Chapter 18. The emphasis is alignment behavior, safety constraints, and feedback loops, not generic fine-tuning or production monitoring.

5.1 KL-constrained optimal policy

KL-constrained optimal policy belongs in the canonical scope of preference optimization rlhf and dpo. The object is the preference-aligned policy, not merely a prompt trick or a moderation label. We study how data, losses, policies, review processes, and safety constraints shape a model's conditional distribution over responses.

A compact way to read this subsection is through the local symbol (x,y_w,y_l). It marks the alignment object being transformed: an instruction policy, a preference pair, a violation classifier, a guardrail action, or a feedback event. The details differ, but the discipline is the same: state the object, state the loss or decision rule, then audit the behavioral side effects.

$$\mathcal{L}_{\mathrm{DPO}}(\theta) = -\mathbb{E}\log\sigma\left(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\right).$$

For kl-constrained optimal policy, this formula should not be treated as a slogan. It defines which tokens, responses, comparisons, or decisions receive gradient or operational weight. A change in masking, sampling, rubric wording, or thresholding changes the effective objective even if the model architecture is unchanged.

Examples:

• Treat kl-constrained optimal policy as part of the model contract and store the exact data version.
• Record the prompt template, role format, policy version, and decoder settings.
• Compare aligned and reference policies on both helpfulness and safety slices.
• Use held-out examples that were not used to tune refusals or rewards.
• Inspect failure cases before declaring the objective successful.

Non-examples:

• Calling a model aligned because it sounds polite on a few prompts.
• Training on refusals without measuring over-refusal on benign requests.
• Using a reward model as ground truth without calibration or adversarial checks.
• Shipping a guardrail threshold without measuring false positive and false negative rates.
• Letting feedback logs change training without provenance or consent controls.

A useful implementation pattern is to separate policy, data, and measurement. The policy says what behavior is desired. The data supplies examples, comparisons, attacks, or feedback events. The measurement checks whether the updated system moved in the intended direction without unacceptable regressions.

policy text/rubric
      |
      v
training or guardrail data  ->  objective/threshold  ->  aligned system
      |                                                   |
      v                                                   v
audit metadata                                      held-out safety eval

Worked reasoning pattern for kl-constrained optimal policy:

1. Name the target behavior in plain language.
2. Write the mathematical variable that represents it.
3. Specify which examples or comparisons estimate it.
4. Choose the optimization loss or runtime decision rule.
5. Define the regression metric that would prove the change became worse.

Three details are especially easy to miss in alignment work. First, the user intent distribution is not the same as the pretraining distribution. Second, safety labels are not ordinary class labels; they encode policy judgments that can change by context. Third, optimization pressure finds shortcuts, so every proxy must be monitored for Goodhart-style failures.

AI connection: KL-constrained optimal policy is part of the post-training stack used by modern assistant systems. It links the base language model to human intent, safety policy, and deployment constraints without pretending that a single loss can capture all values. The goal is not perfect alignment by formula; it is a repeatable loop where evidence, objectives, and safeguards improve together.

5.2 Implicit reward

Implicit reward belongs in the canonical scope of preference optimization rlhf and dpo. The object is the preference-aligned policy, not merely a prompt trick or a moderation label. We study how data, losses, policies, review processes, and safety constraints shape a model's conditional distribution over responses.

A compact way to read this subsection is through the local symbol (x,y_w,y_l). It marks the alignment object being transformed: an instruction policy, a preference pair, a violation classifier, a guardrail action, or a feedback event. The details differ, but the discipline is the same: state the object, state the loss or decision rule, then audit the behavioral side effects.

$$\mathcal{L}_{\mathrm{DPO}}(\theta) = -\mathbb{E}\log\sigma\left(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\right).$$

For implicit reward, this formula should not be treated as a slogan. It defines which tokens, responses, comparisons, or decisions receive gradient or operational weight. A change in masking, sampling, rubric wording, or thresholding changes the effective objective even if the model architecture is unchanged.

Examples:

• Treat implicit reward as part of the model contract and store the exact data version.
• Record the prompt template, role format, policy version, and decoder settings.
• Compare aligned and reference policies on both helpfulness and safety slices.
• Use held-out examples that were not used to tune refusals or rewards.
• Inspect failure cases before declaring the objective successful.

Non-examples:

• Calling a model aligned because it sounds polite on a few prompts.
• Training on refusals without measuring over-refusal on benign requests.
• Using a reward model as ground truth without calibration or adversarial checks.
• Shipping a guardrail threshold without measuring false positive and false negative rates.
• Letting feedback logs change training without provenance or consent controls.

A useful implementation pattern is to separate policy, data, and measurement. The policy says what behavior is desired. The data supplies examples, comparisons, attacks, or feedback events. The measurement checks whether the updated system moved in the intended direction without unacceptable regressions.

policy text/rubric
      |
      v
training or guardrail data  ->  objective/threshold  ->  aligned system
      |                                                   |
      v                                                   v
audit metadata                                      held-out safety eval

Worked reasoning pattern for implicit reward:

1. Name the target behavior in plain language.
2. Write the mathematical variable that represents it.
3. Specify which examples or comparisons estimate it.
4. Choose the optimization loss or runtime decision rule.
5. Define the regression metric that would prove the change became worse.

Three details are especially easy to miss in alignment work. First, the user intent distribution is not the same as the pretraining distribution. Second, safety labels are not ordinary class labels; they encode policy judgments that can change by context. Third, optimization pressure finds shortcuts, so every proxy must be monitored for Goodhart-style failures.

AI connection: Implicit reward is part of the post-training stack used by modern assistant systems. It links the base language model to human intent, safety policy, and deployment constraints without pretending that a single loss can capture all values. The goal is not perfect alignment by formula; it is a repeatable loop where evidence, objectives, and safeguards improve together.

5.3 DPO loss

DPO loss belongs in the canonical scope of preference optimization rlhf and dpo. The object is the preference-aligned policy, not merely a prompt trick or a moderation label. We study how data, losses, policies, review processes, and safety constraints shape a model's conditional distribution over responses.

A compact way to read this subsection is through the local symbol (x,y_w,y_l). It marks the alignment object being transformed: an instruction policy, a preference pair, a violation classifier, a guardrail action, or a feedback event. The details differ, but the discipline is the same: state the object, state the loss or decision rule, then audit the behavioral side effects.

$$\mathcal{L}_{\mathrm{DPO}}(\theta) = -\mathbb{E}\log\sigma\left(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\right).$$

For dpo loss, this formula should not be treated as a slogan. It defines which tokens, responses, comparisons, or decisions receive gradient or operational weight. A change in masking, sampling, rubric wording, or thresholding changes the effective objective even if the model architecture is unchanged.

Examples:

• Treat dpo loss as part of the model contract and store the exact data version.
• Record the prompt template, role format, policy version, and decoder settings.
• Compare aligned and reference policies on both helpfulness and safety slices.
• Use held-out examples that were not used to tune refusals or rewards.
• Inspect failure cases before declaring the objective successful.

Non-examples:

• Calling a model aligned because it sounds polite on a few prompts.
• Training on refusals without measuring over-refusal on benign requests.
• Using a reward model as ground truth without calibration or adversarial checks.
• Shipping a guardrail threshold without measuring false positive and false negative rates.
• Letting feedback logs change training without provenance or consent controls.

A useful implementation pattern is to separate policy, data, and measurement. The policy says what behavior is desired. The data supplies examples, comparisons, attacks, or feedback events. The measurement checks whether the updated system moved in the intended direction without unacceptable regressions.

policy text/rubric
      |
      v
training or guardrail data  ->  objective/threshold  ->  aligned system
      |                                                   |
      v                                                   v
audit metadata                                      held-out safety eval

Worked reasoning pattern for dpo loss:

1. Name the target behavior in plain language.
2. Write the mathematical variable that represents it.
3. Specify which examples or comparisons estimate it.
4. Choose the optimization loss or runtime decision rule.
5. Define the regression metric that would prove the change became worse.

Three details are especially easy to miss in alignment work. First, the user intent distribution is not the same as the pretraining distribution. Second, safety labels are not ordinary class labels; they encode policy judgments that can change by context. Third, optimization pressure finds shortcuts, so every proxy must be monitored for Goodhart-style failures.

AI connection: DPO loss is part of the post-training stack used by modern assistant systems. It links the base language model to human intent, safety policy, and deployment constraints without pretending that a single loss can capture all values. The goal is not perfect alignment by formula; it is a repeatable loop where evidence, objectives, and safeguards improve together.

5.4 $\beta$ tradeoff

$\beta$ tradeoff belongs in the canonical scope of preference optimization rlhf and dpo. The object is the preference-aligned policy, not merely a prompt trick or a moderation label. We study how data, losses, policies, review processes, and safety constraints shape a model's conditional distribution over responses.

A compact way to read this subsection is through the local symbol (x,y_w,y_l). It marks the alignment object being transformed: an instruction policy, a preference pair, a violation classifier, a guardrail action, or a feedback event. The details differ, but the discipline is the same: state the object, state the loss or decision rule, then audit the behavioral side effects.

$$\mathcal{L}_{\mathrm{DPO}}(\theta) = -\mathbb{E}\log\sigma\left(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\right).$$

For $\beta$ tradeoff, this formula should not be treated as a slogan. It defines which tokens, responses, comparisons, or decisions receive gradient or operational weight. A change in masking, sampling, rubric wording, or thresholding changes the effective objective even if the model architecture is unchanged.

Examples:

• Treat $\beta$ tradeoff as part of the model contract and store the exact data version.
• Record the prompt template, role format, policy version, and decoder settings.
• Compare aligned and reference policies on both helpfulness and safety slices.
• Use held-out examples that were not used to tune refusals or rewards.
• Inspect failure cases before declaring the objective successful.

Non-examples:

• Calling a model aligned because it sounds polite on a few prompts.
• Training on refusals without measuring over-refusal on benign requests.
• Using a reward model as ground truth without calibration or adversarial checks.
• Shipping a guardrail threshold without measuring false positive and false negative rates.
• Letting feedback logs change training without provenance or consent controls.

A useful implementation pattern is to separate policy, data, and measurement. The policy says what behavior is desired. The data supplies examples, comparisons, attacks, or feedback events. The measurement checks whether the updated system moved in the intended direction without unacceptable regressions.

policy text/rubric
      |
      v
training or guardrail data  ->  objective/threshold  ->  aligned system
      |                                                   |
      v                                                   v
audit metadata                                      held-out safety eval

Worked reasoning pattern for $\beta$ tradeoff:

1. Name the target behavior in plain language.
2. Write the mathematical variable that represents it.
3. Specify which examples or comparisons estimate it.
4. Choose the optimization loss or runtime decision rule.
5. Define the regression metric that would prove the change became worse.

Three details are especially easy to miss in alignment work. First, the user intent distribution is not the same as the pretraining distribution. Second, safety labels are not ordinary class labels; they encode policy judgments that can change by context. Third, optimization pressure finds shortcuts, so every proxy must be monitored for Goodhart-style failures.

AI connection: $\beta$ tradeoff is part of the post-training stack used by modern assistant systems. It links the base language model to human intent, safety policy, and deployment constraints without pretending that a single loss can capture all values. The goal is not perfect alignment by formula; it is a repeatable loop where evidence, objectives, and safeguards improve together.

5.5 Gradient interpretation

Gradient interpretation belongs in the canonical scope of preference optimization rlhf and dpo. The object is the preference-aligned policy, not merely a prompt trick or a moderation label. We study how data, losses, policies, review processes, and safety constraints shape a model's conditional distribution over responses.

A compact way to read this subsection is through the local symbol (x,y_w,y_l). It marks the alignment object being transformed: an instruction policy, a preference pair, a violation classifier, a guardrail action, or a feedback event. The details differ, but the discipline is the same: state the object, state the loss or decision rule, then audit the behavioral side effects.

$$\mathcal{L}_{\mathrm{DPO}}(\theta) = -\mathbb{E}\log\sigma\left(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\right).$$

For gradient interpretation, this formula should not be treated as a slogan. It defines which tokens, responses, comparisons, or decisions receive gradient or operational weight. A change in masking, sampling, rubric wording, or thresholding changes the effective objective even if the model architecture is unchanged.

Examples:

• Treat gradient interpretation as part of the model contract and store the exact data version.
• Record the prompt template, role format, policy version, and decoder settings.
• Compare aligned and reference policies on both helpfulness and safety slices.
• Use held-out examples that were not used to tune refusals or rewards.
• Inspect failure cases before declaring the objective successful.

Non-examples:

• Calling a model aligned because it sounds polite on a few prompts.
• Training on refusals without measuring over-refusal on benign requests.
• Using a reward model as ground truth without calibration or adversarial checks.
• Shipping a guardrail threshold without measuring false positive and false negative rates.
• Letting feedback logs change training without provenance or consent controls.

A useful implementation pattern is to separate policy, data, and measurement. The policy says what behavior is desired. The data supplies examples, comparisons, attacks, or feedback events. The measurement checks whether the updated system moved in the intended direction without unacceptable regressions.

policy text/rubric
      |
      v
training or guardrail data  ->  objective/threshold  ->  aligned system
      |                                                   |
      v                                                   v
audit metadata                                      held-out safety eval

Worked reasoning pattern for gradient interpretation:

1. Name the target behavior in plain language.
2. Write the mathematical variable that represents it.
3. Specify which examples or comparisons estimate it.
4. Choose the optimization loss or runtime decision rule.
5. Define the regression metric that would prove the change became worse.

Three details are especially easy to miss in alignment work. First, the user intent distribution is not the same as the pretraining distribution. Second, safety labels are not ordinary class labels; they encode policy judgments that can change by context. Third, optimization pressure finds shortcuts, so every proxy must be monitored for Goodhart-style failures.

AI connection: Gradient interpretation is part of the post-training stack used by modern assistant systems. It links the base language model to human intent, safety policy, and deployment constraints without pretending that a single loss can capture all values. The goal is not perfect alignment by formula; it is a repeatable loop where evidence, objectives, and safeguards improve together.