Lesson overview | Previous part | Lesson overview

Adversarial Game Theory: Part 6: AI Applications to References

6. AI Applications

AI Applications develops the part of adversarial game theory specified by the approved Chapter 23 table of contents. The treatment is game-theoretic, not merely an optimization recipe.

6.1 jailbreak defenses

Jailbreak defenses belongs to the canonical scope of Adversarial Game Theory. The central object is not a single optimizer but a system of decision makers whose objectives interact.

For this subsection, the working scope is attacker-defender games, threat sets, robust optimization, Stackelberg security games, adversarial examples, and adaptive evaluation. We use players, action sets, strategies, payoffs, and response rules. The key question is whether a proposed behavior is stable when another agent adapts.

$$R_{\mathrm{rob}}(\theta)=\mathbb{E}_{(\mathbf{x},y)}\left[\max_{\boldsymbol{\delta}\in\mathcal{S}}\mathcal{L}(f_\theta(\mathbf{x}+\boldsymbol{\delta}),y)\right].$$

The formula gives the mathematical handle for jailbreak defenses. In game theory, this expression should always be read with the opponent's decision rule in mind. A policy that is optimal in isolation may be exploitable once another player observes and responds to it.

Operational definition.

Generative, evaluation, and deployment games arise when model behavior changes in response to the measurement or defense mechanism.

Worked reading.

In a GAN, the discriminator improves its classifier while the generator improves samples to fool it. In red-team evaluation, the attacker improves examples after seeing failures of the defense.

Three examples of jailbreak defenses:

1. GAN generator-discriminator training.
2. Jailbreak discovery against a deployed policy layer.
3. Benchmark gaming where systems optimize for the public metric instead of the intended task.

Two non-examples clarify the boundary:

1. One-time evaluation on a frozen hidden test set.
2. A content filter measured only against historical prompts.

Proof or verification habit for jailbreak defenses:

The mathematical proof obligation is to identify the adaptive loop and the payoff each side optimizes.

single-agent optimization:    choose theta to minimize L(theta)
game-theoretic optimization:  choose pi_i while others choose pi_-i
adversarial objective:        choose defense against best attack
multi-agent learning:         policies change the environment itself

In AI systems, jailbreak defenses is useful because modern models are deployed into adaptive environments: users learn prompt tricks, attackers search for failures, evaluators change rubrics, and other agents compete for resources.

Many LLM safety and evaluation failures are game failures: optimizing the metric changes the population of attempts.

Notebook implementation will use small synthetic payoff matrices and learning dynamics. This keeps the mathematics executable while avoiding external datasets or heavyweight game solvers.

Checklist for using jailbreak defenses responsibly:

• State the players and their objectives.
• State the action spaces and information structure.
• Decide whether the game is zero-sum, general-sum, cooperative, or adversarial.
• Identify pure, mixed, or policy strategies.
• Compute best responses or exploitability before claiming stability.
• Separate equilibrium analysis from welfare analysis.
• Explain what changes if opponents adapt.

Local diagnostic: Ask who can observe the metric, adapt to it, and benefit from adaptation.

This chapter follows Chapter 22 by adding strategic adaptation. Causal inference asks what happens when we intervene. Game theory asks what happens when other decision makers anticipate or respond to that intervention.

Modern AI makes the distinction practical. A deployed model can be optimized against by users, attackers, competitors, automated evaluators, and other models. Jailbreak defenses gives the language to reason about that pressure.

A final diagnostic question is whether a decision remains good after another agent learns from it. If not, the analysis needs game theory, not just prediction, causality, or optimization.

6.2 adversarial training

Adversarial training belongs to the canonical scope of Adversarial Game Theory. The central object is not a single optimizer but a system of decision makers whose objectives interact.

For this subsection, the working scope is attacker-defender games, threat sets, robust optimization, Stackelberg security games, adversarial examples, and adaptive evaluation. We use players, action sets, strategies, payoffs, and response rules. The key question is whether a proposed behavior is stable when another agent adapts.

$$a_A^*(a_D)\in\arg\max_{a_A}u_A(a_A,a_D).$$

The formula gives the mathematical handle for adversarial training. In game theory, this expression should always be read with the opponent's decision rule in mind. A policy that is optimal in isolation may be exploitable once another player observes and responds to it.

Operational definition.

Adversarial training and governance both treat the opponent as adaptive rather than as a fixed noise source.

Worked reading.

Training solves an approximate inner attack problem, then updates the model on those attacks. Governance designs rules and monitoring under the expectation that actors respond strategically.

Three examples of adversarial training:

1. PGD adversarial training.
2. Adaptive jailbreak evaluation.
3. Policy rules that anticipate model providers optimizing around metrics.

Two non-examples clarify the boundary:

1. A static checklist.
2. One red-team run treated as exhaustive.

Proof or verification habit for adversarial training:

The argument must connect the adaptation model to the defense or policy mechanism.

single-agent optimization:    choose theta to minimize L(theta)
game-theoretic optimization:  choose pi_i while others choose pi_-i
adversarial objective:        choose defense against best attack
multi-agent learning:         policies change the environment itself

In AI systems, adversarial training is useful because modern models are deployed into adaptive environments: users learn prompt tricks, attackers search for failures, evaluators change rubrics, and other agents compete for resources.

Robust AI governance needs game-theoretic assumptions because rules create incentives.

Notebook implementation will use small synthetic payoff matrices and learning dynamics. This keeps the mathematics executable while avoiding external datasets or heavyweight game solvers.

Checklist for using adversarial training responsibly:

• State the players and their objectives.
• State the action spaces and information structure.
• Decide whether the game is zero-sum, general-sum, cooperative, or adversarial.
• Identify pure, mixed, or policy strategies.
• Compute best responses or exploitability before claiming stability.
• Separate equilibrium analysis from welfare analysis.
• Explain what changes if opponents adapt.

Local diagnostic: Specify the adaptive opponent, not only the defense.

This chapter follows Chapter 22 by adding strategic adaptation. Causal inference asks what happens when we intervene. Game theory asks what happens when other decision makers anticipate or respond to that intervention.

Modern AI makes the distinction practical. A deployed model can be optimized against by users, attackers, competitors, automated evaluators, and other models. Adversarial training gives the language to reason about that pressure.

A final diagnostic question is whether a decision remains good after another agent learns from it. If not, the analysis needs game theory, not just prediction, causality, or optimization.

6.3 model extraction and poisoning

Model extraction and poisoning belongs to the canonical scope of Adversarial Game Theory. The central object is not a single optimizer but a system of decision makers whose objectives interact.

For this subsection, the working scope is attacker-defender games, threat sets, robust optimization, Stackelberg security games, adversarial examples, and adaptive evaluation. We use players, action sets, strategies, payoffs, and response rules. The key question is whether a proposed behavior is stable when another agent adapts.

$$\min_G\max_D \mathbb{E}_{\mathbf{x}\sim p_{\mathrm{data}}}\log D(\mathbf{x})+\mathbb{E}_{\mathbf{z}\sim p_{\mathbf{z}}}\log(1-D(G(\mathbf{z}))).$$

The formula gives the mathematical handle for model extraction and poisoning. In game theory, this expression should always be read with the opponent's decision rule in mind. A policy that is optimal in isolation may be exploitable once another player observes and responds to it.

Operational definition.

Generative, evaluation, and deployment games arise when model behavior changes in response to the measurement or defense mechanism.

Worked reading.

In a GAN, the discriminator improves its classifier while the generator improves samples to fool it. In red-team evaluation, the attacker improves examples after seeing failures of the defense.

Three examples of model extraction and poisoning:

1. GAN generator-discriminator training.
2. Jailbreak discovery against a deployed policy layer.
3. Benchmark gaming where systems optimize for the public metric instead of the intended task.

Two non-examples clarify the boundary:

1. One-time evaluation on a frozen hidden test set.
2. A content filter measured only against historical prompts.

Proof or verification habit for model extraction and poisoning:

The mathematical proof obligation is to identify the adaptive loop and the payoff each side optimizes.

single-agent optimization:    choose theta to minimize L(theta)
game-theoretic optimization:  choose pi_i while others choose pi_-i
adversarial objective:        choose defense against best attack
multi-agent learning:         policies change the environment itself

In AI systems, model extraction and poisoning is useful because modern models are deployed into adaptive environments: users learn prompt tricks, attackers search for failures, evaluators change rubrics, and other agents compete for resources.

Many LLM safety and evaluation failures are game failures: optimizing the metric changes the population of attempts.

Notebook implementation will use small synthetic payoff matrices and learning dynamics. This keeps the mathematics executable while avoiding external datasets or heavyweight game solvers.

Checklist for using model extraction and poisoning responsibly:

• State the players and their objectives.
• State the action spaces and information structure.
• Decide whether the game is zero-sum, general-sum, cooperative, or adversarial.
• Identify pure, mixed, or policy strategies.
• Compute best responses or exploitability before claiming stability.
• Separate equilibrium analysis from welfare analysis.
• Explain what changes if opponents adapt.

Local diagnostic: Ask who can observe the metric, adapt to it, and benefit from adaptation.

This chapter follows Chapter 22 by adding strategic adaptation. Causal inference asks what happens when we intervene. Game theory asks what happens when other decision makers anticipate or respond to that intervention.

Modern AI makes the distinction practical. A deployed model can be optimized against by users, attackers, competitors, automated evaluators, and other models. Model extraction and poisoning gives the language to reason about that pressure.

A final diagnostic question is whether a decision remains good after another agent learns from it. If not, the analysis needs game theory, not just prediction, causality, or optimization.

6.4 robust retrieval and tool gates

Robust retrieval and tool gates belongs to the canonical scope of Adversarial Game Theory. The central object is not a single optimizer but a system of decision makers whose objectives interact.

For this subsection, the working scope is attacker-defender games, threat sets, robust optimization, Stackelberg security games, adversarial examples, and adaptive evaluation. We use players, action sets, strategies, payoffs, and response rules. The key question is whether a proposed behavior is stable when another agent adapts.

$$a_A\in A_A,\qquad a_D\in A_D.$$

The formula gives the mathematical handle for robust retrieval and tool gates. In game theory, this expression should always be read with the opponent's decision rule in mind. A policy that is optimal in isolation may be exploitable once another player observes and responds to it.

Operational definition.

A threat model defines the attacker's allowed moves. Robust optimization then trains or evaluates against the worst allowed move.

Worked reading.

For an $\ell_\infty$ perturbation set, PGD repeatedly steps in the gradient-sign direction and projects back into the allowed box.

Three examples of robust retrieval and tool gates:

1. Image perturbations bounded by a norm.
2. Prompt transformations allowed by a jailbreak policy.
3. Retrieval poisoning constrained by an index-insertion budget.

Two non-examples clarify the boundary:

1. Any attack the modeler can imagine but has not formalized.
2. Random corruption treated as adaptive attack.

Proof or verification habit for robust retrieval and tool gates:

The nested objective is proved meaningful only after the feasible attack set is stated. The inner maximum is over that set, not over all possible bad events.

single-agent optimization:    choose theta to minimize L(theta)
game-theoretic optimization:  choose pi_i while others choose pi_-i
adversarial objective:        choose defense against best attack
multi-agent learning:         policies change the environment itself

In AI systems, robust retrieval and tool gates is useful because modern models are deployed into adaptive environments: users learn prompt tricks, attackers search for failures, evaluators change rubrics, and other agents compete for resources.

Adversarial training improves robustness to the modeled threat, not to every strategic behavior.

Notebook implementation will use small synthetic payoff matrices and learning dynamics. This keeps the mathematics executable while avoiding external datasets or heavyweight game solvers.

Checklist for using robust retrieval and tool gates responsibly:

• State the players and their objectives.
• State the action spaces and information structure.
• Decide whether the game is zero-sum, general-sum, cooperative, or adversarial.
• Identify pure, mixed, or policy strategies.
• Compute best responses or exploitability before claiming stability.
• Separate equilibrium analysis from welfare analysis.
• Explain what changes if opponents adapt.

Local diagnostic: Write the set $\mathcal{S}$ before writing the max.

This chapter follows Chapter 22 by adding strategic adaptation. Causal inference asks what happens when we intervene. Game theory asks what happens when other decision makers anticipate or respond to that intervention.

Modern AI makes the distinction practical. A deployed model can be optimized against by users, attackers, competitors, automated evaluators, and other models. Robust retrieval and tool gates gives the language to reason about that pressure.

A final diagnostic question is whether a decision remains good after another agent learns from it. If not, the analysis needs game theory, not just prediction, causality, or optimization.

6.5 governance under adaptive opponents

Governance under adaptive opponents belongs to the canonical scope of Adversarial Game Theory. The central object is not a single optimizer but a system of decision makers whose objectives interact.

For this subsection, the working scope is attacker-defender games, threat sets, robust optimization, Stackelberg security games, adversarial examples, and adaptive evaluation. We use players, action sets, strategies, payoffs, and response rules. The key question is whether a proposed behavior is stable when another agent adapts.

$$R_{\mathrm{rob}}(\theta)=\mathbb{E}_{(\mathbf{x},y)}\left[\max_{\boldsymbol{\delta}\in\mathcal{S}}\mathcal{L}(f_\theta(\mathbf{x}+\boldsymbol{\delta}),y)\right].$$

The formula gives the mathematical handle for governance under adaptive opponents. In game theory, this expression should always be read with the opponent's decision rule in mind. A policy that is optimal in isolation may be exploitable once another player observes and responds to it.

Operational definition.

Adversarial training and governance both treat the opponent as adaptive rather than as a fixed noise source.

Worked reading.

Training solves an approximate inner attack problem, then updates the model on those attacks. Governance designs rules and monitoring under the expectation that actors respond strategically.

Three examples of governance under adaptive opponents:

1. PGD adversarial training.
2. Adaptive jailbreak evaluation.
3. Policy rules that anticipate model providers optimizing around metrics.

Two non-examples clarify the boundary:

1. A static checklist.
2. One red-team run treated as exhaustive.

Proof or verification habit for governance under adaptive opponents:

The argument must connect the adaptation model to the defense or policy mechanism.

single-agent optimization:    choose theta to minimize L(theta)
game-theoretic optimization:  choose pi_i while others choose pi_-i
adversarial objective:        choose defense against best attack
multi-agent learning:         policies change the environment itself

In AI systems, governance under adaptive opponents is useful because modern models are deployed into adaptive environments: users learn prompt tricks, attackers search for failures, evaluators change rubrics, and other agents compete for resources.

Robust AI governance needs game-theoretic assumptions because rules create incentives.

Notebook implementation will use small synthetic payoff matrices and learning dynamics. This keeps the mathematics executable while avoiding external datasets or heavyweight game solvers.

Checklist for using governance under adaptive opponents responsibly:

• State the players and their objectives.
• State the action spaces and information structure.
• Decide whether the game is zero-sum, general-sum, cooperative, or adversarial.
• Identify pure, mixed, or policy strategies.
• Compute best responses or exploitability before claiming stability.
• Separate equilibrium analysis from welfare analysis.
• Explain what changes if opponents adapt.

Local diagnostic: Specify the adaptive opponent, not only the defense.

This chapter follows Chapter 22 by adding strategic adaptation. Causal inference asks what happens when we intervene. Game theory asks what happens when other decision makers anticipate or respond to that intervention.

Modern AI makes the distinction practical. A deployed model can be optimized against by users, attackers, competitors, automated evaluators, and other models. Governance under adaptive opponents gives the language to reason about that pressure.

A final diagnostic question is whether a decision remains good after another agent learns from it. If not, the analysis needs game theory, not just prediction, causality, or optimization.

7. Common Mistakes

8. Exercises

1. (*) Work through a game-theory task for adversarial game theory.

   • (a) State the players, actions, and payoffs.
   • (b) Compute or characterize best responses.
   • (c) Decide whether the proposed joint strategy is stable.
   • (d) Interpret the result for an AI, LLM, or adversarial system.

2. (*) Work through a game-theory task for adversarial game theory.

   • (a) State the players, actions, and payoffs.
   • (b) Compute or characterize best responses.
   • (c) Decide whether the proposed joint strategy is stable.
   • (d) Interpret the result for an AI, LLM, or adversarial system.

3. (*) Work through a game-theory task for adversarial game theory.

   • (a) State the players, actions, and payoffs.
   • (b) Compute or characterize best responses.
   • (c) Decide whether the proposed joint strategy is stable.
   • (d) Interpret the result for an AI, LLM, or adversarial system.

4. (**) Work through a game-theory task for adversarial game theory.

   • (a) State the players, actions, and payoffs.
   • (b) Compute or characterize best responses.
   • (c) Decide whether the proposed joint strategy is stable.
   • (d) Interpret the result for an AI, LLM, or adversarial system.

5. (**) Work through a game-theory task for adversarial game theory.

   • (a) State the players, actions, and payoffs.
   • (b) Compute or characterize best responses.
   • (c) Decide whether the proposed joint strategy is stable.
   • (d) Interpret the result for an AI, LLM, or adversarial system.

6. (**) Work through a game-theory task for adversarial game theory.

   • (a) State the players, actions, and payoffs.
   • (b) Compute or characterize best responses.
   • (c) Decide whether the proposed joint strategy is stable.
   • (d) Interpret the result for an AI, LLM, or adversarial system.

7. (***) Work through a game-theory task for adversarial game theory.

   • (a) State the players, actions, and payoffs.
   • (b) Compute or characterize best responses.
   • (c) Decide whether the proposed joint strategy is stable.
   • (d) Interpret the result for an AI, LLM, or adversarial system.

8. (***) Work through a game-theory task for adversarial game theory.

   • (a) State the players, actions, and payoffs.
   • (b) Compute or characterize best responses.
   • (c) Decide whether the proposed joint strategy is stable.
   • (d) Interpret the result for an AI, LLM, or adversarial system.

9. (***) Work through a game-theory task for adversarial game theory.

   • (a) State the players, actions, and payoffs.
   • (b) Compute or characterize best responses.
   • (c) Decide whether the proposed joint strategy is stable.
   • (d) Interpret the result for an AI, LLM, or adversarial system.

10. (***) Work through a game-theory task for adversarial game theory.

• (a) State the players, actions, and payoffs.
• (b) Compute or characterize best responses.
• (c) Decide whether the proposed joint strategy is stable.
• (d) Interpret the result for an AI, LLM, or adversarial system.

9. Why This Matters for AI

10. Conceptual Bridge

Adversarial Game Theory follows causal inference because interventions often change incentives. Chapter 22 asks what changes when an action is taken. Chapter 23 asks what happens when other agents see that action, learn from it, and respond strategically.

The backward bridge is intervention. A policy change can have a causal effect, but if users or attackers adapt, the effect becomes part of a game. The forward bridge is measure theory: later probability foundations make the stochastic strategies, repeated games, and distributional assumptions more rigorous.

+--------------------------------------------------------------+
| Chapter 22: intervention and causal mechanisms               |
| Chapter 23: strategic adaptation and adversarial objectives   |
| Chapter 24: rigorous probability and measure foundations      |
+--------------------------------------------------------------+

References

• Madry et al.. Towards Deep Learning Models Resistant to Adversarial Attacks. https://arxiv.org/abs/1706.06083
• Goodfellow et al.. Generative Adversarial Nets. https://arxiv.org/abs/1406.2661
• Nisan et al.. Algorithmic Game Theory. https://doi.org/10.1017/CBO9780511800481
• Brown and Sandholm. Superhuman AI for multiplayer poker. https://www.science.org/doi/10.1126/science.aay2400