Lesson overview | Previous part | Lesson overview

Error Analysis and Ablations: Part 7: Regression-Test Workflow to References

7. Regression-Test Workflow

Regression-Test Workflow is the part of error analysis and ablations that turns the approved TOC into a concrete learning path. The subsections below keep the focus on Chapter 17's canonical job: measurement, reliability, uncertainty, and decision support for AI systems.

7.1 Converting failures into eval cases

Converting failures into eval cases is part of the canonical scope of error analysis and ablations. In this chapter, the object under study is not merely a dataset or a model, but the full failure slice and causal comparison: the items, prompts, outputs, graders, uncertainty statements, and decision rules that turn model behavior into evidence.

The basic mathematical pattern is an empirical estimator. For a model or system $m$ evaluated on items $z_1,\ldots,z_n$, the local estimate is written

$$\Delta_{\mathrm{ablate}} = \frac{1}{n}\sum_{i=1}^n \mathbb{1}\{e_i \in E\}.$$

The formula is intentionally simple. The difficulty lies in deciding what counts as an item, which loss or score is meaningful, whether the items are independent, and whether the estimate answers the real product or research question. For converting failures into eval cases, those choices determine whether the reported number is evidence or decoration.

A useful invariant is that every evaluation claim should be reproducible as a tuple $(m,\mathcal{T},\pi,g,\rho)$, where $m$ is the system, $\mathcal{T}$ is the task sample, $\pi$ is the prompt or intervention policy, $g$ is the grader, and $\rho$ is the aggregation rule. If any part of this tuple is missing, the number cannot be audited.

Examples of correct use:

• Report converting failures into eval cases with item count, prompt protocol, grader version, and a confidence interval.
• Use paired comparisons when two models answer the same evaluation items.
• Inspect at least one meaningful slice before concluding that the aggregate result is reliable.
• Store raw outputs so future graders can be replayed without querying the model again.
• Document whether the metric is measuring capability, reliability, user value, or risk.

Non-examples:

• A leaderboard point estimate without sample size.
• A benchmark score produced with an undocumented prompt template.
• A model-graded result without judge identity, rubric, or agreement check.
• A robustness claim measured only on the easiest in-distribution examples.
• An online win declared before the randomization and logging checks pass.

Worked evaluation pattern for converting failures into eval cases:

1. Define the evaluation population in words before writing code.
2. Choose the smallest metric set that answers the decision question.
3. Compute the point estimate and an uncertainty statement together.
4. Run a slice or paired analysis to check whether the aggregate hides structure.
5. Archive raw outputs, scores, and seeds before changing the prompt or grader.

For AI systems, converting failures into eval cases is especially delicate because the same model can be used with many prompts, decoding policies, tools, retrieval contexts, and safety filters. The measured quantity is therefore a property of the system configuration, not just the base weights.

Implementation checklist:

• Use deterministic seeds for synthetic or sampled evaluation subsets.
• Print metric denominators, not only percentages.
• Keep missing, invalid, timeout, and refusal outcomes explicit.
• Prefer typed result records over loose CSV columns.
• Separate raw model outputs from normalized grader inputs.
• Track the smallest reproducible command that generated the result.
• Record whether the estimate is item-weighted, token-weighted, user-weighted, or domain-weighted.
• Write the decision rule before seeing the final score whenever the result will guide a release.

The mathematical habit to build is skepticism with structure. A score is not ignored because it is noisy; it is interpreted through the design that produced it. Converting failures into eval cases is one place where that habit becomes concrete.

7.2 Versioning error suites

Versioning error suites is part of the canonical scope of error analysis and ablations. In this chapter, the object under study is not merely a dataset or a model, but the full failure slice and causal comparison: the items, prompts, outputs, graders, uncertainty statements, and decision rules that turn model behavior into evidence.

The basic mathematical pattern is an empirical estimator. For a model or system $m$ evaluated on items $z_1,\ldots,z_n$, the local estimate is written

$$\Delta_{\mathrm{ablate}} = \frac{1}{n}\sum_{i=1}^n \mathbb{1}\{e_i \in E\}.$$

The formula is intentionally simple. The difficulty lies in deciding what counts as an item, which loss or score is meaningful, whether the items are independent, and whether the estimate answers the real product or research question. For versioning error suites, those choices determine whether the reported number is evidence or decoration.

A useful invariant is that every evaluation claim should be reproducible as a tuple $(m,\mathcal{T},\pi,g,\rho)$, where $m$ is the system, $\mathcal{T}$ is the task sample, $\pi$ is the prompt or intervention policy, $g$ is the grader, and $\rho$ is the aggregation rule. If any part of this tuple is missing, the number cannot be audited.

Examples of correct use:

• Report versioning error suites with item count, prompt protocol, grader version, and a confidence interval.
• Use paired comparisons when two models answer the same evaluation items.
• Inspect at least one meaningful slice before concluding that the aggregate result is reliable.
• Store raw outputs so future graders can be replayed without querying the model again.
• Document whether the metric is measuring capability, reliability, user value, or risk.

Non-examples:

• A leaderboard point estimate without sample size.
• A benchmark score produced with an undocumented prompt template.
• A model-graded result without judge identity, rubric, or agreement check.
• A robustness claim measured only on the easiest in-distribution examples.
• An online win declared before the randomization and logging checks pass.

Worked evaluation pattern for versioning error suites:

1. Define the evaluation population in words before writing code.
2. Choose the smallest metric set that answers the decision question.
3. Compute the point estimate and an uncertainty statement together.
4. Run a slice or paired analysis to check whether the aggregate hides structure.
5. Archive raw outputs, scores, and seeds before changing the prompt or grader.

For AI systems, versioning error suites is especially delicate because the same model can be used with many prompts, decoding policies, tools, retrieval contexts, and safety filters. The measured quantity is therefore a property of the system configuration, not just the base weights.

Implementation checklist:

• Use deterministic seeds for synthetic or sampled evaluation subsets.
• Print metric denominators, not only percentages.
• Keep missing, invalid, timeout, and refusal outcomes explicit.
• Prefer typed result records over loose CSV columns.
• Separate raw model outputs from normalized grader inputs.
• Track the smallest reproducible command that generated the result.
• Record whether the estimate is item-weighted, token-weighted, user-weighted, or domain-weighted.
• Write the decision rule before seeing the final score whenever the result will guide a release.

The mathematical habit to build is skepticism with structure. A score is not ignored because it is noisy; it is interpreted through the design that produced it. Versioning error suites is one place where that habit becomes concrete.

7.3 Avoiding overfit fixes

Avoiding overfit fixes is part of the canonical scope of error analysis and ablations. In this chapter, the object under study is not merely a dataset or a model, but the full failure slice and causal comparison: the items, prompts, outputs, graders, uncertainty statements, and decision rules that turn model behavior into evidence.

The basic mathematical pattern is an empirical estimator. For a model or system $m$ evaluated on items $z_1,\ldots,z_n$, the local estimate is written

$$\Delta_{\mathrm{ablate}} = \frac{1}{n}\sum_{i=1}^n \mathbb{1}\{e_i \in E\}.$$

The formula is intentionally simple. The difficulty lies in deciding what counts as an item, which loss or score is meaningful, whether the items are independent, and whether the estimate answers the real product or research question. For avoiding overfit fixes, those choices determine whether the reported number is evidence or decoration.

A useful invariant is that every evaluation claim should be reproducible as a tuple $(m,\mathcal{T},\pi,g,\rho)$, where $m$ is the system, $\mathcal{T}$ is the task sample, $\pi$ is the prompt or intervention policy, $g$ is the grader, and $\rho$ is the aggregation rule. If any part of this tuple is missing, the number cannot be audited.

Examples of correct use:

• Report avoiding overfit fixes with item count, prompt protocol, grader version, and a confidence interval.
• Use paired comparisons when two models answer the same evaluation items.
• Inspect at least one meaningful slice before concluding that the aggregate result is reliable.
• Store raw outputs so future graders can be replayed without querying the model again.
• Document whether the metric is measuring capability, reliability, user value, or risk.

Non-examples:

• A leaderboard point estimate without sample size.
• A benchmark score produced with an undocumented prompt template.
• A model-graded result without judge identity, rubric, or agreement check.
• A robustness claim measured only on the easiest in-distribution examples.
• An online win declared before the randomization and logging checks pass.

Worked evaluation pattern for avoiding overfit fixes:

1. Define the evaluation population in words before writing code.
2. Choose the smallest metric set that answers the decision question.
3. Compute the point estimate and an uncertainty statement together.
4. Run a slice or paired analysis to check whether the aggregate hides structure.
5. Archive raw outputs, scores, and seeds before changing the prompt or grader.

For AI systems, avoiding overfit fixes is especially delicate because the same model can be used with many prompts, decoding policies, tools, retrieval contexts, and safety filters. The measured quantity is therefore a property of the system configuration, not just the base weights.

Implementation checklist:

• Use deterministic seeds for synthetic or sampled evaluation subsets.
• Print metric denominators, not only percentages.
• Keep missing, invalid, timeout, and refusal outcomes explicit.
• Prefer typed result records over loose CSV columns.
• Separate raw model outputs from normalized grader inputs.
• Track the smallest reproducible command that generated the result.
• Record whether the estimate is item-weighted, token-weighted, user-weighted, or domain-weighted.
• Write the decision rule before seeing the final score whenever the result will guide a release.

The mathematical habit to build is skepticism with structure. A score is not ignored because it is noisy; it is interpreted through the design that produced it. Avoiding overfit fixes is one place where that habit becomes concrete.

7.4 Budgeting regression tests

Budgeting regression tests is part of the canonical scope of error analysis and ablations. In this chapter, the object under study is not merely a dataset or a model, but the full failure slice and causal comparison: the items, prompts, outputs, graders, uncertainty statements, and decision rules that turn model behavior into evidence.

The basic mathematical pattern is an empirical estimator. For a model or system $m$ evaluated on items $z_1,\ldots,z_n$, the local estimate is written

$$\Delta_{\mathrm{ablate}} = \frac{1}{n}\sum_{i=1}^n \mathbb{1}\{e_i \in E\}.$$

The formula is intentionally simple. The difficulty lies in deciding what counts as an item, which loss or score is meaningful, whether the items are independent, and whether the estimate answers the real product or research question. For budgeting regression tests, those choices determine whether the reported number is evidence or decoration.

A useful invariant is that every evaluation claim should be reproducible as a tuple $(m,\mathcal{T},\pi,g,\rho)$, where $m$ is the system, $\mathcal{T}$ is the task sample, $\pi$ is the prompt or intervention policy, $g$ is the grader, and $\rho$ is the aggregation rule. If any part of this tuple is missing, the number cannot be audited.

Examples of correct use:

• Report budgeting regression tests with item count, prompt protocol, grader version, and a confidence interval.
• Use paired comparisons when two models answer the same evaluation items.
• Inspect at least one meaningful slice before concluding that the aggregate result is reliable.
• Store raw outputs so future graders can be replayed without querying the model again.
• Document whether the metric is measuring capability, reliability, user value, or risk.

Non-examples:

• A leaderboard point estimate without sample size.
• A benchmark score produced with an undocumented prompt template.
• A model-graded result without judge identity, rubric, or agreement check.
• A robustness claim measured only on the easiest in-distribution examples.
• An online win declared before the randomization and logging checks pass.

Worked evaluation pattern for budgeting regression tests:

1. Define the evaluation population in words before writing code.
2. Choose the smallest metric set that answers the decision question.
3. Compute the point estimate and an uncertainty statement together.
4. Run a slice or paired analysis to check whether the aggregate hides structure.
5. Archive raw outputs, scores, and seeds before changing the prompt or grader.

For AI systems, budgeting regression tests is especially delicate because the same model can be used with many prompts, decoding policies, tools, retrieval contexts, and safety filters. The measured quantity is therefore a property of the system configuration, not just the base weights.

Implementation checklist:

• Use deterministic seeds for synthetic or sampled evaluation subsets.
• Print metric denominators, not only percentages.
• Keep missing, invalid, timeout, and refusal outcomes explicit.
• Prefer typed result records over loose CSV columns.
• Separate raw model outputs from normalized grader inputs.
• Track the smallest reproducible command that generated the result.
• Record whether the estimate is item-weighted, token-weighted, user-weighted, or domain-weighted.
• Write the decision rule before seeing the final score whenever the result will guide a release.

The mathematical habit to build is skepticism with structure. A score is not ignored because it is noisy; it is interpreted through the design that produced it. Budgeting regression tests is one place where that habit becomes concrete.

7.5 Closing the debug loop

Closing the debug loop is part of the canonical scope of error analysis and ablations. In this chapter, the object under study is not merely a dataset or a model, but the full failure slice and causal comparison: the items, prompts, outputs, graders, uncertainty statements, and decision rules that turn model behavior into evidence.

The basic mathematical pattern is an empirical estimator. For a model or system $m$ evaluated on items $z_1,\ldots,z_n$, the local estimate is written

$$\Delta_{\mathrm{ablate}} = \frac{1}{n}\sum_{i=1}^n \mathbb{1}\{e_i \in E\}.$$

The formula is intentionally simple. The difficulty lies in deciding what counts as an item, which loss or score is meaningful, whether the items are independent, and whether the estimate answers the real product or research question. For closing the debug loop, those choices determine whether the reported number is evidence or decoration.

A useful invariant is that every evaluation claim should be reproducible as a tuple $(m,\mathcal{T},\pi,g,\rho)$, where $m$ is the system, $\mathcal{T}$ is the task sample, $\pi$ is the prompt or intervention policy, $g$ is the grader, and $\rho$ is the aggregation rule. If any part of this tuple is missing, the number cannot be audited.

Examples of correct use:

• Report closing the debug loop with item count, prompt protocol, grader version, and a confidence interval.
• Use paired comparisons when two models answer the same evaluation items.
• Inspect at least one meaningful slice before concluding that the aggregate result is reliable.
• Store raw outputs so future graders can be replayed without querying the model again.
• Document whether the metric is measuring capability, reliability, user value, or risk.

Non-examples:

• A leaderboard point estimate without sample size.
• A benchmark score produced with an undocumented prompt template.
• A model-graded result without judge identity, rubric, or agreement check.
• A robustness claim measured only on the easiest in-distribution examples.
• An online win declared before the randomization and logging checks pass.

Worked evaluation pattern for closing the debug loop:

1. Define the evaluation population in words before writing code.
2. Choose the smallest metric set that answers the decision question.
3. Compute the point estimate and an uncertainty statement together.
4. Run a slice or paired analysis to check whether the aggregate hides structure.
5. Archive raw outputs, scores, and seeds before changing the prompt or grader.

For AI systems, closing the debug loop is especially delicate because the same model can be used with many prompts, decoding policies, tools, retrieval contexts, and safety filters. The measured quantity is therefore a property of the system configuration, not just the base weights.

Implementation checklist:

• Use deterministic seeds for synthetic or sampled evaluation subsets.
• Print metric denominators, not only percentages.
• Keep missing, invalid, timeout, and refusal outcomes explicit.
• Prefer typed result records over loose CSV columns.
• Separate raw model outputs from normalized grader inputs.
• Track the smallest reproducible command that generated the result.
• Record whether the estimate is item-weighted, token-weighted, user-weighted, or domain-weighted.
• Write the decision rule before seeing the final score whenever the result will guide a release.

The mathematical habit to build is skepticism with structure. A score is not ignored because it is noisy; it is interpreted through the design that produced it. Closing the debug loop is one place where that habit becomes concrete.

8. Common Mistakes

9. Exercises

1. (*) Aggregate metrics hide failure modes. (a) Define the relevant evaluation object. (b) Write the estimator in LaTeX notation. (c) Give one example where the estimator is reliable. (d) Give one example where the same number would be misleading. (e) Describe what the theory notebook should verify computationally.

2. (*) Failures as structured data. (a) Define the relevant evaluation object. (b) Write the estimator in LaTeX notation. (c) Give one example where the estimator is reliable. (d) Give one example where the same number would be misleading. (e) Describe what the theory notebook should verify computationally.

3. (*) Ablations as causal probes. (a) Define the relevant evaluation object. (b) Write the estimator in LaTeX notation. (c) Give one example where the estimator is reliable. (d) Give one example where the same number would be misleading. (e) Describe what the theory notebook should verify computationally.

4. (**) Debugging without benchmark overfitting. (a) Define the relevant evaluation object. (b) Write the estimator in LaTeX notation. (c) Give one example where the estimator is reliable. (d) Give one example where the same number would be misleading. (e) Describe what the theory notebook should verify computationally.

5. (**) From one bug to a regression suite. (a) Define the relevant evaluation object. (b) Write the estimator in LaTeX notation. (c) Give one example where the estimator is reliable. (d) Give one example where the same number would be misleading. (e) Describe what the theory notebook should verify computationally.

6. (**) Error set. (a) Define the relevant evaluation object. (b) Write the estimator in LaTeX notation. (c) Give one example where the estimator is reliable. (d) Give one example where the same number would be misleading. (e) Describe what the theory notebook should verify computationally.

7. (***) Confusion matrix. (a) Define the relevant evaluation object. (b) Write the estimator in LaTeX notation. (c) Give one example where the estimator is reliable. (d) Give one example where the same number would be misleading. (e) Describe what the theory notebook should verify computationally.

8. (***) Slice and subgroup. (a) Define the relevant evaluation object. (b) Write the estimator in LaTeX notation. (c) Give one example where the estimator is reliable. (d) Give one example where the same number would be misleading. (e) Describe what the theory notebook should verify computationally.

9. (***) Counterfactual example. (a) Define the relevant evaluation object. (b) Write the estimator in LaTeX notation. (c) Give one example where the estimator is reliable. (d) Give one example where the same number would be misleading. (e) Describe what the theory notebook should verify computationally.

10. (***) Ablation effect and interaction. (a) Define the relevant evaluation object. (b) Write the estimator in LaTeX notation. (c) Give one example where the estimator is reliable. (d) Give one example where the same number would be misleading. (e) Describe what the theory notebook should verify computationally.

10. Why This Matters for AI

11. Conceptual Bridge

This section sits after the training-data pipeline because evaluation depends on clean holdouts, contamination audits, and well-documented data provenance. It does not repeat those pipeline mechanics; it consumes their outputs as the basis for credible measurement.

It also sits before alignment and production chapters. Alignment asks how to shape model behavior with supervised data, preferences, policies, and feedback. Production MLOps asks how deployed systems are observed and maintained over time. Error Analysis and Ablations supplies the measurement discipline both chapters need.

The recurring mathematical pattern is empirical risk with uncertainty. Whether the object is a benchmark item, a calibrated probability, a shifted subgroup, an ablation comparison, or an online treatment effect, the learner should ask: what distribution generated this evidence, what estimator did we compute, and what decision is justified by the uncertainty?

16 Data Pipeline
    -> clean eval data, manifests, decontamination
17 Evaluation and Reliability
    -> benchmarks, calibration, robustness, ablations, online tests
18 Alignment and Safety
    -> SFT, preferences, policies, human feedback
19 Production ML and MLOps
    -> monitoring, serving, retraining, observability

References

• Hypothesis Testing
• HELM
• OpenAI Evals
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Trustworthy Online Controlled Experiments
• Concentration Inequalities
• Estimation Theory
• Hypothesis Testing