Lesson overview | Previous part | Lesson overview

Data Mixture Optimization: Part 4: Quality and Scaling Effects to References

4. Quality and Scaling Effects

Quality and Scaling Effects gives the conceptual and mathematical layer for data mixture optimization. The local variables in this section should be read as pipeline objects: documents, records, tokens, filters, weights, shards, and manifests.

4.1 Effective tokens $D_{\mathrm{eff}}$

Effective tokens $D_{\mathrm{eff}}$ is part of the canonical scope of data mixture optimization. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For mixture, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

4.2 Repeated-data scaling

Repeated-data scaling is part of the canonical scope of data mixture optimization. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For simplex, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

4.3 Domain coverage

Domain coverage is part of the canonical scope of data mixture optimization. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record- level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For domain, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

4.4 Data age

Data age is part of the canonical scope of data mixture optimization. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For proxy model, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

4.5 Synthetic data preview

Synthetic data preview is part of the canonical scope of data mixture optimization. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For DRO, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

5. Optimization Methods

Optimization Methods gives the conceptual and mathematical layer for data mixture optimization. The local variables in this section should be read as pipeline objects: documents, records, tokens, filters, weights, shards, and manifests.

5.1 Grid search

Grid search is part of the canonical scope of data mixture optimization. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For mixture, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

5.2 Bayesian optimization preview

Bayesian optimization preview is part of the canonical scope of data mixture optimization. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For simplex, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

5.3 Proxy-model sweeps

Proxy-model sweeps is part of the canonical scope of data mixture optimization. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record- level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For domain, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

5.4 DoReMi/group DRO

DoReMi/group DRO is part of the canonical scope of data mixture optimization. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record- level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For proxy model, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

5.5 Data mixing law extrapolation

Data mixing law extrapolation is part of the canonical scope of data mixture optimization. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For DRO, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

6. Evaluation and Constraints

Evaluation and Constraints gives the conceptual and mathematical layer for data mixture optimization. The local variables in this section should be read as pipeline objects: documents, records, tokens, filters, weights, shards, and manifests.

6.1 Multi-domain validation loss

Multi-domain validation loss is part of the canonical scope of data mixture optimization. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For mixture, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

6.2 Downstream score aggregation

Downstream score aggregation is part of the canonical scope of data mixture optimization. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For simplex, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

6.3 Safety/quality constraints

Safety/quality constraints is part of the canonical scope of data mixture optimization. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For domain, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

6.4 License constraints

License constraints is part of the canonical scope of data mixture optimization. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For proxy model, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

6.5 Robustness to target mismatch

Robustness to target mismatch is part of the canonical scope of data mixture optimization. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For DRO, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

7. Common Mistakes

8. Exercises

1. (*) Build a synthetic mixture example, compute its validation signal, and explain which downstream stage would fail if the signal were wrong.
2. (*) Build a synthetic simplex example, compute its validation signal, and explain which downstream stage would fail if the signal were wrong.
3. (*) Build a synthetic domain example, compute its validation signal, and explain which downstream stage would fail if the signal were wrong.
4. (**) Build a synthetic proxy model example, compute its validation signal, and explain which downstream stage would fail if the signal were wrong.
5. (**) Build a synthetic DRO example, compute its validation signal, and explain which downstream stage would fail if the signal were wrong.
6. (**) Build a synthetic effective tokens example, compute its validation signal, and explain which downstream stage would fail if the signal were wrong.
7. (**) Build a synthetic validation loss example, compute its validation signal, and explain which downstream stage would fail if the signal were wrong.
8. (***) Build a synthetic mixture example, compute its validation signal, and explain which downstream stage would fail if the signal were wrong.
9. (***) Build a synthetic simplex example, compute its validation signal, and explain which downstream stage would fail if the signal were wrong.
10. (***) Build a synthetic domain example, compute its validation signal, and explain which downstream stage would fail if the signal were wrong.

9. Why This Matters for AI

Data pipeline quality is model quality in delayed form. The model eventually converts these records into gradients; any unresolved ambiguity becomes either wasted compute, misleading evaluation, memorization risk, or irreproducible science.

10. Conceptual Bridge

This section connects the previous and next pieces of the curriculum as follows:

raw sources -> records -> validation -> assembly -> audits -> documentation -> mixture

The next section is [Capability Benchmarks](../../17-Evaluation-and- Reliability/01-Capability-Benchmarks/notes.md). It uses the contracts established here and moves one step further through the LLM data pipeline.

References

• DataComp-LM
• FineWeb
• Dolma
• WIMBD
• Deduplicating Training Data Makes Language Models Better
• DoReMi
• Data Cards
• A Pretrainer's Guide to Training Data
• Hugging Face Datasets loading docs