Benchmarking with Criterion - External Tooling

Criterion.rs (criterion crate) is the standard statistical microbenchmarking framework for stable Rust. Unlike a naive single-run timer loop that can be skewed by OS context switches, CPU turbo-boost warmup, or background processes, Criterion warms up the CPU and instruction cache, collects hundreds of multi-iteration samples, performs bootstrap statistical analysis ($p$-values and confidence intervals), and automatically compares results against the previous baseline run to detect performance regressions.

Criterion Execution Pipeline (`cargo bench`):

  1. Warmup Phase (default 3.0s)
     ---> Fills CPU L1/L2 caches & stabilizes CPU clock frequency
  2. Measurement Phase (default 100 samples over 5.0s)
     ---> Runs increasing iteration counts (N, 2N, 3N...) & fits linear regression
  3. Statistical Analysis & Baseline Comparison
     ---> Computes [lower, estimate, upper] 95% CI, outliers, & % change vs prior run
  4. Report Generation
     ---> Writes terminal summary + interactive SVG/HTML plots in `target/criterion/`

Setting Up Criterion in Cargo.toml and benches/

To configure Criterion on a package, add criterion under [dev-dependencies] and declare a [[bench]] target with harness = false so cargo bench runs Criterion's main function instead of the standard libtest runner:

# File: Cargo.toml
[dev-dependencies]
criterion = { version = "0.5", features = ["html_reports"] }

[[bench]]
name = "hash_benchmarks"
harness = false

Parameterized Benchmarks (BenchmarkId) and Throughput Reporting

Inside benches/hash_benchmarks.rs, use BenchmarkGroup, Throughput::Bytes, and BenchmarkId to compare multiple algorithms across varying input sizes (64 B, 1 KiB, 64 KiB) and measure throughput in GiB/s:

// Structure of `benches/hash_benchmarks.rs` when using the `criterion` crate:
//
// use criterion::{black_box, criterion_group, criterion_main, BenchmarkId, Criterion, Throughput};
//
// fn bench_hashes(c: &mut Criterion) {
//     let mut group = c.benchmark_group("checksum_comparison");
//     for size in [64usize, 1024, 16384] {
//         let payload = vec![0x5Au8; size];
//         group.throughput(Throughput::Bytes(size as u64));
//
//         group.bench_with_input(BenchmarkId::new("fnv1a", size), &payload, |b, data| {
//             b.iter(|| fnv1a_hash(black_box(data)));
//         });
//
//         group.bench_with_input(BenchmarkId::new("adler32", size), &payload, |b, data| {
//             b.iter(|| adler32_hash(black_box(data)));
//         });
//     }
//     group.finish();
// }
// criterion_group!(benches, bench_hashes);
// criterion_main!(benches);

Excluding Setup and Drop Overhead (iter_batched)

When a benchmarked function mutates or consumes its input by value (for example, vec.sort_unstable() or drop(tree)), calling input.clone() directly inside b.iter(|| ...) measures the cost of clone() + drop() alongside the algorithm itself!

Criterion provides b.iter_batched(setup_fn, routine_fn, BatchSize::SmallInput) to run setup and teardown outside the timed measurement window. Below is a self-contained demonstration of linear-slope timing and throughput calculation matching Criterion's model:

use std::hint::black_box;
use std::time::Instant;

pub fn fnv1a_64(bytes: &[u8]) -> u64 {
    bytes.iter().fold(0xcbf29ce484222325u64, |acc, &b| {
        (acc ^ u64::from(b)).wrapping_mul(0x100000001b3)
    })
}

fn main() {
    let payload = vec![0x42u8; 4096];

    // 1. Warmup phase (stabilize instruction & data caches)
    for _ in 0..500 {
        black_box(fnv1a_64(black_box(&payload)));
    }

    // 2. Sampled measurement phase
    let iters = 2_000u64;
    let start = Instant::now();
    for _ in 0..iters {
        black_box(fnv1a_64(black_box(&payload)));
    }
    let elapsed = start.elapsed();

    let ns_per_iter = elapsed.as_nanos() as f64 / (iters as f64);
    let total_bytes = (payload.len() as u64) * iters;
    let gib_per_sec = (total_bytes as f64 / (1024.0 * 1024.0 * 1024.0)) / elapsed.as_secs_f64();

    println!(
        "fnv1a_64 (4096B): {:.1} ns/iter ({:.2} GiB/s) > 0: {}",
        ns_per_iter,
        gib_per_sec,
        ns_per_iter > 0.0
    );
}

fnv1a_64 (4096B): 1850.4 ns/iter (2.06 GiB/s) > 0: true

Saving and Comparing Named Baselines

Before starting an optimization pass, save a named Criterion baseline so every subsequent edit can be compared directly against that reference point:

# 1. Record baseline before changing code:
cargo bench --bench hash_benchmarks -- --save-baseline main_branch

# 2. After optimizing code, compare against the saved baseline without overwriting it:
cargo bench --bench hash_benchmarks -- --baseline main_branch