Measuring Before Optimizing

Measuring before optimizing applies the scientific method to performance engineering: instead of guessing where a bottleneck lives or assuming an edit made code faster, you establish a reproducible benchmark baseline, profile the bottleneck to form a falsifiable hypothesis, apply one targeted change, and verify both correctness and speedup against the baseline.

The 5-Step Performance Engineering Loop:

  +---> 1. Define Metric & Save Baseline (`cargo bench -- --save-baseline base`)
  |                            |
  |                            v
  |     2. Profile Hot Path (`samply` / `perf` / `dhat` / `cargo asm`)
  |                            |
  |                            v
  |     3. Formulate Hypothesis ("Function X allocates 3 Strings per line")
  |                            |
  |                            v
  |     4. Implement Single Targeted Change (Replace `split().collect()` with iterator)
  |                            |
  |                            v
  +---- 5. Re-Measure & Verify (`cargo test` + `cargo bench -- --baseline base`)
           - Faster? Commit & record new baseline!
           - No change or slower? `git checkout .` (Revert complexity!)

Key Metrics to Measure Across Workloads

Computing Latency Percentiles (p50, p95, p99) in Rust

When benchmarking request handlers or pipelines, recording individual operation durations and sorting them to inspect p50 (median), p95, and p99 (tail latency) reveals whether an optimization improved both typical and worst-case execution:

use std::time::{Duration, Instant};

#[derive(Debug)]
pub struct LatencyDistribution {
    pub count: usize,
    pub min: Duration,
    pub p50: Duration,
    pub p95: Duration,
    pub p99: Duration,
    pub max: Duration,
}

impl LatencyDistribution {
    pub fn from_samples(mut samples: Vec<Duration>) -> Option<Self> {
        if samples.is_empty() {
            return None;
        }
        samples.sort_unstable();
        let n = samples.len();
        let idx = |pct: usize| ((n - 1) * pct) / 100;

        Some(Self {
            count: n,
            min: samples[0],
            p50: samples[idx(50)],
            p95: samples[idx(95)],
            p99: samples[idx(99)],
            max: samples[n - 1],
        })
    }
}

/// Baseline implementation: allocates a `Vec<&str>` on the heap for every log line!
pub fn sum_csv_fields_allocating(line: &str) -> u64 {
    let parts: Vec<&str> = line.split(',').collect();
    parts
        .iter()
        .filter_map(|s| s.trim().parse::<u64>().ok())
        .sum()
}

/// Optimized implementation: iterates over `line.split(',')` lazily with zero heap allocations!
pub fn sum_csv_fields_zero_alloc(line: &str) -> u64 {
    line.split(',')
        .filter_map(|s| s.trim().parse::<u64>().ok())
        .sum()
}

fn main() {
    let csv_line = "10, 20, 30, 40, 50, 60, 70, 80, 90, 100";

    // 1. Always verify equivalence of outputs first!
    assert_eq!(
        sum_csv_fields_allocating(csv_line),
        sum_csv_fields_zero_alloc(csv_line)
    );

    // 2. Collect latency samples across 200 runs
    let mut samples = Vec::with_capacity(200);
    for _ in 0..200 {
        let t0 = Instant::now();
        for _ in 0..50 {
            std::hint::black_box(sum_csv_fields_zero_alloc(std::hint::black_box(csv_line)));
        }
        samples.push(t0.elapsed());
    }

    let dist = LatencyDistribution::from_samples(samples).unwrap();
    println!(
        "Measured {} samples: min <= p50 <= p95 <= p99 <= max: {}",
        dist.count,
        dist.min <= dist.p50 && dist.p50 <= dist.p95 && dist.p95 <= dist.p99 && dist.p99 <= dist.max
    );
}

Measured 200 samples: min <= p50 <= p95 <= p99 <= max: true

Controlling Environmental Noise During Measurements

To ensure your measurements reflect real code differences rather than environmental drift:

1. Run both the baseline and candidate back-to-back (or interleaved via Criterion's BenchmarkGroup) on an idle machine connected to AC power.
2. Close heavy background applications (browsers, Docker containers, IDE background indexing) that compete for L3 cache and trigger thermal throttling.
3. Use instruction/cache counters (iai-callgrind or dhat) alongside wall-clock time when verifying small (< 3%) optimizations.