What it is
Criterion is a statistics-driven benchmarking library for Rust that detects performance regressions between runs and produces detailed reports.
Define benchmarks in benches/. Criterion runs many iterations, discards warm-up, computes confidence intervals and reports change from the previous run.
Installation
cargo add --dev criterion --features html_reportsGetting started
The smallest useful thing you can do with it, and what each part means.
rust
use criterion::{black_box, criterion_group, criterion_main, Criterion};
fn bench_parse(c: &mut Criterion) {
let input = std::fs::read_to_string("fixtures/large.json").unwrap();
c.bench_function("parse_json", |b| {
// black_box stops the optimiser eliminating the work entirely.
b.iter(|| parse(black_box(&input)))
});
}
criterion_group!(benches, bench_parse);
criterion_main!(benches);Advanced usage
Where the library earns its place over a simpler alternative.
rust
fn bench_sizes(c: &mut Criterion) {
let mut group = c.benchmark_group("sort");
for size in [100, 1_000, 10_000, 100_000] {
group.throughput(Throughput::Elements(size as u64));
group.bench_with_input(BenchmarkId::new("std", size), &size, |b, &n| {
b.iter_batched(
|| random_vec(n), // setup, not timed
|mut v| v.sort_unstable(),
BatchSize::SmallInput,
)
});
group.bench_with_input(BenchmarkId::new("rayon", size), &size, |b, &n| {
b.iter_batched(|| random_vec(n), |mut v| v.par_sort_unstable(), BatchSize::SmallInput)
});
}
group.finish();
}Errors and fixes
The failures you are most likely to hit, and what actually resolves them.
- Timings are implausibly fast
- The work was optimised away. Add black_box around both the input and the returned value.
- Results vary wildly between runs
- Background load or CPU frequency scaling. Close other work, and compare relative change rather than absolute times.
Best practices
- Wrap inputs and results in black_box or the optimiser will measure nothing.
- Benchmark in release mode — cargo bench does this, but ad hoc timing often does not.
- Use iter_batched when the operation consumes or mutates its input.
- Commit a baseline and compare against it; absolute numbers vary by machine, regressions do not.
Background
Why it exists, and what it was reacting to.
Naive benchmarks are misleading: the optimiser deletes unused work, and timing noise swamps small differences. Criterion applies proper statistical methods and compares against saved baselines.