Benchmarking with JMH

A microbenchmark measures how fast a small piece of code runs, such as one method or one loop. In Java this is surprisingly hard to do correctly, because the JVM changes the code while it runs: it interprets it first, compiles it later, removes work whose result is unused, and collects garbage at unpredictable moments. JMH is the benchmarking tool, written by the OpenJDK team, that handles these problems for you. In this lesson, you will see why simple timing goes wrong, write JMH benchmarks, and read their results.

Why Simple Timing Misleads

Let's see an example that times the same work five times in a row with System.nanoTime():

public class NaiveTiming {
    static long sumOfSquares(int n) {
        long sum = 0;
        for (int i = 0; i < n; i++) {
            sum += (long) i * i;
        }
        return sum;
    }
 
    public static void main(String[] args) {
        long check = 0;
        for (int round = 1; round <= 5; round++) {
            long start = System.nanoTime();
            check += sumOfSquares(1_000_000);
            long micros = (System.nanoTime() - start) / 1000;
            System.out.println("Round " + round + ": " + micros + " µs");
        }
        System.out.println("check " + check);
    }
}

Round 1: 957 µs
Round 2: 722 µs
Round 3: 237 µs
Round 4: 231 µs
Round 5: 219 µs
check 1666664166667500000

Explanation: The same work gets about four times faster after a couple of rounds, because the JIT compiler kicks in (see "Tiered Compilation in Java"). Your numbers will differ. A single measurement would have been badly wrong, and that is only the first of several traps.

Setting Up JMH

The easiest start is the official Maven archetype, which creates a ready project:

mvn archetype:generate -DarchetypeGroupId=org.openjdk.jmh \
    -DarchetypeArtifactId=jmh-java-benchmark-archetype \
    -DgroupId=com.example -DartifactId=benchmarks -Dversion=1.0
cd benchmarks
mvn clean package
java -jar target/benchmarks.jar

In an existing project, add the jmh-core dependency and the jmh-generator-annprocess annotation processor. JMH generates extra code from your annotations at compile time, so the processor is required. Benchmark classes should be in a package.

The Main Annotations

Example: String Concatenation vs StringBuilder

This benchmark compares building a string with += in a loop against StringBuilder, for 10 and 100 pieces:

package bench;
 
import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.options.OptionsBuilder;
 
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 3, time = 1)
@Measurement(iterations = 5, time = 1)
@Fork(1)
@State(Scope.Thread)
public class ConcatBench {
    @Param({"10", "100"})
    int count;
 
    @Benchmark
    public String plusInLoop() {
        String s = "";
        for (int i = 0; i < count; i++) {
            s += i;
        }
        return s;
    }
 
    @Benchmark
    public String stringBuilder() {
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < count; i++) {
            sb.append(i);
        }
        return sb.toString();
    }
 
    public static void main(String[] args) throws Exception {
        new Runner(new OptionsBuilder()
            .include(ConcatBench.class.getSimpleName())
            .build()).run();
    }
}

While it runs, JMH prints each iteration (warmup first), and then a summary per benchmark:

# JMH version: 1.37
# VM version: JDK 25.0.5.0.3, OpenJDK 64-Bit Server VM, 25.0.5.0.3
# Warmup: 3 iterations, 1 s each
# Measurement: 5 iterations, 1 s each
# Benchmark mode: Average time, time/op
# Benchmark: bench.ConcatBench.plusInLoop
# Parameters: (count = 10)
 
# Warmup Iteration   1: 54.886 ns/op
# Warmup Iteration   2: 52.536 ns/op
# Warmup Iteration   3: 49.962 ns/op
Iteration   1: 51.051 ns/op
Iteration   2: 50.540 ns/op
Iteration   3: 50.445 ns/op
Iteration   4: 50.987 ns/op
Iteration   5: 50.023 ns/op
 
Result "bench.ConcatBench.plusInLoop":
  50.609 ±(99.9%) 1.626 ns/op [Average]

At the end comes the table that matters:

Benchmark                  (count)  Mode  Cnt    Score    Error  Units
ConcatBench.plusInLoop          10  avgt    5   50.609 ±  1.626  ns/op
ConcatBench.plusInLoop         100  avgt    5  702.731 ± 15.511  ns/op
ConcatBench.stringBuilder       10  avgt    5    6.381 ±  0.127  ns/op
ConcatBench.stringBuilder      100  avgt    5  139.047 ±  3.010  ns/op

Explanation: Each row is one benchmark with one @Param value. Score is the average time per call, Error is the 99.9% confidence margin, and Cnt is the number of measured iterations. += creates a new String every time, so it is about 5 to 8 times slower here, and the gap grows with count. Each method returns its result so the JIT can't throw the work away. Your numbers will differ by machine.

The Dead Code Trap in Action

This benchmark calls Math.log twice: once ignoring the result, once returning it.

@State(Scope.Thread)
public class DeadCodeBench {
    double x = Math.PI;
 
    @Benchmark
    public void resultIgnored() {
        Math.log(x);
    }
 
    @Benchmark
    public double resultReturned() {
        return Math.log(x);
    }
}

Benchmark                     Mode  Cnt  Score   Error  Units
DeadCodeBench.resultIgnored   avgt    5  0.245 ± 0.007  ns/op
DeadCodeBench.resultReturned  avgt    5  2.673 ± 0.252  ns/op

Explanation: 0.245 ns is less than one CPU cycle, which is impossible for a logarithm. The JIT saw that the result was unused and removed the call, so resultIgnored measures an empty method. Returning the value makes JMH consume it, and the real cost appears. When a method produces several values, pass each one to a Blackhole parameter with bh.consume(value).

Measuring Allocation with the GC Profiler

Adding .addProfiler("gc") to the options (or -prof gc on the command line) also reports bytes allocated per operation. In "JIT Optimizations in Java" this showed that a Point record passed to a Blackhole allocates 24 bytes per call, while one that stays inside the method allocates nothing, thanks to escape analysis.

Common Mistakes

Benchmarking with System.nanoTime() and a loop. You measure warmup, dead code and noise.

Ignoring the Error column. If two scores' ranges overlap, you haven't shown a difference.

Using constants as inputs. Math.log(Math.PI) written directly may be computed once at compile time. Read inputs from @State fields.

Running benchmarks on a busy laptop. Close other programs and run several forks for important results.

Drawing big conclusions from a microbenchmark. A 5 ns difference rarely matters if the request takes 50 ms.

Interview Corner

Practice

1. Easy: Change NaiveTiming to run 20 rounds and note the round after which the time stops falling.
2. Harder: Create a JMH project with the archetype and compare ArrayList.get(i) and LinkedList.get(i) for list sizes 100 and 10,000 using @Param. Explain the results using "Difference Between ArrayList and LinkedList".

Key Takeaways