Java Benchmarking with JMH

Introduction

Benchmarking Java code is notoriously difficult. A "naive" timing measurement using System.currentTimeMillis() or System.nanoTime() inside a for loop is almost always inaccurate. This is because the JVM's Just-In-Time (JIT) compiler applies heavy optimizations at runtime, such as dead code elimination and constant folding.

To solve this, the OpenJDK team developed JMH (Java Microbenchmark Harness), which is the official and industry-standard tool for building, running, and analyzing microbenchmarks on the JVM.

Core Concepts of JMH

• Warmup Iterations: The JVM takes time to "warm up" (JIT-compile bytecode into machine code). JMH runs your code multiple times before taking any measurements to ensure the JVM has reached a steady state of optimization.
• Forks: JMH runs benchmarks in separate JVM processes (forks). This ensures that results are completely isolated and not influenced by previous runs, shared state, or memory profile history.
• Blackholes (Blackhole.consume()): To prevent the JIT compiler from optimizing away "dead code" (code whose results are never used or printed), you pass your return values into a Blackhole. This tricks the JVM into believing the result is required, forcing it to actually execute the code.

Benchmark Modes

JMH allows you to configure what exactly you are measuring via the @BenchmarkMode annotation:

• Throughput: Operations per unit of time (e.g., ops/sec). Good for measuring capacity.
• AverageTime: Average time per operation. Good for measuring latency.
• SampleTime: Takes samples of execution time to understand the distribution (percentiles).
• SingleShotTime: Time for a single operation, useful for testing cold-start performance without warmup.

Code Example

The best way to start a JMH project is via the Maven archetype. Here is an example of what a standard benchmark class looks like:

import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;
import java.util.concurrent.TimeUnit;

// Measure the average time it takes to execute the method
@BenchmarkMode(Mode.AverageTime)
// Output the results in Nanoseconds
@OutputTimeUnit(TimeUnit.NANOSECONDS)
// Define state so it is shared per thread
@State(Scope.Thread)
// Configure warmup and measurement iterations (optional, defaults are usually fine)
@Warmup(iterations = 3, time = 1, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
public class StringConcatenationBenchmark {

    @Benchmark
    public void testStringPlus(Blackhole bh) {
        String a = "Hello";
        String b = "World";
        String result = a + b;
        
        // Consume the result to prevent dead code elimination
        bh.consume(result);
    }

    @Benchmark
    public void testStringBuilder(Blackhole bh) {
        StringBuilder sb = new StringBuilder();
        sb.append("Hello");
        sb.append("World");
        String result = sb.toString();
        
        bh.consume(result);
    }
}

Best Practices

1. Never Use System.nanoTime() for Microbenchmarks: The JVM's adaptive nature will invalidate these manual measurements.
2. Separate Benchmark Code: Keep your benchmark code in a separate module or project from your production application code.
3. Read the Official Samples: The creators of JMH strongly recommend reading the official JMH samples included in the OpenJDK repository. They explain dozens of potential benchmarking pitfalls.