JMH-Benchmarks module ===================== This module contains benchmarks written using [JMH](https://openjdk.java.net/projects/code-tools/jmh/) from OpenJDK. Writing correct micro-benchmarks in Java (or another JVM language) is difficult and there are many non-obvious pitfalls (many due to compiler optimizations). JMH is a framework for running and analyzing benchmarks (micro or macro) written in Java (or another JVM language). * [JMH-Benchmarks module](#jmh-benchmarks-module) * [Running benchmarks](#running-benchmarks) * [Using JMH with async profiler](#using-jmh-with-async-profiler) * [Using JMH GC profiler](#using-jmh-gc-profiler) * [Using JMH Java Flight Recorder profiler](#using-jmh-java-flight-recorder-profiler) * [JMH Options](#jmh-options) * [Writing benchmarks](#writing-benchmarks) * [SolrCloud MiniCluster Benchmark Setup](#solrcloud-minicluster-benchmark-setup) * [MiniCluster Metrics](#minicluster-metrics) * [Benchmark Repeatability](#benchmark-repeatability) ## Running benchmarks If you want to set specific JMH flags or only run certain benchmarks, passing arguments via gradle tasks is cumbersome. The process has been simplified by the provided `jmh.sh` script. The default behavior is to run all benchmarks: `./jmh.sh` Pass a pattern or name after the command to select the benchmarks: `./jmh.sh CloudIndexing` Check which benchmarks match the provided pattern: `./jmh.sh -l CloudIndexing` Run a specific test and overrides the number of forks, iterations and sets warm-up iterations to `2`: `./jmh.sh -f 2 -i 2 -wi 2 CloudIndexing` Run a specific test with async and GC profilers on Linux and flame graph output: `./jmh.sh -prof gc -prof async:libPath=/path/to/libasyncProfiler.so\;output=flamegraph\;dir=profile-results CloudIndexing` ### Using JMH with async profiler It's good practice to check profiler output for micro-benchmarks in order to verify that they represent the expected application behavior and measure what you expect to measure. Some example pitfalls include the use of expensive mocks or accidental inclusion of test setup code in the benchmarked code. JMH includes [async-profiler](https://github.com/jvm-profiling-tools/async-profiler) integration that makes this easy: `./jmh.sh -prof async:libPath=/path/to/libasyncProfiler.so\;dir=profile-results` With flame graph output: `./jmh.sh -prof async:libPath=/path/to/libasyncProfiler.so\;output=flamegraph\;dir=profile-results` Simultaneous cpu, allocation and lock profiling with async profiler 2.0 and jfr output: `./jmh.sh -prof async:libPath=/path/to/libasyncProfiler.so\;output=jfr\;alloc\;lock\;dir=profile-results CloudIndexing` A number of arguments can be passed to configure async profiler, run the following for a description: `./jmh.sh -prof async:help` You can also skip specifying libPath if you place the async profiler lib in a predefined location, such as one of the locations in the env variable `LD_LIBRARY_PATH` if it has been set (many Linux distributions set this env variable, Arch by default does not), or `/usr/lib` should work. #### OS Permissions for Async Profiler Async Profiler uses perf to profile native code in addition to Java code. It will need the following for the necessary access. ```bash echo 0 > /proc/sys/kernel/kptr_restrict echo 1 > /proc/sys/kernel/perf_event_paranoid ``` or ```bash sudo sysctl -w kernel.kptr_restrict=0 sudo sysctl -w kernel.perf_event_paranoid=1 ``` ### Using JMH GC profiler You can run a benchmark with `-prof gc` to measure its allocation rate: `./jmh.sh -prof gc:dir=profile-results` Of particular importance is the `norm` alloc rates, which measure the allocations per operation rather than allocations per second. ### Using JMH Java Flight Recorder profiler JMH comes with a variety of built-in profilers. Here is an example of using JFR: `./jmh.sh -prof jfr:dir=profile-results\;configName=jfr-profile.jfc` In this example we point to the included configuration file with configName, but you could also do something like settings=default or settings=profile. ### Benchmark Outputs By default, output that benchmarks generate is created in the build/work directory. You can change this location by setting the workBaseDir system property like this: -jvmArgsAppend -DworkBaseDir=/data3/bench_work If a profiler generates output, it will generally be written to the current working directory - that is the benchmark module directory itself. You can usually change this via the dir option, for example: ./jmh.sh -prof jfr:dir=build/work/profile-results JsonFaceting ### Using a Separate MiniCluster Base Directory If you have a special case MiniCluster you have generated, such as one you have prepared with very large indexes for a search benchmark run, you can change the base directory used by the profiler for the MiniCluster with the miniClusterBaseDir system property. This is for search based benchmarks in general and the MiniCluster wil not be removed automatically by the benchmark. ### JMH Options Some common JMH options are: ```text Usage: ./jmh.sh [regexp*] [options] [opt] means optional argument. means required argument. "+" means comma-separated list of values. "time" arguments accept time suffixes, like "100ms". Command line options usually take precedence over annotations. [arguments] Benchmarks to run (regexp+). (default: .*) -bm Benchmark mode. Available modes are: [Throughput/thrpt, AverageTime/avgt, SampleTime/sample, SingleShotTime/ss, All/all]. (default: Throughput) -bs Batch size: number of benchmark method calls per operation. Some benchmark modes may ignore this setting, please check this separately. (default: 1) -e Benchmarks to exclude from the run. -f How many times to fork a single benchmark. Use 0 to disable forking altogether. Warning: disabling forking may have detrimental impact on benchmark and infrastructure reliability, you might want to use different warmup mode instead. (default: 5) -foe Should JMH fail immediately if any benchmark had experienced an unrecoverable error? This helps to make quick sanity tests for benchmark suites, as well as make the automated runs with checking error codes. (default: false) -gc Should JMH force GC between iterations? Forcing the GC may help to lower the noise in GC-heavy benchmarks, at the expense of jeopardizing GC ergonomics decisions. Use with care. (default: false) -h Display help, and exit. -i Number of measurement iterations to do. Measurement iterations are counted towards the benchmark score. (default: 1 for SingleShotTime, and 5 for all other modes) -jvm Use given JVM for runs. This option only affects forked runs. -jvmArgs Use given JVM arguments. Most options are inherited from the host VM options, but in some cases you want to pass the options only to a forked VM. Either single space-separated option line, or multiple options are accepted. This option only affects forked runs. -jvmArgsAppend Same as jvmArgs, but append these options after the already given JVM args. -jvmArgsPrepend Same as jvmArgs, but prepend these options before the already given JVM arg. -l List the benchmarks that match a filter, and exit. -lp List the benchmarks that match a filter, along with parameters, and exit. -lprof List profilers, and exit. -lrf List machine-readable result formats, and exit. -o Redirect human-readable output to a given file. -opi Override operations per invocation, see @OperationsPerInvocation Javadoc for details. (default: 1) -p Benchmark parameters. This option is expected to be used once per parameter. Parameter name and parameter values should be separated with equals sign. Parameter values should be separated with commas. -prof Use profilers to collect additional benchmark data. Some profilers are not available on all JVMs and/or all OSes. Please see the list of available profilers with -lprof. -r