criterium.core — criterium 0.4.6

Criterium measures the computation time of an expression. It is designed to address some of the pitfalls of benchmarking, and benchmarking on the JVM in particular.

This includes:

• statistical processing of multiple evaluations
• inclusion of a warm-up period, designed to allow the JIT compiler to optimise its code
• purging of gc before testing, to isolate timings from GC state prior to testing
• a final forced GC after testing to estimate impact of cleanup on the timing results

Usage: (use 'criterium.core) (bench (Thread/sleep 1000) :verbose) (with-progress-reporting (bench (Thread/sleep 1000) :verbose)) (report-result (benchmark (Thread/sleep 1000)) :verbose) (report-result (quick-bench (Thread/sleep 1000)))

References: See http://www.ellipticgroup.com/html/benchmarkingArticle.html for a Java benchmarking library. The accompanying article describes many of the JVM benchmarking pitfalls.

See http://hackage.haskell.org/package/criterion for a Haskell benchmarking library that applies many of the same statistical techniques.

*default-benchmark-opts* clj

*default-quick-bench-opts* clj

*final-gc-problem-threshold* clj

Fraction of excution time allowed for final cleanup before a warning is issued.

*max-gc-attempts* clj

Maximum number of attempts to run finalisers and gc.

*report-debug* clj

*report-progress* clj

*report-warn* clj

*sample-count* clj

Number of executions required

*target-execution-time* clj

Target elapsed time for execution for a single measurement.

*use-mxbean-for-times* clj

*warmup-jit-period* clj

Time period used to let the code run so that jit compiler can do its work.

add-default-options clj

(add-default-options options defaults)

add-outlier clj

(add-outlier low-severe low-mild high-mild high-severe counts x)

bench clj macro

(bench expr & opts)

Convenience macro for benchmarking an expression, expr. Results are reported to out in human readable format. Options for report format are: :os, :runtime, and :verbose.

benchmark clj macro

(benchmark expr options)

Benchmark an expression. This tries its best to eliminate sources of error. This also means that it runs for a while. It will typically take 70s for a fast test expression (less than 1s run time) or 10s plus 60 run times for longer running expressions.

benchmark* clj

(benchmark* f
            {:keys [samples warmup-jit-period target-execution-time
                    gc-before-sample overhead supress-jvm-option-warnings]
             :as options})

Benchmark a function. This tries its best to eliminate sources of error. This also means that it runs for a while. It will typically take 70s for a fast test expression (less than 1s run time) or 10s plus 60 run times for longer running expressions.

benchmark-round-robin clj macro

(benchmark-round-robin exprs options)

benchmark-round-robin* clj

(benchmark-round-robin* exprs options)

benchmark-stats clj

(benchmark-stats times opts)

bootstrap clj

(bootstrap data statistic size rng-factory)

Bootstrap a statistic. Statistic can produce multiple statistics as a vector so you can use juxt to pass multiple statistics. http://en.wikipedia.org/wiki/Bootstrapping_(statistics)

bootstrap-bca clj

(bootstrap-bca data statistic size alpha rng-factory)

Bootstrap a statistic. Statistic can produce multiple statistics as a vector so you can use juxt to pass multiple statistics. http://en.wikipedia.org/wiki/Bootstrapping_(statistics)

clear-cache clj

(clear-cache)

clear-cache-linux clj

(clear-cache-linux)

clear-cache-mac clj

(clear-cache-mac)

collect-samples clj

(collect-samples sample-count execution-count f gc-before-sample)

estimate-execution-count clj

(estimate-execution-count period f gc-before-sample estimated-fn-time)

Estimate the number of executions required in order to have at least the specified execution period, check for the jvm to have constant class loader and compilation state.

estimate-overhead clj

(estimate-overhead)

Calculate a conservative estimate of the timing loop overhead.

estimated-overhead clj

(estimated-overhead)

estimated-overhead! clj

(estimated-overhead!)

Sets the estimated overhead.

estimated-overhead-cache clj

execute-expr clj

(execute-expr n f)

Time the execution of n invocations of f. See execute-expr-core-timed-part.

execute-expr-core-timed-part clj

(execute-expr-core-timed-part n f)

Performs the part of execute-expr where we actually measure the elapsed run time. Evaluates (f) n times, each time saving the return value as an Object in mutable-place.

The idea is that except for the call to (f), the only things done during each iteration are a few arithmetic operations and comparisons to 0 on primitive longs, and the storage of the return value.

The JVM is not free to optimize away the calls to f because the return values are saved in mutable-place.

extract-report-options clj

(extract-report-options opts)

Extract reporting options from the given options vector. Returns a two element vector containing the reporting options followed by the non-reporting options

final-gc clj

(final-gc)

Time a final clean up of JVM memory. If this time is significant compared to the runtime, then the runtime should maybe include this time.

final-gc-warn clj

(final-gc-warn execution-time final-gc-time)

force-gc clj

(force-gc)

(force-gc max-attempts)

Force garbage collection and finalisers so that execution time associated with this is not incurred later. Up to max-attempts are made.

format-value clj

(format-value value scale unit)

heap-used clj

(heap-used)

Report a (inconsistent) snapshot of the heap memory used.

jvm-class-loader-state clj

(jvm-class-loader-state)

jvm-compilation-state clj

(jvm-compilation-state)

Returns the total compilation time for the JVM instance.

jvm-jit-name clj

(jvm-jit-name)

Returns the name of the JIT compiler.

JvmClassLoaderState clj

JvmCompilationState clj

memory clj

(memory)

Report a (inconsistent) snapshot of the memory situation.

mutable-place clj

MutablePlace clj protocol

Provides a mutable place

get-place clj

(get-place _)

Get mutable field value.

set-place clj

(set-place _ v)

Set mutable field to value.

ns-to-s clj

os-details clj

(os-details)

Return the operating system details as a hash.

outlier-count clj

(outlier-count low-severe low-mild high-mild high-severe)

outlier-effect clj

(outlier-effect var-out-min)

Return a keyword describing the effect of outliers on the estimate of mean runtime.

outlier-significance clj

(outlier-significance mean-estimate variance-estimate n)

Find the significance of outliers given boostrapped mean and variance estimates. See http://www.ellipticgroup.com/misc/article_supplement.pdf, p17.

OutlierCount clj

outliers clj

(outliers data)

Find the outliers in the data using a boxplot technique.

point-estimate clj

(point-estimate estimate)

point-estimate-ci clj

(point-estimate-ci estimate)

quick-bench clj macro

(quick-bench expr & opts)

Convenience macro for benchmarking an expression, expr. Results are reported to out in human readable format. Options for report format are: :os, :runtime, and :verbose.

quick-benchmark clj macro

(quick-benchmark expr options)

Benchmark an expression. Less rigorous benchmark (higher uncertainty).

quick-benchmark* clj

(quick-benchmark* f {:as options})

Benchmark an expression. Less rigorous benchmark (higher uncertainty).

replace-ret-val-in-time-body-result clj

(replace-ret-val-in-time-body-result [elapsed-time _] new-ret-val)

report clj

(report format-string & values)

Print format output

report-estimate clj

(report-estimate msg estimate significance)

report-estimate-sqrt clj

(report-estimate-sqrt msg estimate significance)

report-outliers clj

(report-outliers results)

report-point-estimate clj

(report-point-estimate msg estimate)

(report-point-estimate msg estimate quantile)

report-point-estimate-sqrt clj

(report-point-estimate-sqrt msg estimate)

report-result clj

(report-result results & opts)

run-benchmark clj

(run-benchmark sample-count
               warmup-jit-period
               target-execution-time
               f
               gc-before-sample
               overhead)

Benchmark an expression. This tries its best to eliminate sources of error. This also means that it runs for a while. It will typically take 70s for a quick test expression (less than 1s run time) or 10s plus 60 run times for longer running expressions.

run-benchmarks-round-robin clj

(run-benchmarks-round-robin sample-count
                            warmup-jit-period
                            target-execution-time
                            exprs
                            gc-before-sample)

Benchmark multiple expressions in a 'round robin' fashion. Very similar to run-benchmark, except it takes multiple expressions in a sequence instead of only one (each element of the sequence should be a map with keys :f and :expr-string). It runs the following steps in sequence:

1. Execute each expr once

2. Run expression 1 for at least warmup-jit-period nanoseconds so the JIT has an opportunity to optimize it. Then do the same for each of the other expressions.

3. Run expression 1 many times to estimate how many times it must be executed to take a total of target-execution-time nanoseconds. The result is a number of iterations n-exec1 for expression 1. Do the same for each of the other expressions, each with the same target-execution-time, each resulting in its own independent number of executions.

4. Run expression 1 n-exec1 times, measuring the total elapsed time. Do the same for the rest of the expressions.

5. Repeat step 4 a total of sample-count times.

runtime-details clj

(runtime-details)

Return the runtime details as a hash.

s-to-ns clj

scale-time clj

(scale-time measurement)

Determine a scale factor and unit for displaying a time.

StateChanged clj protocol

Interrogation of differences in a state.

state-changed? clj

(state-changed? state)

Check to see if a state delta represents no change

state-delta clj

(state-delta state-1 state-2)

Return a state object for the difference between two states

system-properties clj

(system-properties)

Return the operating system details.

time-body clj macro

(time-body expr)

(time-body expr pre)

Returns a vector containing execution time and result of specified function.

time-body-with-jvm-state clj macro

(time-body-with-jvm-state expr)

(time-body-with-jvm-state expr pre)

Returns a vector containing execution time, change in loaded and unloaded class counts, change in compilation time and result of specified function.

timestamp clj macro

(timestamp)

Obtain a timestamp

timestamp-2 clj

(timestamp-2)

Obtain a timestamp, possibly using MXBean.

warmup-for-jit clj

(warmup-for-jit warmup-period f)

Run expression for the given amount of time to enable JIT compilation.

warn-on-suspicious-jvm-options clj

(warn-on-suspicious-jvm-options)

Warn if the JIT options are suspicious looking.

with-progress-reporting clj macro

(with-progress-reporting expr)

Macro to enable progress reporting during the benchmark.