Criterium measures the computation time of an expression. It is designed to address some of the pitfalls of benchmarking, and benchmarking on the JVM in particular.
This includes:
Usage: (use 'criterium.core) (bench (Thread/sleep 1000) :verbose) (with-progress-reporting (bench (Thread/sleep 1000) :verbose)) (report-result (benchmark (Thread/sleep 1000)) :verbose) (report-result (quick-bench (Thread/sleep 1000)))
References: See http://www.ellipticgroup.com/html/benchmarkingArticle.html for a Java benchmarking library. The accompanying article describes many of the JVM benchmarking pitfalls.
See http://hackage.haskell.org/package/criterion for a Haskell benchmarking library that applies many of the same statistical techniques.
Criterium measures the computation time of an expression. It is designed to address some of the pitfalls of benchmarking, and benchmarking on the JVM in particular. This includes: - statistical processing of multiple evaluations - inclusion of a warm-up period, designed to allow the JIT compiler to optimise its code - purging of gc before testing, to isolate timings from GC state prior to testing - a final forced GC after testing to estimate impact of cleanup on the timing results Usage: (use 'criterium.core) (bench (Thread/sleep 1000) :verbose) (with-progress-reporting (bench (Thread/sleep 1000) :verbose)) (report-result (benchmark (Thread/sleep 1000)) :verbose) (report-result (quick-bench (Thread/sleep 1000))) References: See http://www.ellipticgroup.com/html/benchmarkingArticle.html for a Java benchmarking library. The accompanying article describes many of the JVM benchmarking pitfalls. See http://hackage.haskell.org/package/criterion for a Haskell benchmarking library that applies many of the same statistical techniques.
Fraction of excution time allowed for final cleanup before a warning is issued.
Fraction of excution time allowed for final cleanup before a warning is issued.
Maximum number of attempts to run finalisers and gc.
Maximum number of attempts to run finalisers and gc.
Target elapsed time for execution for a single measurement.
Target elapsed time for execution for a single measurement.
Time period used to let the code run so that jit compiler can do its work.
Time period used to let the code run so that jit compiler can do its work.
(bench expr & opts)
Convenience macro for benchmarking an expression, expr. Results are reported to out in human readable format. Options for report format are: :os, :runtime, and :verbose.
Convenience macro for benchmarking an expression, expr. Results are reported to *out* in human readable format. Options for report format are: :os, :runtime, and :verbose.
(benchmark expr options)
Benchmark an expression. This tries its best to eliminate sources of error. This also means that it runs for a while. It will typically take 70s for a fast test expression (less than 1s run time) or 10s plus 60 run times for longer running expressions.
Benchmark an expression. This tries its best to eliminate sources of error. This also means that it runs for a while. It will typically take 70s for a fast test expression (less than 1s run time) or 10s plus 60 run times for longer running expressions.
(benchmark* f
{:keys [samples warmup-jit-period target-execution-time
gc-before-sample overhead supress-jvm-option-warnings]
:as options})
Benchmark a function. This tries its best to eliminate sources of error. This also means that it runs for a while. It will typically take 70s for a fast test expression (less than 1s run time) or 10s plus 60 run times for longer running expressions.
Benchmark a function. This tries its best to eliminate sources of error. This also means that it runs for a while. It will typically take 70s for a fast test expression (less than 1s run time) or 10s plus 60 run times for longer running expressions.
(bootstrap data statistic size rng-factory)
Bootstrap a statistic. Statistic can produce multiple statistics as a vector so you can use juxt to pass multiple statistics. http://en.wikipedia.org/wiki/Bootstrapping_(statistics)
Bootstrap a statistic. Statistic can produce multiple statistics as a vector so you can use juxt to pass multiple statistics. http://en.wikipedia.org/wiki/Bootstrapping_(statistics)
(bootstrap-bca data statistic size alpha rng-factory)
Bootstrap a statistic. Statistic can produce multiple statistics as a vector so you can use juxt to pass multiple statistics. http://en.wikipedia.org/wiki/Bootstrapping_(statistics)
Bootstrap a statistic. Statistic can produce multiple statistics as a vector so you can use juxt to pass multiple statistics. http://en.wikipedia.org/wiki/Bootstrapping_(statistics)
(estimate-execution-count period f gc-before-sample estimated-fn-time)
Estimate the number of executions required in order to have at least the specified execution period, check for the jvm to have constant class loader and compilation state.
Estimate the number of executions required in order to have at least the specified execution period, check for the jvm to have constant class loader and compilation state.
(estimate-overhead)
Calculate a conservative estimate of the timing loop overhead.
Calculate a conservative estimate of the timing loop overhead.
(estimated-overhead!)
Sets the estimated overhead.
Sets the estimated overhead.
(execute-expr n f)
Time the execution of n
invocations of f
. See
execute-expr-core-timed-part
.
Time the execution of `n` invocations of `f`. See `execute-expr-core-timed-part`.
(execute-expr-core-timed-part n f)
Performs the part of execute-expr where we actually measure the elapsed run
time. Evaluates (f)
n
times, each time saving the return value as an
Object in mutable-place
.
The idea is that except for the call to (f), the only things done during each iteration are a few arithmetic operations and comparisons to 0 on primitive longs, and the storage of the return value.
The JVM is not free to optimize away the calls to f because the return values
are saved in mutable-place
.
This array is at most +max-obj-array-size+ elements long, to save memory. An artificially intelligent JVM might be able to determine that if n is larger than +max-obj-array-size+, some of the return values are overwritten and thus those calls need not be made. I doubt we will see that kind of optimization any time soon, and perhaps some JVM rules even prohibit doing so since the writes to ret-vals-arr could potentially be read by a different thread.
Performs the part of execute-expr where we actually measure the elapsed run time. Evaluates `(f)` `n` times, each time saving the return value as an Object in `mutable-place`. The idea is that except for the call to (f), the only things done during each iteration are a few arithmetic operations and comparisons to 0 on primitive longs, and the storage of the return value. The JVM is not free to optimize away the calls to f because the return values are saved in `mutable-place`. This array is at most +max-obj-array-size+ elements long, to save memory. An artificially intelligent JVM might be able to determine that if n is larger than +max-obj-array-size+, some of the return values are overwritten and thus those calls need not be made. I doubt we will see that kind of optimization any time soon, and perhaps some JVM rules even prohibit doing so since the writes to ret-vals-arr could potentially be read by a different thread.
(extract-report-options opts)
Extract reporting options from the given options vector. Returns a two element vector containing the reporting options followed by the non-reporting options
Extract reporting options from the given options vector. Returns a two element vector containing the reporting options followed by the non-reporting options
(final-gc)
Time a final clean up of JVM memory. If this time is significant compared to the runtime, then the runtime should maybe include this time.
Time a final clean up of JVM memory. If this time is significant compared to the runtime, then the runtime should maybe include this time.
(force-gc)
(force-gc max-attempts)
Force garbage collection and finalisers so that execution time associated with this is not incurred later. Up to max-attempts are made.
Force garbage collection and finalisers so that execution time associated with this is not incurred later. Up to max-attempts are made.
(heap-used)
Report a (inconsistent) snapshot of the heap memory used.
Report a (inconsistent) snapshot of the heap memory used.
(jvm-compilation-state)
Returns the total compilation time for the JVM instance.
Returns the total compilation time for the JVM instance.
(jvm-jit-name)
Returns the name of the JIT compiler.
Returns the name of the JIT compiler.
(memory)
Report a (inconsistent) snapshot of the memory situation.
Report a (inconsistent) snapshot of the memory situation.
Provides a mutable place
Provides a mutable place
(get-place _)
Get mutable field value.
Get mutable field value.
(set-place _ v)
Set mutable field to value.
Set mutable field to value.
(os-details)
Return the operating system details as a hash.
Return the operating system details as a hash.
(outlier-effect var-out-min)
Return a keyword describing the effect of outliers on the estimate of mean runtime.
Return a keyword describing the effect of outliers on the estimate of mean runtime.
(outlier-significance mean-estimate variance-estimate n)
Find the significance of outliers given boostrapped mean and variance estimates. See http://www.ellipticgroup.com/misc/article_supplement.pdf, p17.
Find the significance of outliers given boostrapped mean and variance estimates. See http://www.ellipticgroup.com/misc/article_supplement.pdf, p17.
(outliers data)
Find the outliers in the data using a boxplot technique.
Find the outliers in the data using a boxplot technique.
(quick-bench expr & opts)
Convenience macro for benchmarking an expression, expr. Results are reported to out in human readable format. Options for report format are: :os, :runtime, and :verbose.
Convenience macro for benchmarking an expression, expr. Results are reported to *out* in human readable format. Options for report format are: :os, :runtime, and :verbose.
(quick-benchmark expr options)
Benchmark an expression. Less rigorous benchmark (higher uncertainty).
Benchmark an expression. Less rigorous benchmark (higher uncertainty).
(quick-benchmark* f {:as options})
Benchmark an expression. Less rigorous benchmark (higher uncertainty).
Benchmark an expression. Less rigorous benchmark (higher uncertainty).
(replace-ret-val-in-time-body-result [elapsed-time _] new-ret-val)
(report-point-estimate msg estimate)
(report-point-estimate msg estimate quantile)
(run-benchmark sample-count
warmup-jit-period
target-execution-time
f
gc-before-sample
overhead)
Benchmark an expression. This tries its best to eliminate sources of error. This also means that it runs for a while. It will typically take 70s for a quick test expression (less than 1s run time) or 10s plus 60 run times for longer running expressions.
Benchmark an expression. This tries its best to eliminate sources of error. This also means that it runs for a while. It will typically take 70s for a quick test expression (less than 1s run time) or 10s plus 60 run times for longer running expressions.
(run-benchmarks-round-robin sample-count
warmup-jit-period
target-execution-time
exprs
gc-before-sample)
Benchmark multiple expressions in a 'round robin' fashion. Very similar to run-benchmark, except it takes multiple expressions in a sequence instead of only one (each element of the sequence should be a map with keys :f and :expr-string). It runs the following steps in sequence:
Execute each expr once
Run expression 1 for at least warmup-jit-period nanoseconds so the JIT has an opportunity to optimize it. Then do the same for each of the other expressions.
Run expression 1 many times to estimate how many times it must be executed to take a total of target-execution-time nanoseconds. The result is a number of iterations n-exec1 for expression 1. Do the same for each of the other expressions, each with the same target-execution-time, each resulting in its own independent number of executions.
Run expression 1 n-exec1 times, measuring the total elapsed time. Do the same for the rest of the expressions.
Repeat step 4 a total of sample-count times.
Benchmark multiple expressions in a 'round robin' fashion. Very similar to run-benchmark, except it takes multiple expressions in a sequence instead of only one (each element of the sequence should be a map with keys :f and :expr-string). It runs the following steps in sequence: 1. Execute each expr once 2. Run expression 1 for at least warmup-jit-period nanoseconds so the JIT has an opportunity to optimize it. Then do the same for each of the other expressions. 3. Run expression 1 many times to estimate how many times it must be executed to take a total of target-execution-time nanoseconds. The result is a number of iterations n-exec1 for expression 1. Do the same for each of the other expressions, each with the same target-execution-time, each resulting in its own independent number of executions. 4. Run expression 1 n-exec1 times, measuring the total elapsed time. Do the same for the rest of the expressions. 5. Repeat step 4 a total of sample-count times.
(runtime-details)
Return the runtime details as a hash.
Return the runtime details as a hash.
(scale-time measurement)
Determine a scale factor and unit for displaying a time.
Determine a scale factor and unit for displaying a time.
Interrogation of differences in a state.
Interrogation of differences in a state.
(state-changed? state)
Check to see if a state delta represents no change
Check to see if a state delta represents no change
(state-delta state-1 state-2)
Return a state object for the difference between two states
Return a state object for the difference between two states
(system-properties)
Return the operating system details.
Return the operating system details.
(time-body expr)
(time-body expr pre)
Returns a vector containing execution time and result of specified function.
Returns a vector containing execution time and result of specified function.
(time-body-with-jvm-state expr)
(time-body-with-jvm-state expr pre)
Returns a vector containing execution time, change in loaded and unloaded class counts, change in compilation time and result of specified function.
Returns a vector containing execution time, change in loaded and unloaded class counts, change in compilation time and result of specified function.
(timestamp-2)
Obtain a timestamp, possibly using MXBean.
Obtain a timestamp, possibly using MXBean.
(warmup-for-jit warmup-period f)
Run expression for the given amount of time to enable JIT compilation.
Run expression for the given amount of time to enable JIT compilation.
(warn-on-suspicious-jvm-options)
Warn if the JIT options are suspicious looking.
Warn if the JIT options are suspicious looking.
(with-progress-reporting expr)
Macro to enable progress reporting during the benchmark.
Macro to enable progress reporting during the benchmark.
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close