Liking cljdoc? Tell your friends :D

criterium.core

Criterium measures the computation time of an expression. It is designed to address some of the pitfalls of benchmarking, and benchmarking on the JVM in particular.

This includes:

  • statistical processing of multiple evaluations
  • inclusion of a warm-up period, designed to allow the JIT compiler to optimise its code
  • purging of gc before testing, to isolate timings from GC state prior to testing
  • a final forced GC after testing to estimate impact of cleanup on the timing results

Usage: (use 'criterium.core) (bench (Thread/sleep 1000) :verbose) (with-progress-reporting (bench (Thread/sleep 1000) :verbose)) (report-result (benchmark (Thread/sleep 1000)) :verbose) (report-result (quick-bench (Thread/sleep 1000)))

References: See http://www.ellipticgroup.com/html/benchmarkingArticle.html for a Java benchmarking library. The accompanying article describes many of the JVM benchmarking pitfalls.

See http://hackage.haskell.org/package/criterion for a Haskell benchmarking library that applies many of the same statistical techniques.

Criterium measures the computation time of an expression.  It is
designed to address some of the pitfalls of benchmarking, and benchmarking on
the JVM in particular.

This includes:
  - statistical processing of multiple evaluations
  - inclusion of a warm-up period, designed to allow the JIT compiler to
    optimise its code
  - purging of gc before testing, to isolate timings from GC state prior
    to testing
  - a final forced GC after testing to estimate impact of cleanup on the
    timing results

Usage:
  (use 'criterium.core)
  (bench (Thread/sleep 1000) :verbose)
  (with-progress-reporting (bench (Thread/sleep 1000) :verbose))
  (report-result (benchmark (Thread/sleep 1000)) :verbose)
  (report-result (quick-bench (Thread/sleep 1000)))

References:
See http://www.ellipticgroup.com/html/benchmarkingArticle.html for a Java
benchmarking library.  The accompanying article describes many of the JVM
benchmarking pitfalls.

See http://hackage.haskell.org/package/criterion for a Haskell benchmarking
library that applies many of the same statistical techniques.

*default-benchmark-opts*

source

*default-quick-bench-opts*

source

*final-gc-problem-threshold*

Fraction of excution time allowed for final cleanup before a warning is issued.

Fraction of excution time allowed for final cleanup before a
warning is issued.
sourceraw docstring

*max-gc-attempts*

Maximum number of attempts to run finalisers and gc.

Maximum number of attempts to run finalisers and gc.
sourceraw docstring

*report-debug*

source

*report-progress*

source

*report-warn*

source

*sample-count*

Number of executions required

Number of executions required
sourceraw docstring

*target-execution-time*

Target elapsed time for execution for a single measurement.

Target elapsed time for execution for a single measurement.
sourceraw docstring

*use-mxbean-for-times*

source

*warmup-jit-period*

Time period used to let the code run so that jit compiler can do its work.

Time period used to let the code run so that jit compiler can do
its work.
sourceraw docstring

add-default-options

(add-default-options options defaults)
source

add-outlier

(add-outlier low-severe low-mild high-mild high-severe counts x)
source

benchmacro

(bench expr & opts)

Convenience macro for benchmarking an expression, expr. Results are reported to out in human readable format. Options for report format are: :os, :runtime, and :verbose.

Convenience macro for benchmarking an expression, expr.  Results are reported
  to *out* in human readable format. Options for report format are: :os,
:runtime, and :verbose.
sourceraw docstring

benchmarkmacro

(benchmark expr options)

Benchmark an expression. This tries its best to eliminate sources of error. This also means that it runs for a while. It will typically take 70s for a fast test expression (less than 1s run time) or 10s plus 60 run times for longer running expressions.

Benchmark an expression. This tries its best to eliminate sources of error.
This also means that it runs for a while.  It will typically take 70s for a
fast test expression (less than 1s run time) or 10s plus 60 run times for
longer running expressions.
sourceraw docstring

benchmark*

(benchmark* f
            {:keys [samples warmup-jit-period target-execution-time
                    gc-before-sample overhead
                    supress-jvm-option-warnings],
             :as options})

Benchmark a function. This tries its best to eliminate sources of error. This also means that it runs for a while. It will typically take 70s for a fast test expression (less than 1s run time) or 10s plus 60 run times for longer running expressions.

Benchmark a function. This tries its best to eliminate sources of error.
This also means that it runs for a while.  It will typically take 70s for a
fast test expression (less than 1s run time) or 10s plus 60 run times for
longer running expressions.
sourceraw docstring

benchmark-round-robinmacro

(benchmark-round-robin exprs options)
source

benchmark-round-robin*

(benchmark-round-robin* exprs options)
source

benchmark-stats

(benchmark-stats times opts)
source

bootstrap

(bootstrap data statistic size rng-factory)

Bootstrap a statistic. Statistic can produce multiple statistics as a vector so you can use juxt to pass multiple statistics. http://en.wikipedia.org/wiki/Bootstrapping_(statistics)

Bootstrap a statistic. Statistic can produce multiple statistics as a vector
so you can use juxt to pass multiple statistics.
http://en.wikipedia.org/wiki/Bootstrapping_(statistics)
sourceraw docstring

bootstrap-bca

(bootstrap-bca data statistic size alpha rng-factory)

Bootstrap a statistic. Statistic can produce multiple statistics as a vector so you can use juxt to pass multiple statistics. http://en.wikipedia.org/wiki/Bootstrapping_(statistics)

Bootstrap a statistic. Statistic can produce multiple statistics as a vector
so you can use juxt to pass multiple statistics.
http://en.wikipedia.org/wiki/Bootstrapping_(statistics)
sourceraw docstring

clear-cache

(clear-cache)
source

clear-cache-linux

(clear-cache-linux)
source

clear-cache-mac

(clear-cache-mac)
source

collect-samples

(collect-samples sample-count execution-count f gc-before-sample)
source

estimate-execution-count

(estimate-execution-count period f gc-before-sample estimated-fn-time)

Estimate the number of executions required in order to have at least the specified execution period, check for the jvm to have constant class loader and compilation state.

Estimate the number of executions required in order to have at least the
specified execution period, check for the jvm to have constant class loader
and compilation state.
sourceraw docstring

estimate-overhead

(estimate-overhead)

Calculate a conservative estimate of the timing loop overhead.

Calculate a conservative estimate of the timing loop overhead.
sourceraw docstring

estimated-overhead

(estimated-overhead)
source

estimated-overhead!

(estimated-overhead!)

Sets the estimated overhead.

Sets the estimated overhead.
sourceraw docstring

estimated-overhead-cache

source

execute-expr

(execute-expr n f)

Time the execution of n invocations of f. See execute-expr-core-timed-part.

Time the execution of `n` invocations of `f`. See
`execute-expr-core-timed-part`.
sourceraw docstring

execute-expr-core-timed-part

(execute-expr-core-timed-part n f)

Performs the part of execute-expr where we actually measure the elapsed run time. Evaluates (f) n times, each time saving the return value as an Object in mutable-place.

The idea is that except for the call to (f), the only things done during each iteration are a few arithmetic operations and comparisons to 0 on primitive longs, and the storage of the return value.

The JVM is not free to optimize away the calls to f because the return values are saved in mutable-place.

Performs the part of execute-expr where we actually measure the elapsed run
time.  Evaluates `(f)` `n` times, each time saving the return value as an
Object in `mutable-place`.

The idea is that except for the call to (f), the only things done during each
iteration are a few arithmetic operations and comparisons to 0 on primitive
longs, and the storage of the return value.

The JVM is not free to optimize away the calls to f because the return values
are saved in `mutable-place`.
sourceraw docstring

extract-report-options

(extract-report-options opts)

Extract reporting options from the given options vector. Returns a two element vector containing the reporting options followed by the non-reporting options

Extract reporting options from the given options vector.  Returns a two
element vector containing the reporting options followed by the non-reporting
options
sourceraw docstring

final-gc

(final-gc)

Time a final clean up of JVM memory. If this time is significant compared to the runtime, then the runtime should maybe include this time.

Time a final clean up of JVM memory. If this time is significant compared to
the runtime, then the runtime should maybe include this time.
sourceraw docstring

final-gc-warn

(final-gc-warn execution-time final-gc-time)
source

force-gc

(force-gc)
(force-gc max-attempts)

Force garbage collection and finalisers so that execution time associated with this is not incurred later. Up to max-attempts are made.

Force garbage collection and finalisers so that execution time associated
with this is not incurred later. Up to max-attempts are made.
sourceraw docstring

format-value

(format-value value scale unit)
source

heap-used

(heap-used)

Report a (inconsistent) snapshot of the heap memory used.

Report a (inconsistent) snapshot of the heap memory used.
sourceraw docstring

jvm-class-loader-state

(jvm-class-loader-state)
source

jvm-compilation-state

(jvm-compilation-state)

Returns the total compilation time for the JVM instance.

Returns the total compilation time for the JVM instance.
sourceraw docstring

jvm-jit-name

(jvm-jit-name)

Returns the name of the JIT compiler.

Returns the name of the JIT compiler.
sourceraw docstring

JvmClassLoaderState

source

JvmCompilationState

source

memory

(memory)

Report a (inconsistent) snapshot of the memory situation.

Report a (inconsistent) snapshot of the memory situation.
sourceraw docstring

mutable-place

source

MutablePlaceprotocol

Provides a mutable place

Provides a mutable place
get-place
(get-place _)

Get mutable field value.

Get mutable field value.

set-place
(set-place _ v)

Set mutable field to value.

Set mutable field to value.

sourceraw docstring

ns-to-s

source

os-details

(os-details)

Return the operating system details as a hash.

Return the operating system details as a hash.
sourceraw docstring

outlier-count

(outlier-count low-severe low-mild high-mild high-severe)
source

outlier-effect

(outlier-effect var-out-min)

Return a keyword describing the effect of outliers on the estimate of mean runtime.

Return a keyword describing the effect of outliers on the estimate of mean
runtime.
sourceraw docstring

outlier-significance

(outlier-significance mean-estimate variance-estimate n)

Find the significance of outliers given boostrapped mean and variance estimates. See http://www.ellipticgroup.com/misc/article_supplement.pdf, p17.

Find the significance of outliers given boostrapped mean and variance
estimates.
See http://www.ellipticgroup.com/misc/article_supplement.pdf, p17.
sourceraw docstring

OutlierCount

source

outliers

(outliers data)

Find the outliers in the data using a boxplot technique.

Find the outliers in the data using a boxplot technique.
sourceraw docstring

point-estimate

(point-estimate estimate)
source

point-estimate-ci

(point-estimate-ci estimate)
source

quick-benchmacro

(quick-bench expr & opts)

Convenience macro for benchmarking an expression, expr. Results are reported to out in human readable format. Options for report format are: :os, :runtime, and :verbose.

Convenience macro for benchmarking an expression, expr.  Results are reported
to *out* in human readable format. Options for report format are: :os,
:runtime, and :verbose.
sourceraw docstring

quick-benchmarkmacro

(quick-benchmark expr options)

Benchmark an expression. Less rigorous benchmark (higher uncertainty).

Benchmark an expression. Less rigorous benchmark (higher uncertainty).
sourceraw docstring

quick-benchmark*

(quick-benchmark* f {:as options})

Benchmark an expression. Less rigorous benchmark (higher uncertainty).

Benchmark an expression. Less rigorous benchmark (higher uncertainty).
sourceraw docstring

replace-ret-val-in-time-body-result

(replace-ret-val-in-time-body-result [elapsed-time _] new-ret-val)
source

report

(report format-string & values)

Print format output

Print format output
sourceraw docstring

report-estimate

(report-estimate msg estimate significance)
source

report-estimate-sqrt

(report-estimate-sqrt msg estimate significance)
source

report-outliers

(report-outliers results)
source

report-point-estimate

(report-point-estimate msg estimate)
(report-point-estimate msg estimate quantile)
source

report-point-estimate-sqrt

(report-point-estimate-sqrt msg estimate)
source

report-result

(report-result results & opts)
source

run-benchmark

(run-benchmark sample-count
               warmup-jit-period
               target-execution-time
               f
               gc-before-sample
               overhead)

Benchmark an expression. This tries its best to eliminate sources of error. This also means that it runs for a while. It will typically take 70s for a quick test expression (less than 1s run time) or 10s plus 60 run times for longer running expressions.

Benchmark an expression. This tries its best to eliminate sources of error.
This also means that it runs for a while.  It will typically take 70s for a
quick test expression (less than 1s run time) or 10s plus 60 run times for
longer running expressions.
sourceraw docstring

run-benchmarks-round-robin

(run-benchmarks-round-robin sample-count
                            warmup-jit-period
                            target-execution-time
                            exprs
                            gc-before-sample)

Benchmark multiple expressions in a 'round robin' fashion. Very similar to run-benchmark, except it takes multiple expressions in a sequence instead of only one (each element of the sequence should be a map with keys :f and :expr-string). It runs the following steps in sequence:

  1. Execute each expr once

  2. Run expression 1 for at least warmup-jit-period nanoseconds so the JIT has an opportunity to optimize it. Then do the same for each of the other expressions.

  3. Run expression 1 many times to estimate how many times it must be executed to take a total of target-execution-time nanoseconds. The result is a number of iterations n-exec1 for expression 1. Do the same for each of the other expressions, each with the same target-execution-time, each resulting in its own independent number of executions.

  4. Run expression 1 n-exec1 times, measuring the total elapsed time. Do the same for the rest of the expressions.

  5. Repeat step 4 a total of sample-count times.

Benchmark multiple expressions in a 'round robin' fashion.  Very
similar to run-benchmark, except it takes multiple expressions in a
sequence instead of only one (each element of the sequence should be a
map with keys :f and :expr-string).  It runs the following steps in
sequence:

1. Execute each expr once

2. Run expression 1 for at least warmup-jit-period nanoseconds so the
   JIT has an opportunity to optimize it.  Then do the same for each
   of the other expressions.

3. Run expression 1 many times to estimate how many times it must be
   executed to take a total of target-execution-time nanoseconds.  The
   result is a number of iterations n-exec1 for expression 1.  Do the
   same for each of the other expressions, each with the same
   target-execution-time, each resulting in its own independent number
   of executions.

4. Run expression 1 n-exec1 times, measuring the total elapsed time.
   Do the same for the rest of the expressions.

5. Repeat step 4 a total of sample-count times.
sourceraw docstring

runtime-details

(runtime-details)

Return the runtime details as a hash.

Return the runtime details as a hash.
sourceraw docstring

s-to-ns

source

scale-time

(scale-time measurement)

Determine a scale factor and unit for displaying a time.

Determine a scale factor and unit for displaying a time.
sourceraw docstring

StateChangedprotocol

Interrogation of differences in a state.

Interrogation of differences in a state.
state-changed?
(state-changed? state)

Check to see if a state delta represents no change

Check to see if a state delta represents no change

state-delta
(state-delta state-1 state-2)

Return a state object for the difference between two states

Return a state object for the difference between two states

sourceraw docstring

system-properties

(system-properties)

Return the operating system details.

Return the operating system details.
sourceraw docstring

time-bodymacro

(time-body expr)
(time-body expr pre)

Returns a vector containing execution time and result of specified function.

Returns a vector containing execution time and result of specified function.
sourceraw docstring

time-body-with-jvm-statemacro

(time-body-with-jvm-state expr)
(time-body-with-jvm-state expr pre)

Returns a vector containing execution time, change in loaded and unloaded class counts, change in compilation time and result of specified function.

Returns a vector containing execution time, change in loaded and unloaded
class counts, change in compilation time and result of specified function.
sourceraw docstring

timestampmacro

(timestamp)

Obtain a timestamp

Obtain a timestamp
sourceraw docstring

timestamp-2

(timestamp-2)

Obtain a timestamp, possibly using MXBean.

Obtain a timestamp, possibly using MXBean.
sourceraw docstring

warmup-for-jit

(warmup-for-jit warmup-period f)

Run expression for the given amount of time to enable JIT compilation.

Run expression for the given amount of time to enable JIT compilation.
sourceraw docstring

warn-on-suspicious-jvm-options

(warn-on-suspicious-jvm-options)

Warn if the JIT options are suspicious looking.

Warn if the JIT options are suspicious looking.
sourceraw docstring

with-progress-reportingmacro

(with-progress-reporting expr)

Macro to enable progress reporting during the benchmark.

Macro to enable progress reporting during the benchmark.
sourceraw docstring

cljdoc is a website building & hosting documentation for Clojure/Script libraries

× close