charred.parallel
Parallelism helpers
Parallelism helpers
default-executor-serviceclj
(default-executor-service)Default executor service that is created via 'newCachedThreadPool with a custom thread factory that creates daemon threads. This is an executor service that is suitable for blocking operations as it creates new threads as needed.
Default executor service that is created via 'newCachedThreadPool with a custom thread factory that creates daemon threads. This is an executor service that is suitable for blocking operations as it creates new threads as needed.
in-fork-join-task?clj
(in-fork-join-task?)indexed-map-reduceclj
(indexed-map-reduce num-iters indexed-map-fn)(indexed-map-reduce num-iters indexed-map-fn reduce-fn)(indexed-map-reduce num-iters indexed-map-fn reduce-fn options)Execute indexed-map-fn over n-groups subranges of (range num-iters).
Then call reduce-fn passing in entire in order result value sequence.
num-iters- Indexes are the values of(range num-iters).indexed-map-fn- Function that takes two integers, start-idx and group-len and returns a value. These values are then reduced usingreduce-fn.reduce-fn- Function from sequence of values to result.max-batch-size- Safepoint batch size, defaults to 64000. For more information, see java safepoint documentation.
Implementation:
This function uses the ForkJoinPool/commonPool to parallelize iteration over
(range num-iters) indexes via splitting the index space up into
(>= n-groups ForkJoinPool/getCommonPoolParallelism) tasks. In order to respect
safepoints, n-groups may be only be allowed to iterate over up to max-batch-size
indexes.
If the current thread is already in the common pool, this function executes in the current thread.
Execute `indexed-map-fn` over `n-groups` subranges of `(range num-iters)`. Then call `reduce-fn` passing in entire in order result value sequence. * `num-iters` - Indexes are the values of `(range num-iters)`. * `indexed-map-fn` - Function that takes two integers, start-idx and group-len and returns a value. These values are then reduced using `reduce-fn`. * `reduce-fn` - Function from sequence of values to result. * `max-batch-size` - Safepoint batch size, defaults to 64000. For more information, see [java safepoint documentation](https://medium.com/software-under-the-hood/under-the-hood-java-peak-safepoints-dd45af07d766). Implementation: This function uses the `ForkJoinPool/commonPool` to parallelize iteration over (range num-iters) indexes via splitting the index space up into `(>= n-groups ForkJoinPool/getCommonPoolParallelism)` tasks. In order to respect safepoints, n-groups may be only be allowed to iterate over up to `max-batch-size` indexes. If the current thread is already in the common pool, this function executes in the current thread.
queue-supplierclj
(queue-supplier src-fn & [options])Given a supplier or clojure fn, create a new thread that will read that fn and place the results into a queue of a fixed size. Returns new suplier. Iteration stops when the src-fn returns nil.
Options:
:queue-depth- Queue depth. Defaults to 16.:log-fn- When set a message is logged when the iteration is finished. If no error was encounted log-fn receives the message. If an error is encountered it receives the exception followed by the message. log-fn must be able to take either 1 or 2 arguments.:executor-service- Which executor service to use to run the thread. Defaults to a default one created viadefault-executor-service.:close-fn- Function to call to close upstream iteration. When not provided src-fn is checked and if it implements AutoCloseable then it's close method is called.
Given a supplier or clojure fn, create a new thread that will read that fn and place the results into a queue of a fixed size. Returns new suplier. Iteration stops when the src-fn returns nil. Options: * `:queue-depth` - Queue depth. Defaults to 16. * `:log-fn` - When set a message is logged when the iteration is finished. If no error was encounted log-fn receives the message. If an error is encountered it receives the exception followed by the message. log-fn *must* be able to take either 1 or 2 arguments. * `:executor-service` - Which executor service to use to run the thread. Defaults to a default one created via [[default-executor-service]]. * `:close-fn` - Function to call to close upstream iteration. When not provided src-fn is checked and if it implements AutoCloseable then it's close method is called.
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close