parallel.core — parallel 0.10

*mutable* clj

amap clj

(amap f a)

(amap threshold f a)

Applies f in parallel to the elements in the array. The threshold decides how big a chunk of computation should be before going sequential and it's given a default based on the number of available cores.

and clj macro

(and & args)

Like core/and but each expression is evaluated in parralel. It does not short-circuit.

args clj macro

(args f & args)

Call f with each argument evaluated in parralel. This is roughly equivalent to the expansion: (p/args + 1 2 3) => (let [a (future 1) b (future 2) c (future 3)] (+ @a @b @c))

armap clj

(armap f a)

(armap threshold f a)

Applies f in parallel over the reverse of the array. The threshold decides how big is the chunk of sequential computation, with a default of alength / twice the CPUs. Performs better than sequential for non-trivial transforms.

arswap clj

(arswap f low high radius a)

Arrays reverse-swap of the regions identified by: [low, low + radius]....[high - radius, high] Takes transformation f to apply to each item. Preconditions: (pos? (alength a)), (< low high), (pos? radius)

chunk-size clj

(chunk-size coll nchunks)

Calculates the necessary chunk-size to obtain the given number of splits during a parallel fold. nchunks needs to be a power of two.

count clj

(count xform coll)

(count n xform coll)

distinct clj

(distinct coll & xforms)

Returns a non-lazy and unordered sequence of the distinct elements in coll. It does not support null values that need to be removed before calling. Also accepts an optional list of transducers that is applied before removing duplicates. When bound with mutable dynamic var, returns a java.util.Set.

do clj macro

(do & body)

Like core/do but forms evaluate in paralell.

doto clj macro

(doto x & forms)

Like core/doto but forms evaluate in parallel.

external-sort clj

(external-sort fetchf ids)

(external-sort cmp fetchf ids)

(external-sort n cmp fetchf ids)

Allows large datasets (that would otherwise not fit into memory) to be sorted in parallel. It performs the following on a vector of 'ids' and 'fetchf', a function from chunk->data:

• split ids into chunks of approximate size 'n'
• call 'fetchf' on a chunk and expects actual data in return
• sort actual data using 'cmp' ('compare' by default)
• save result to temporary files (deleted when the JVM exits)
• lazily concat files in order as they are requested

fold clj

(fold reducef coll)

(fold combinef reducef coll)

(fold n combinef reducef coll)

Like reducers fold, but with stateful transducers support. Expect reducef to be built using p/xrf to defer initialization. n is the number-of-chunks instead of chunk size. n must be a power of 2 and defaults to 32.

Folder clj protocol

folder clj

(folder coll)

(folder coll nchunks)

foldvec clj

(foldvec v n combinef f)

A general purpose reducers/foldvec taking a generic f to apply at the leaf instead of reduce.

frequencies clj

(frequencies input)

(frequencies input custom-xforms)

Like clojure.core/frequencies, but executes in parallel. It takes an optional comp of transducers to apply to coll before the frequency is calculated.

group-by clj

(group-by f coll & xforms)

Similar to core/group-by, but executes in parallel. It takes an optional list of transducers to apply to the items in coll before generating the groups. Differently from core/group-by, the order of the items in each value vector can change between runs. It's generally 2x-5x faster than core/group-by (without xducers). If dealing with a Java mutable map with Queue type values is not a problem, a further 2x speedup can be achieved by: (binding [p/mutable true] (p/group-by f coll)) Restrictions: it does not support nil values.

let clj macro

(let bindings & body)

Evaluates bindings in parallel and returns the result of evaluating body in the context of those bindings. Bindings have to be indpendent from each other.

max clj

(max coll & xforms)

Find the min in coll in parallel. Accepts optional transducers to apply to coll before searching the min. Effective for coll size >10k items. 4000 is an approximate minimal chunk size.

min clj

(min coll & xforms)

Find the min in coll in parallel. Accepts optional transducers to apply to coll before searching the min. Effective for coll size >10k items. 4000 is an approximate minimal chunk size.

ncpu clj

or clj macro

(or & args)

Like core/or but each expression is evaluated in parralel. It does not short-circuit.

pmap clj

(pmap f input & [n])

Like pmap but eager and unordered. It runs n parallel threads (default 100) independently from the chunk size or the number of cores.

process-folder clj

(process-folder folder xforms)

(process-folder folder reducef combinef xforms)

Applies xforms to all lines of all files inside folder. It supports statful transducers (for example, to skip the header, group stuff, etc.) By default it produces a vector of results, but you can pass a different reducef+combinef to use different data structures.

show-chunks clj

(show-chunks coll nchunks)

Shows chunk sizes for the desired chunk number on a given collection coll.

slurp clj

(slurp file)

(slurp file parsef)

Loads a java.io.File in parallel. By default, the loaded byte array is converted into an UTF-8 string. It takes an optional parsef function of the byte array for additional (or different) processing. When mutable var is true it returns the byte array as is.

sort clj

(sort coll)

(sort cmp coll)

(sort threshold cmp coll)

Splits input coll into chunk of 'threshold' (default 8192) size then sorts chunks in parallel. Input needs conversion into a native array before splitting. More effective for large colls (> 1M elements) or non trivial comparators. Set mutable to 'true' to access the raw results as a mutable array.

transduce clj

(transduce xform f coll)

(transduce xform f combinef coll)

(transduce n xform f combinef coll)

Similar to core/transduce, but executes transducers in parallel. Instead of init, it accepts a combinef to combine results back from parallel execution When combinef is present, it takes precedence over f to establish the initial value for the reduction.

unchunk-map clj

(unchunk-map f coll)

update-vals clj

(update-vals input f)

Use f to update the values of a map in parallel. It performs well with non-trivial f, otherwise is outperformed by reduce-kv. For larger maps (> 100k keys), the final transformation from mutable to persistent dominates over trivial f trasforms. You can access the raw mutable java.util.Map by setting the dynamic binding mutable to true. Restrictions: does not support nil values.

xrf clj

(xrf rf & xforms)

Expects a reducing function rf and a list of transducers (or comp thereof). Use with p/fold to compose any chain of transducers applied to a reducing function to run in parallel.