brokvolli.multi — com.sagevisuals/brokvolli 0-SNAPSHOT0

Multi-threaded variants of transduce and brokvolli.single/transduce-kv.

Warning: Use stateful transducers only with extreme caution.

Rules of thumb for using multi-threaded transducing functions in this namespace:

• Collections contain more than 100,000 elements.
• Per-element task involves more than ten computations.
• Partition threshold selected to induce at least ten partitions.

transduce clj

(transduce xform f coll)

(transduce combine xform f coll)

(transduce n combine xform f coll)

Like clojure.core/transduce, but potentially multi-threaded. xform is a composed transducer 'stack' and f is a reducing function, each analogous to their clojure.core/transduce counterparts. This variant of transduce does not accept an explicit 'init' parameter; it must be provided by the arity-0 of f. Returns (combine) when coll is empty or nil.

xform has three arities:

1. Zero args provides the 'init' value of the sub-reduction.
2. One arg provides the 'completing' action on the sub-collection.
3. Two args is the step function, applying the function to the accumulating value and the next element.

Warning: Use stateful transducers only with extreme caution.

combine is a function of three arities that governs how to gather the results of the sub-reductions:

1. Zero args provides the output if coll is empty or nil.
2. One arg is the 'completing' action applied just before the final value.
3. Two args combines the 'left' and 'right' sub-reduction results.

combine defaults to f.

coll must implement clojure.lang.IReduceInit. n is size at which coll is partitioned for multi-threaded processing, defaulting to 512. When n does not divide evenly into (size coll), the locations of the partition boundaries are an implementation detail.

Example, f also provides combine:

(transduce (map identity) + [1 2 3]) ;; => 6

Example with explicit partitioning,

1. Partition coll into partitions with max of two elements,
2. Compute (increment) on different threads, and
3. Combine (concatenate):

(require '[brokvolli.core :refer [concatv]])

(transduce 2 concatv (map inc) conj [11 22 33]) ;; => [12 23 34]

See [[concatv]] for a helper function that concatenates eagerly and efficiently.

Example, 'stack of xforms':

(transduce 3
           concatv
           (comp (map inc)
                 (filter even?)
                 (remove #(< 70 %)))
           conj
           [11 22 33 44 55 66 77 88])
;; => [12 34 56]

Like clojure.core/reduce, elements of hash-maps are peeled off as map entries, 2-tuples of key and value.

Example, working with an associative collection (i.e., a hash-map):

(transduce merge
           (comp (map #(update % 1 inc))
                 (filter #(even? (second %)))
                 (remove #(< 70 (second %))))
           (completing
            (fn
              ([] {})
              ([result value] (conj result value))))
           {:a 11 :b 22 :c 33 :d 44 :e 55 :f 66 :g 77 :h 88 :i 99})
;; => {:e 56, :c 34, :a 12}

See transduce-kv for a transduce variant that behaves like reduce-kv.

transduce-kv clj

(transduce-kv xform f coll)

(transduce-kv combine xform f coll)

(transduce-kv n combine xform f coll)

Multi-threaded variant of single/transduce-kv. xform, f, and combine are the similar to transduce. xform is a transducer stack, f is a reducing function with an additional arity three which consumes the accumulated value, the key/index, and the next element of coll.

As with single/transduce-kv, when used with a transducer stack adapted with [[kv-ize]], the key/index is available as [[keydex]] at any "layer" within the transducer stack; it may be ignored to suit.

coll must implement clojure.lang.IKVReduce.

n is size at which coll is partitioned for multi-threaded processing, defaulting to 512. When n does not divide evenly into (size coll), the locations of the partition boundaries are an implementation detail.

Warning: Use stateful transducers only with extreme caution.

Examples:

(require '[brokvolli.core :refer [concatv tconj]]
         '[brokvolli.transducers-kv :refer [map-kv]])

(transduce-kv (map-kv (fn [keydex x] (+ x (inc keydex)))) tconj [11 22 33])
;; => [12 24 36]

;; same result, but with explicit splitting into threads (also, must use a dedicated combining function, e.g., `concatv`)
(transduce-kv 2 concatv (map-kv (fn [keydex x] (+ x (inc keydex)))) tconj [11 22 33])
;; => [12 24 36]

;; value  keydex  (+ value (inc keydex))  eval  result
;; 11     0       (+ 11 (inc 0))          12    [12]
;; 22     1       (+ 22 (inc 1))          24    [12 24]
;; 33     2       (+ 33 (inc 2))          36    [12 24 36]

See also single/transduce-kv and multi/transduce.