ham-fisted.api — com.cnuernber/ham-fisted 1.000-beta-90

Fast mutable and immutable associative data structures based on bitmap trie hashmaps. Mutable pathways implement the java.util.Map or Set interfaces including in-place update features such as compute or computeIfPresent.

Mutable maps or sets can be turned into their immutable counterparts via the Clojure persistent! call. This allows working in a mutable space for convenience and performance then switching to an immutable pathway when necessary. Note: after persistent! one should never backdoor mutate map or set again as this will break the contract of immutability. Immutable data structures also support conversion to transient via transient.

Map keysets (.keySet) are full PersistentHashSets of keys.

Maps and sets support metadata but setting the metadata on mutable objects returns a new mutable object that shares the backing store leading to possible issues. Metadata is transferred to the persistent versions of the mutable/transient objects upon persistent!.

Very fast versions of union, difference and intersection are provided for maps and sets with the map version of union and difference requiring an extra argument, a java.util.BiFunction or an IFn taking 2 arguments to merge the left and right sides into the final map. These implementations of union, difference, and intersection are the fastest implementation of these operations we know of on the JVM.

Additionally a fast value update pathway is provided, enabling quickly updating all the values in a given map. Additionally, a new map primitive

• mapmap - allows transforming a given map into a new map quickly by mapping across all the entries.

Unlike the standard Java objects, mutation-via-iterator is not supported.

->bi-function clj

(->bi-function cljfn)

Convert an object to a java.util.BiFunction. Object can either already be a bi-function or an IFn to be invoked with 2 arguments.

->collection clj

(->collection item)

Ensure item is an implementation of java.util.Collection.

->consumer clj

(->consumer cfn)

Return an instance of a consumer, double consumer, or long consumer.

->function clj

(->function cljfn)

Convert an object to a java Function. Object can either already be a Function or an IFn to be invoked.

->long-predicate clj

(->long-predicate f)

->random-access clj

(->random-access item)

Ensure item is derived from java.util.List and java.util.RandomAccess and thus supports constant time random addressing.

->reducible clj

(->reducible item)

Ensure item either implements IReduceInit or java.util.Collection. For arrays this will return an object that has a much more efficient reduction pathway than the base Clojure reducer.

add-all! clj

(add-all! l1 l2)

Add all items from l2 to l1. l1 is expected to be a java.util.List implementation. Returns l1.

apply-concat clj

(apply-concat args)

Faster lazy noncaching version of (apply concat)

argsort clj

(argsort coll)

(argsort comp coll)

Sort a collection of data returning an array of indexes. The collection must be random access and the return value is an integer array of indexes which will read the input data in sorted order. Faster implementations are provided when the collection is an integer, long, or double array. See also reindex.

array-list clj

(array-list)

(array-list data)

Create an implementation of java.util.ArrayList.

assoc clj

(assoc m a b)

(assoc m a b c d)

(assoc m a b c d e f)

(assoc m a b c d e f g h)

(assoc m a b c d e f g h & args)

Drop in faster or equivalent replacement for clojure.core/assoc especially for small numbers of keyval pairs.

assoc! clj

(assoc! obj k v)

assoc! that works on transient collections, implementations of java.util.Map and RandomAccess java.util.List implementations. Be sure to keep track of return value as some implementations return a different return value than the first argument.

assoc-in clj macro

(assoc-in m ks v)

Assoc-in - more efficient replacement if ks is a known compile time constant or a vector. See the caveats in the README before using this exact function.

bi-consumer clj macro

(bi-consumer arg1 arg2 & code)

bi-function clj macro

(bi-function arg1 arg2 & code)

Create an implementation of java.util.function.BiFunction.

binary-operator clj macro

(binary-operator arg1 arg2 & code)

Create an implementation of java.util.function.BinaryOperator

binary-search clj

(binary-search coll v)

(binary-search coll v comp)

Binary search. Coll must be a sorted random access container. comp must be an implementation of java.lang.Comparator. If you know your container's type, such as a double array, then comp should be a fastutil DoubleComparator.

The most efficient method will be to convert coll to random access using ->random-access, so for a pure double array it is slightly better to call ->random-access outside this function before the function call.

This search defaults to the slower java.util.Collections search using clojure's built in compare - reason being that that allows you to search for a double number in a vector of only longs. If you want an accelerated search you can explicitly pass in a nil comparator but you need to make sure that you are searching for the rough datatype in the data - e.g. long in a byte array or a double in a double for float array. Searching for doubles in integer arrays with an accelerated search will probably result in confusing results.

ham-fisted.api> (def data (->random-access (double-array (range 10))))
#'ham-fisted.api/data
ham-fisted.api> (binary-search data 0)
0
ham-fisted.api> (binary-search data -1)
0
ham-fisted.api> (binary-search data 1)
1
ham-fisted.api> (binary-search data 1.1)
2
ham-fisted.api> (binary-search data 10)
10
ham-fisted.api> (binary-search data 11)
10
ham-fisted.api> ;;be wary of datatype conversions in typed containers
ham-fisted.api> (def data (->random-access (int-array (range 10))))
#'ham-fisted.api/data
ham-fisted.api> (binary-search data 1)
1
ham-fisted.api> (binary-search data 1.1)
  2
ham-fisted.api> ;;accelerated search - flattens input to container datatype
ham-fisted.api> (binary-search data 1.1 nil)
1

bind-double-consumer-reducer! clj

(bind-double-consumer-reducer! ctor)

(bind-double-consumer-reducer! cls-type ctor)

Bind a classtype as a double consumer parallel reducer - the consumer must implement DoubleConsumer, ham_fisted.Reducible, and IDeref.

boolean-array clj

(boolean-array)

(boolean-array data)

boolean-array-list clj

(boolean-array-list)

(boolean-array-list data)

byte-array clj

(byte-array)

(byte-array data)

byte-array-list clj

(byte-array-list)

(byte-array-list data)

char-array clj

(char-array)

(char-array data)

char-array-list clj

(char-array-list)

(char-array-list data)

clear! clj

(clear! map-or-coll)

Mutably clear a map, set, list or implementation of java.util.Collection.

clear-memoized-fn! clj

(clear-memoized-fn! memoized-fn)

Clear a memoized function backing store.

comp-nan-first clj

A comparator that sorts null, NAN first, natural order

comp-nan-last clj

A comparator that sorts null, NAN last, natural order

compose-reducers clj

(compose-reducers reducers)

(compose-reducers options reducers)

Given a map or sequence of reducers return a new reducer that produces a map or vector of results.

If data is a sequence then context is guaranteed to be an object array.

Options:

• :rfn-datatype - One of nil, :int64, or :float64. This indicates that the rfn's should all be uniform as accepting longs, doubles, or generically objects. Defaults to nil.

compute! clj

(compute! m k bfn)

Compute a new value in a map derived from an existing value. bfn gets passed k, v where k may be nil. If the function returns nil the corresponding key is removed from the map.

See Map.compute

An example bfn for counting occurrences would be #(if % (inc (long %)) 1).

compute-if-absent! clj

(compute-if-absent! m k bfn)

Compute a value if absent from the map. Useful for memoize-type operations. Must use mutable maps. bfn gets passed k.

See map.computeIfAbsent

compute-if-present! clj

(compute-if-present! m k bfn)

Compute a new value if the value already exists and is non-nil in the hashmap. Must use mutable maps. bfn gets passed k, v where v is non-nil.

See Map.computeIfPresent

concata clj

(concata)

(concata v1)

(concata v1 v2)

(concata v1 v2 & args)

non-lazily concat a set of items returning an object array. This always returns an object array an may return an empty array whereas concat may return nil.

concatv clj

(concatv)

(concatv v1)

(concatv v1 v2)

(concatv v1 v2 & args)

non-lazily concat a set of items returning a persistent vector.

conj! clj

(conj! obj val)

conj! that works on transient collections, implementations of java.util.Set and RandomAccess java.util.List implementations. Be sure to keep track of return value as some implementations return a different return value than the first argument.

constant-count clj

(constant-count data)

Constant time count. Returns nil if input doesn't have a constant time count.

constant-countable? clj

(constant-countable? data)

Return true if data has a constant time count.

consume! clj

(consume! consumer coll)

Consumer a collection. This is simply a reduction where the return value is ignored.

Returns the consumer.

consumer clj macro

(consumer varname & code)

Create an instance of a java.util.function.Consumer

consumer-accumulator clj

(consumer-accumulator c v)

Generic reduction function using a consumer

consumer-preducer clj

(consumer-preducer constructor)

Bind a consumer as a parallel reducer.

Consumer must implement java.util.function.Consumer, ham_fisted.Reducible and clojure.lang.IDeref.

Returns instance of type bound.

See documentation for [[declare-double-consumer-preducer!]].

consumer-reducer clj

(consumer-reducer ctor)

Make a parallel double consumer reducer given a function that takes no arguments and is guaranteed to produce a double consumer which also implements Reducible and IDeref

custom-counted-ireduce clj macro

(custom-counted-ireduce n-elems rfn acc & code)

Custom implementation of IReduceInit and nothing else. This can be the most efficient way to pass data to other interfaces. Also see custom-ireduce if the object does not need to be counted and see reduced-> for implementation helper.

custom-ireduce clj macro

(custom-ireduce rfn acc & code)

Custom implementation of IReduceInit and nothing else. This can be the most efficient way to pass data to other interfaces. Also see custom-counted-ireduce if the object should also implement ICounted. See reduced-> for implementation helper.

darange clj

(darange end)

(darange start end)

(darange start end step)

Return a double array holding the values of the range. Useing ->collection to get an implementation of java.util.List that supports the normal Clojure interfaces.

default-hash-provider clj

Default hash provider - currently set to the hybrid hash provider.

difference clj

(difference map1 map2)

Take the difference of two maps (or sets) returning a new map. Return value is a map1 (or set1) without the keys present in map2.

double->long clj macro

(double->long varname & code)

Create a function that receives a double and returns a long

double->obj clj macro

(double->obj varname & code)

double-accumulator clj macro

(double-accumulator accvar varvar & code)

Type-hinted double reduction accumulator. consumer:

  ham-fisted.api> (reduce (double-accumulator acc v (+ (double acc) v))
                             0.0
                             (range 1000))
#<SimpleSum@2fbcf20: 499500.0>
ham-fisted.api> @*1
499500.0

double-array clj macro

(double-array)

(double-array data)

double-array-list clj

(double-array-list)

(double-array-list cap-or-data)

An array list that is as fast as java.util.ArrayList for add,get, etc but includes many accelerated operations such as fill and an accelerated addAll when the src data is an array list.

double-binary-operator clj macro

(double-binary-operator lvar rvar & code)

Create a binary operator that is specialized for double values. Useful to speed up operations such as sorting or summation.

double-consumer clj macro

(double-consumer varname & code)

Create an instance of a java.util.function.DoubleConsumer

double-consumer-accumulator clj

Converts from a double consumer to a double reduction accumulator that returns the consumer:

ham-fisted.api> (reduce double-consumer-accumulator
                             (Sum$SimpleSum.)
                             (range 1000))
#<SimpleSum@2fbcf20: 499500.0>
ham-fisted.api> @*1
499500.0

double-consumer-preducer clj

(double-consumer-preducer constructor)

Return a preducer for a double consumer.

Consumer must implement java.util.function.DoubleConsumer, ham_fisted.Reducible and clojure.lang.IDeref.

user> (require '[ham-fisted.api :as hamf])
nil
user> (import '[java.util.function DoubleConsumer])
java.util.function.DoubleConsumer
user> (import [ham_fisted Reducible])
ham_fisted.Reducible
user> (import '[clojure.lang IDeref])
clojure.lang.IDeref
user> (deftype MeanR [^{:unsynchronized-mutable true :tag 'double} sum
                      ^{:unsynchronized-mutable true :tag 'long} n-elems]
        DoubleConsumer
        (accept [this v] (set! sum (+ sum v)) (set! n-elems (unchecked-inc n-elems)))
        Reducible
        (reduce [this o]
          (set! sum (+ sum (.-sum ^MeanR o)))
          (set! n-elems (+ n-elems (.-n-elems ^MeanR o)))
          this)
        IDeref (deref [this] (/ sum n-elems)))
user.MeanR
user> (hamf/declare-double-consumer-preducer! MeanR (MeanR. 0 0))
nil
  user> (hamf/preduce-reducer (double-consumer-preducer #(MeanR. 0 0)) (hamf/range 200000))
99999.5

double-consumer-reducer clj

(double-consumer-reducer ctor)

Make a parallel double consumer reducer given a function that takes no arguments and is guaranteed to produce a double consumer which also implements Reducible and IDeref

double-predicate clj macro

(double-predicate varname & code)

Create an implementation of java.util.Function.DoublePredicate

double-unary-operator clj macro

(double-unary-operator varname & code)

Create an implementation of java.util.function.DoubleUnaryOperator

drop clj

(drop n coll)

Drop the first N items of the collection. If item is random access, the return value is random-access.

drop-last clj

(drop-last n coll)

Drop the last N values from a collection. IF the input is random access, the result will be random access.

drop-min clj

(drop-min n values)

(drop-min n comp values)

Drop the min n values of a collection. This is not an order-preserving operation.

dvec clj macro

(dvec)

(dvec data)

Create a persistent-vector-compatible list backed by a double array.

empty-map clj

Constant persistent empty map

empty-set clj

Constant persistent empty set

empty-vec clj

Constant persistent empty vec

empty? clj

(empty? coll)

equal-hash-provider clj

Hash provider based on Object.hashCode and Object.equals - this is the same pathway that java.util.HashMap uses and is the overall the fastest hash provider. Hash-based data structures based on this hash provider will be faster to create and access but will not use the hasheq pathway. This is fine for integer keys, strings, keywords, and symbols, but differs for objects such as doubles, floats, and BigDecimals. This was the default hash provider.

equiv-hash-provider clj

Hash provider based on Clojure's hasheq and equiv pathways - the same algorithm that Clojure's persistent data structures use. This hash provider is somewhat (<2x) slower than the equal-hash-provider.

evict-memoized-call clj

(evict-memoized-call memo-fn fn-args)

filterv clj

(filterv pred coll)

Filter a collection into a vector.

first clj

(first coll)

Get the first item of a collection.

float-array clj macro

(float-array)

(float-array data)

float-array-list clj

(float-array-list)

(float-array-list data)

frequencies clj

(frequencies coll)

(frequencies options coll)

Faster implementation of clojure.core/frequencies.

function clj macro

(function arg & code)

Create a java.util.function.Function

fvec clj macro

(fvec)

(fvec data)

Create a persistent-vector-compatible list backed by a float array.

get-in clj macro

(get-in m ks)

(get-in m ks default-value)

get-in drop-in more efficient replacement if ks is a vector especially if ks is known at compile time.

group-by clj

(group-by f coll)

(group-by f options coll)

Group items in collection by the grouping function f. Returns persistent map of keys to persistent vectors.

Options are same as group-by-reduce but this reductions defaults to an ordered reduction.

group-by-consumer clj

(group-by-consumer key-fn reducer coll)

(group-by-consumer key-fn reducer options coll)

Perform a group-by-reduce passing in a reducer. Same options as group-by-reduce - This uses a slightly different pathway - computeIfAbsent - in order to preserve order. In this case the return value of the reduce fn is ignored. This allows things like the linked hash map to preserve initial order of keys. It map also be slightly more efficient because the map itself does not need to check the return value of rfn - something that the .compute primitive does need to do.

Options:

• :skip-finalize? - skip finalization step.

group-by-reduce clj

(group-by-reduce key-fn init-val-fn rfn merge-fn coll)

(group-by-reduce key-fn init-val-fn rfn merge-fn options coll)

Group by key. Apply the reduce-fn with the new value an the return of init-val-fn. Merged maps due to multithreading will be merged with merge-fn in a similar way of preduce.

This type of reduction can be both faster and more importantly use less memory than a reduction of the forms:

  (->> group-by map into)
  ;; or
  (->> group-by mapmap)

Options (which are passed to preduce):

• :map-fn Function which takes no arguments and must return an instance of java.util.Map that supports computeIfAbsent. Some examples:
  • (constantly (java.util.concurrent.ConcurrentHashMap. ...)) Very fast update especially in the case where the keyspace is large.
  • mut-map - Fast merge, fast update, in-place immutable conversion via persistent!.
  • java-hashmap - fast merge, fast update, just a simple java.util.HashMap-based reduction.
  • #(LinkedHashMap.) - When used with options {:ordered? true} the result keys will be in order and the result values will be reduced in order.

group-by-reducer clj

(group-by-reducer key-fn reducer coll)

(group-by-reducer key-fn reducer options coll)

Perform a group-by-reduce passing in a reducer. Same options as group-by-reduce.

Options:

• :skip-finalize? - skip finalization step.

hash-map clj

(hash-map)

(hash-map a b)

(hash-map a b c d)

(hash-map a b c d e f)

(hash-map a b c d e f g h)

(hash-map a b c d e f g h i j)

(hash-map a b c d e f g h i j k l)

(hash-map a b c d e f g h i j k l m n)

(hash-map a b c d e f g h i j k l m n o p)

(hash-map a b c d e f g h i j k l m n o p & args)

Drop-in replacement to Clojure's hash-map function.

hybrid-hash-provider clj

Hash provider opportunistically using IHashEq pathway when provided else falling back to mixhash(obj.hashCode). For equality strictly uses Util.equiv as equality has not shown up to be a profiler bottleneck while generating mumur3 compatible hashes has in some cases (integers). This hash provider provides a middle ground offering more performance for simple datatypes but still using the more robust equiv pathways for more complex datatypes. This is currently the default hash provider for the library.

iarange clj

(iarange end)

(iarange start end)

(iarange start end step)

Return an integer array holding the values of the range. Use ->collection to get a list implementation wrapping for generic access.

immut-list clj

(immut-list)

(immut-list data)

Create a persistent list. Object arrays will be treated as if this new object owns them.

immut-map clj

(immut-map)

(immut-map data)

(immut-map options data)

Create an immutable map. This object supports conversion to a transient map via Clojure's transient function. Duplicate keys are treated as if by assoc.

If data is an object array it is treated as a flat key-value list which is distinctly different than how conj! treats object arrays. You have been warned.

If you know you will have consistently more key/val pairs than 8 you should just use (persistent! (mut-map data)) as that avoids the transition from an arraymap to a persistent hashmap.

Options:

• :hash-provider - An implementation of BitmapTrieCommon$HashProvider. Defaults to the default-hash-provider.

Examples:

ham-fisted.api> (immut-map (obj-ary :a 1 :b 2 :c 3 :d 4))
{:a 1, :b 2, :c 3, :d 4}
ham-fisted.api> (type *1)
ham_fisted.PersistentArrayMap
ham-fisted.api> (immut-map (obj-ary :a 1 :b 2 :c 3 :d 4 :e 5))
{:d 4, :b 2, :c 3, :a 1, :e 5}
ham-fisted.api> (type *1)
ham_fisted.PersistentHashMap
ham-fisted.api> (immut-map [[:a 1][:b 2][:c 3][:d 4][:e 5]])
{:d 4, :b 2, :c 3, :a 1, :e 5}
ham-fisted.api> (type *1)
ham_fisted.PersistentHashMap

immut-set clj

(immut-set)

(immut-set data)

(immut-set options data)

Create an immutable hashset based on a hash table. This object supports conversion to transients via transient.

Options:

• :hash-provider - An implementation of BitmapTrieCommon$HashProvider. Defaults to the default-hash-provider.

in-fork-join-task? clj

(in-fork-join-task?)

True if you are currently running in a fork-join task

inc-consumer clj

(inc-consumer)

(inc-consumer init-value)

Return a consumer that simply increments a long. See java/ham_fisted/Consumers.java for definition.

indexed-accum clj macro

(indexed-accum accvar idxvar varvar & code)

Create an indexed accumulator that recieves and additional long index during a reduction:

ham-fisted.api> (reduce (indexed-accum
                         acc idx v (conj acc [idx v]))
                        []
                        (range 5))
[[0 0] [1 1] [2 2] [3 3] [4 4]]

indexed-double-accum clj macro

(indexed-double-accum accvar idxvar varvar & code)

Create an indexed double accumulator that recieves and additional long index during a reduction:

ham-fisted.api> (reduce (indexed-double-accum
                         acc idx v (conj acc [idx v]))
                        []
                        (range 5))
[[0 0.0] [1 1.0] [2 2.0] [3 3.0] [4 4.0]]

indexed-long-accum clj macro

(indexed-long-accum accvar idxvar varvar & code)

Create an indexed long accumulator that recieves and additional long index during a reduction:

ham-fisted.api> (reduce (indexed-long-accum
                         acc idx v (conj acc [idx v]))
                        []
                        (range 5))
[[0 0] [1 1] [2 2] [3 3] [4 4]]

int-array clj macro

(int-array)

(int-array data)

int-array-list clj

(int-array-list)

(int-array-list cap-or-data)

An array list that is as fast as java.util.ArrayList for add,get, etc but includes many accelerated operations such as fill and an accelerated addAll when the src data is an array list.

intersection clj

(intersection s1 s2)

Intersect the keyspace of set1 and set2 returning a new set. Also works if s1 is a map and s2 is a set - the map is trimmed to the intersecting keyspace of s1 and s2.

into clj

(into container data)

(into container xform data)

Like clojure.core/into, but also designed to handle editable collections, transients, and base java.util.Map, List and Set containers.

into-array clj

(into-array aseq)

(into-array ary-type aseq)

(into-array ary-type mapfn aseq)

Faster version of clojure.core/into-array.

ivec clj macro

(ivec)

(ivec data)

Create a persistent-vector-compatible list backed by an int array.

java-concurrent-hashmap clj

(java-concurrent-hashmap)

(java-concurrent-hashmap data)

Create a java concurrent hashmap which is still the fastest possible way to solve a few concurrent problems.

java-hashmap clj

(java-hashmap)

(java-hashmap data)

(java-hashmap xform data)

(java-hashmap xform options data)

Create a java.util.HashMap. Duplicate keys are treated as if map was created by assoc.

java-hashset clj

(java-hashset)

(java-hashset data)

Create a java hashset which is still the fastest possible way to solve a few problems.

java-linked-hashmap clj

(java-linked-hashmap)

(java-linked-hashmap data)

Linked hash maps perform identically or very nearly so to java.util.HashMaps but they retain the order of insertion and modification.

keys clj

(keys m)

Return the keys of a map. This version allows parallel reduction operations on the returned sequence.

larange clj

(larange end)

(larange start end)

(larange start end step)

Return a long array holding values of the range. Use ->collection get a list implementation for generic access.

last clj

(last coll)

Get the last item in the collection. Constant time for random access lists.

long->double clj macro

(long->double varname & code)

Create a function that receives a long and returns a double

long->obj clj macro

(long->obj varname & code)

Create a function that receives a primitive long and returns an object.

long-accumulator clj macro

(long-accumulator accvar varvar & code)

Type-hinted double reduction accumulator. consumer:

  ham-fisted.api> (reduce (double-accumulator acc v (+ (double acc) v))
                             0.0
                             (range 1000))
#<SimpleSum@2fbcf20: 499500.0>
ham-fisted.api> @*1
499500.0

long-array clj macro

(long-array)

(long-array data)

long-array-list clj

(long-array-list)

(long-array-list cap-or-data)

An array list that is as fast as java.util.ArrayList for add,get, etc but includes many accelerated operations such as fill and an accelerated addAll when the src data is an array list.

long-binary-operator clj macro

(long-binary-operator lvar rvar & code)

Create a binary operator that is specialized for long values. Useful to speed up operations such as sorting or summation.

long-consumer clj macro

(long-consumer varname & code)

Create an instance of a java.util.function.LongConsumer

long-consumer-accumulator clj

Converts from a long consumer to a long reduction accumulator that returns the consumer:

ham-fisted.api> (reduce double-consumer-accumulator
                             (Sum$SimpleSum.)
                             (range 1000))
#<SimpleSum@2fbcf20: 499500.0>
ham-fisted.api> @*1
499500.0

long-consumer-reducer clj

(long-consumer-reducer ctor)

Make a parallel double consumer reducer given a function that takes no arguments and is guaranteed to produce a double consumer which also implements Reducible and IDeref

long-predicate clj macro

(long-predicate varname & code)

Create an implementation of java.util.Function.LongPredicate

long-unary-operator clj macro

(long-unary-operator varname & code)

Create an implementation of java.util.function.LongUnaryOperator

lvec clj macro

(lvec)

(lvec data)

Create a persistent-vector-compatible list backed by a long array.

make-comparator clj macro

(make-comparator lhsvar rhsvar & code)

Make a java comparator.

make-double-comparator clj macro

(make-double-comparator lhsvar rhsvar & code)

Make a comparator that gets passed two double arguments.

make-long-comparator clj macro

(make-long-comparator lhsvar rhsvar & code)

Make a comparator that gets passed two long arguments.

map-intersection clj

(map-intersection bfn map1 map2)

Intersect the keyspace of map1 and map2 returning a new map. Each value is the result of bfn applied to the map1-value and map2-value, respectively. See documentation for map-union.

Clojure's merge functionality can be duplicate via:

(map-intersection (fn [lhs rhs] rhs) map1 map2)

map-keyset clj

(map-keyset m)

Return the keyset of the map. This may not be in the same order as (keys m) or (vals m). For hamf maps, this has the same ordering as (keys m). For both hamf and java hashmaps, the returned implementation of java.util.Set has both more utility and better performance than (keys m).

map-union clj

(map-union bfn map1 map2)

Take the union of two maps returning a new map. bfn is a function that takes 2 arguments, map1-val and map2-val and returns a new value. Has fallback if map1 and map2 aren't backed by bitmap tries.

• bfn - A function taking two arguments and returning one. + is a fine choice.
• map1 - the lhs of the union.
• map2 - the rhs of the union.

Returns a persistent map if input is a persistent map else if map1 is a mutable map map1 is returned with overlapping entries merged. In this way you can pass in a normal java hashmap, a linked java hashmap, or a persistent map and get back a result that matches the input.

map-union-java-hashmap clj

(map-union-java-hashmap bfn lhs rhs)

Take the union of two maps returning a new map. See documentation for [map-union]. Returns a java.util.HashMap.

map-values clj

(map-values m)

Return the values collection of the map. This may not be in the same order as (keys m) or (vals m). For hamf hashmaps, this does have the same order as (vals m).

mapmap clj

(mapmap map-fn src-map)

Clojure's missing piece. Map over the data in src-map, which must be a map or sequence of pairs, using map-fn. map-fn must return either a new key-value pair or nil. Then, remove nil pairs, and return a new map. If map-fn returns more than one pair with the same key later pair will overwrite the earlier pair.

Logically the same as:

(into {} (comp (map map-fn) (remove nil?)) src-map)

mapv clj

(mapv map-fn coll)

(mapv map-fn c1 c2)

(mapv map-fn c1 c2 c3)

(mapv map-fn c1 c2 c3 & args)

Produce a persistent vector from a collection.

mean clj

(mean coll)

(mean options coll)

Return the mean of the collection. Returns double/NaN for empty collections. See options for sum.

memoize clj

(memoize memo-fn)

(memoize memo-fn
         {:keys [write-ttl-ms access-ttl-ms soft-values? weak-values? max-size
                 record-stats? eviction-fn]})

Efficient thread-safe version of clojure.core/memoize.

Also see clear-memoized-fn! evict-memoized-call and memoize-cache-as-map to mutably clear the backing store, manually evict a value, and get a java.util.Map view of the cache backing store.

ham-fisted.api> (def m (memoize (fn [& args] (println "fn called - " args) args)
                                {:write-ttl-ms 1000 :eviction-fn (fn [args rv cause]
                                                                   (println "evicted - " args rv cause))}))
#'ham-fisted.api/m
ham-fisted.api> (m 3)
fn called -  (3)
(3)
ham-fisted.api> (m 4)
fn called -  (4)
(4)evicted -  [3] (3) :expired
ham-fisted.api> (dotimes [idx 4] (do (m 3) (evict-memoized-call m [3])))
fn called -  (3)
fn called -  (3)
fn called -  (3)
fn called -  (3)
nil
ham-fisted.api> (dotimes [idx 4] (do (m 3) #_(evict-memoized-call m [3])))
fn called -  (3)
nil

Options:

• :write-ttl-ms - Time that values should remain in the cache after write in milliseconds.
• :access-ttl-ms - Time that values should remain in the cache after access in milliseconds.
• :soft-values? - When true, the cache will store SoftReferences to the data.
• :weak-values? - When true, the cache will store WeakReferences to the data.
• :max-size - When set, the cache will behave like an LRU cache.
• :record-stats? - When true, the LoadingCache will record access statistics. You can get those via the undocumented function memo-stats.
• :eviction-fn - Function that receives 3 arguments, [args v cause], when a value is evicted. Causes the keywords:collected :expired :explicit :replaced and :size`. See caffeine documentation for cause definitions.

memoize-cache-as-map clj

(memoize-cache-as-map memoized-fn)

Return the memoize backing store as an implementation of java.util.Map.

merge clj

(merge)

(merge m1)

(merge m1 m2)

(merge m1 m2 & args)

Merge 2 maps with the rhs values winning any intersecting keys. Uses map-union with BitmapTrieCommon/rhsWins.

Returns a new persistent map.

merge-with clj

(merge-with f)

(merge-with f m1)

(merge-with f m1 m2)

(merge-with f m1 m2 & args)

Merge (union) any number of maps using f as the merge operator. f gets passed two arguments, lhs-val and rhs-val and must return a new value.

Returns a new persistent map.

mmax-key clj

(mmax-key f data)

Faster and nil-safe version of #(apply max-key %1 %2)

mmin-key clj

(mmin-key f data)

Faster and nil-safe version of #(apply min-key %1 %2)

mode clj

(mode data)

Return the most common occurance in the data.

mut-hashtable-map clj

(mut-hashtable-map)

(mut-hashtable-map data)

(mut-hashtable-map xform data)

(mut-hashtable-map xform options data)

Create a mutable implementation of java.util.Map. This object efficiently implements ITransient map so you can use assoc! and persistent! on it but you can additionally use the various members of the java.util.Map interface such as put, compute, computeIfAbsent, replaceAll and merge.

If data is an object array it is treated as a flat key-value list which is distinctly different than how conj! treats object arrays. You have been warned.

Options:

• :hash-provider - An implementation of BitmapTrieCommon$HashProvider. Defaults to the default-hash-provider.

mut-list clj

(mut-list)

(mut-list data)

Create a mutable java list that is in-place convertible to a persistent list

mut-long-hashtable-map clj

(mut-long-hashtable-map)

(mut-long-hashtable-map data)

(mut-long-hashtable-map xform data)

(mut-long-hashtable-map xform options data)

Create a mutable implementation of java.util.Map. This object efficiently implements ITransient map so you can use assoc! and persistent! on it but you can additionally use the various members of the java.util.Map interface such as put, compute, computeIfAbsent, replaceAll and merge.

If data is an object array it is treated as a flat key-value list which is distinctly different than how conj! treats object arrays. You have been warned.

Options:

• :hash-provider - An implementation of BitmapTrieCommon$HashProvider. Defaults to the default-hash-provider.

mut-map clj

(mut-map)

(mut-map data)

(mut-map xform data)

(mut-map xform options data)

mut-map-rf clj

(mut-map-rf cons-fn)

(mut-map-rf cons-fn finalize-fn)

mut-map-union! clj

(mut-map-union! merge-bifn l r)

Very fast union that may simply update lhs and return it. Both lhs and rhs must be mutable maps. See docs for map-union.

mut-set clj

(mut-set)

(mut-set data)

(mut-set options data)

Create a mutable hashset based on the hashtable. You can create a persistent hashset via the clojure persistent! call.

Options:

• :hash-provider - An implementation of BitmapTrieCommon$HashProvider. Defaults to the default-hash-provider.

mut-trie-map clj

(mut-trie-map)

(mut-trie-map data)

(mut-trie-map xform data)

(mut-trie-map xform options data)

Create a mutable implementation of java.util.Map. This object efficiently implements ITransient map so you can use assoc! and persistent! on it but you can additionally use operations such as put!, remove!, compute-at! and compute-if-absent!. You can create a persistent hashmap via the clojure persistent! call.

If data is an object array it is treated as a flat key-value list which is distinctly different than how conj! treats object arrays. You have been warned.

Options:

• :hash-provider - An implementation of BitmapTrieCommon$HashProvider. Defaults to the default-hash-provider.

mutable-map? clj

(mutable-map? m)

obj->double clj macro

(obj->double)

(obj->double varname & code)

Create a function that converts objects to doubles

obj->long clj macro

(obj->long)

(obj->long varname & code)

Create a function that converts objects to longs

obj-ary clj

(obj-ary)

(obj-ary v0)

(obj-ary v0 v1)

(obj-ary v0 v1 v2)

(obj-ary v0 v1 v2 v3)

(obj-ary v0 v1 v2 v3 v4)

(obj-ary v0 v1 v2 v3 v4 v5)

(obj-ary v0 v1 v2 v3 v4 v5 v6)

(obj-ary v0 v1 v2 v3 v4 v5 v6 v7)

(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8)

(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8 v9)

(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10)

(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11)

(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12)

(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13)

(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14)

(obj-ary v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15)

As quickly as possible, produce an object array from these inputs. Very fast for arities <= 16.

object-array clj

(object-array item)

Faster version of object-array for java collections and strings.

object-array-list clj

(object-array-list)

(object-array-list cap-or-data)

An array list that is as fast as java.util.ArrayList for add,get, etc but includes many accelerated operations such as fill and an accelerated addAll when the src data is an object array based list.

ovec clj macro

(ovec)

(ovec data)

Return an immutable persistent vector like object backed by a single object array.

persistent! clj

(persistent! v)

If object is an ITransientCollection, call clojure.core/persistent!. Else return collection.

pgroups clj

(pgroups n-elems body-fn)

(pgroups n-elems body-fn options)

Run y groups across n-elems. Y is common pool parallelism.

body-fn gets passed two longs, startidx and endidx.

Returns a sequence of the results of body-fn applied to each group of indexes.

Before using this primitive please see if preduce will work.

Options:

• :pgroup-min - when provided n-elems must be more than this value for the computation to be parallelized.
• :batch-size - max batch size. Defaults to 64000.

pmap clj

(pmap map-fn & sequences)

pmap using the commonPool. This is useful for interacting with other primitives, namely pgroups which are also based on this pool. This is a change from Clojure's base pmap in that it uses the ForkJoinPool/commonPool for parallelism as opposed to the agent pool - this makes it compose with pgroups and dtype-next's parallelism system.

Before using this primitive please see if preduce will work.

Is guaranteed to not trigger the need for shutdown-agents.

predicate clj macro

(predicate varname & code)

Create an implementation of java.util.Function.Predicate

preduce clj

(preduce init-val-fn rfn merge-fn coll)

(preduce init-val-fn rfn merge-fn options coll)

Parallelized reduction. Currently coll must either be random access or a lznc map/filter chain based on one or more random access entities, hashmaps and sets from this library or any java.util set, hashmap or concurrent versions of these. If input cannot be parallelized this lowers to a normal serial reduction.

• init-val-fn - Potentially called in reduction threads to produce each initial value.
• rfn - normal clojure reduction function. Typehinting the second argument to double or long will sometimes produce a faster reduction.
• merge-fn - Merge two reduction results into one.

Options:

• :pool - The fork-join pool to use. Defaults to common pool which assumes reduction is cpu-bound.
• :parallelism - What parallelism to use - defaults to pool's getParallelism method.
• :max-batch-size - Rough maximum batch size for indexed or grouped reductions. This can both even out batch times and ensure you don't get into safepoint trouble with jdk-8.
• :min-n - minimum number of elements before initiating a parallelized reduction - Defaults to 1000 but you should customize this particular to your specific reduction.
• :ordered? - True if results should be in order. Unordered results sometimes are slightly faster but again you should test for your specific situation..
• :cat-parallelism - Either :seq-wise or :elem-wise, defaults to :seq-wise. Test for your specific situation, this really is data-dependent. This contols how a concat primitive parallelizes the reduction across its contains. Elemwise means each container's reduction is individually parallelized while seqwise indicates to do a pmap style initial reduction across containers then merge the results.
• :put-timeout-ms - Number of milliseconds to wait for queue space before throwing an exception in unordered reductions. Defaults to 50000.
• :unmerged-result? - Defaults to false. When true, the sequence of results be returned directly without any merge steps in a lazy-noncaching container. Beware the noncaching aspect -- repeatedly evaluating this result may kick off the parallelized reduction multiple times. To ensure caching if unsure call seq on the result ...).

preduce-reducer clj

(preduce-reducer reducer coll)

(preduce-reducer reducer options coll)

Given an instance of ham-fisted.protocols/ParallelReducer, perform a parallel reduction.

In the case where the result is requested unmerged then finalize will be called on each result in a lazy noncaching way. In this case you can use a non-parallelized reducer and simply get a sequence of results as opposed to one.

• reducer - instance of ParallelReducer
• options - Same options as preduce.
• coll - something potentially with a parallelizable reduction.

See options for preduce.

Additional Options:

:skip-finalize? - when true, the reducer's finalize method is not called on the result.

preduce-reducers clj

(preduce-reducers reducers coll)

(preduce-reducers reducers options coll)

Given a map or sequence of ham-fisted.protocols/ParallelReducer, produce a map or sequence of reduced values. Reduces over input coll once in parallel if coll is large enough. See options for preduce.

ham-fisted.api> (preduce-reducers {:sum (Sum.) :mult *} (range 20))
{:mult 0, :sum #<Sum@5082c3b7: {:sum 190.0, :n-elems 20}>}

priority-queue-rf clj

(priority-queue-rf n comp)

range clj

(range)

(range end)

(range start end)

(range start end step)

When given arguments returns a range that implements random access java list interfaces so nth, reverse and friends are efficient.

rcomp clj

A reverse comparator that sorts in descending order

reduce-reducer clj

(reduce-reducer reducer coll)

Serially reduce a reducer.

ham-fisted.api> (reduce-reducer (Sum.) (range 1000))
#<Sum@afbedb: {:sum 499500.0, :n-elems 1000}>

reduce-reducers clj

(reduce-reducers reducers coll)

Serially reduce a map or sequence of reducers into a map or sequence of results.

ham-fisted.api> (reduce-reducers {:a (Sum.) :b *} (range 1 21))
{:b 2432902008176640000, :a #<Sum@6bcebeb1: {:sum 210.0, :n-elems 20}>}

reduced-> clj macro

(reduced-> rfn acc & data)

Helper macro to implement reduce chains checking for if the accumulator is reduced before calling the next expression in data.

(defrecord YMC [year-month ^long count]
  clojure.lang.IReduceInit
  (reduce [this rfn init]
    (let [init (reduced-> rfn init
                   (clojure.lang.MapEntry/create :year-month year-month)
                   (clojure.lang.MapEntry/create :count count))]
      (if (and __extmap (not (reduced? init)))
        (reduce rfn init __extmap)
        init))))

reducer->completef clj

(reducer->completef reducer)

Return fold-compatible pair of [reducef, completef] given a parallel reducer. Note that folded reducers are not finalized as of this time:

ham-fisted.api> (def data (vec (range 200000)))
#'ham-fisted.api/data
ham-fisted.api> (r/fold (reducer->completef (Sum.)) (reducer->rfn (Sum.)) data)
#<Sum@858c206: {:sum 1.99999E10, :n-elems 200000}>

reducer->rf clj

(reducer->rf reducer)

Given a reducer, return a transduce-compatible rf -

ham-fisted.api> (transduce (clojure.core/map #(+ % 2)) (reducer->rf (Sum.)) (range 200))
{:sum 20300.0, :n-elems 200}

reducer-with-finalize clj

(reducer-with-finalize reducer fin-fn)

reducer-xform->reducer clj

(reducer-xform->reducer reducer xform)

Given a reducer and a transducer xform produce a new reducer which will apply the transducer pipeline before is reduction function.

ham-fisted.api> (reduce-reducer (reducer-xform->reducer (Sum.) (clojure.core/filter even?))
                                (range 1000))
#<Sum@479456: {:sum 249500.0, :n-elems 500}>

!! - If you use a stateful transducer here then you must not use the reducer in a parallelized reduction.

reducible-merge clj

(reducible-merge lhs rhs)

Parallel reduction merge function that expects both sides to be an instances of Reducible

reindex clj

(reindex coll indexes)

Permut coll by the given indexes. Result is random-access and the same length as the index collection. Indexes are expected to be in the range of [0->count(coll)).

repeat clj

(repeat v)

(repeat n v)

When called with no arguments, produce an infinite sequence of v. When called with 2 arguments, produce a random access list that produces v at each index.

rest clj

(rest coll)

Version of rest that does uses subvec if collection is random access. This preserves the ability to reduce in parallel over the collection.

reverse clj

(reverse coll)

Reverse a collection or sequence. Constant time reverse is provided for any random access list.

short-array clj

(short-array)

(short-array data)

short-array-list clj

(short-array-list)

(short-array-list data)

shuffle clj

(shuffle coll)

(shuffle coll opts)

shuffle values returning random access container.

Options:

• :seed - If instance of java.util.Random, use this. If integer, use as seed. If not provided a new instance of java.util.Random is created.

sort clj

(sort coll)

(sort comp coll)

Exact replica of clojure.core/sort but instead of wrapping the final object array in a seq which loses the fact the result is countable and random access. Faster implementations are provided when the input is an integer, long, or double array.

The default comparison is nan-last meaning null-last if the input is an undefined container and nan-last if the input is a double or float specific container.

sort-by clj

(sort-by keyfn coll)

(sort-by keyfn comp coll)

Sort a collection by keyfn. Typehinting the return value of keyfn will somewhat increase the speed of the sort :-).

sorta clj

(sorta coll)

(sorta comp coll)

Sort returning an object array.

splice clj

(splice v1 idx v2)

Splice v2 into v1 at idx. Returns a persistent vector.

subvec clj

(subvec m sidx)

(subvec m sidx eidx)

More general version of subvec. Works for any java list implementation including persistent vectors and any array.

sum clj

(sum coll)

(sum options coll)

Very stable high performance summation. Uses both threading and kahans compensated summation.

Options:

• nan-strategy - defaults to :remove. Options are :keep, :remove and :exception.

sum-fast clj

(sum-fast coll)

Fast simple serial double summation. Does not do any nan checking or summation compensation.

sum-stable-nelems clj

(sum-stable-nelems coll)

(sum-stable-nelems options coll)

Stable sum returning map of {:sum :n-elems}. See options for sum.

take clj

(take n coll)

Take the first N values from a collection. If the input is random access, the result will be random access.

take-last clj

(take-last n coll)

Take the last N values of the collection. If the input is random-access, the result will be random-access.

take-min clj

(take-min n values)

(take-min n comp values)

Take the min n values of a collection. This is not an order-preserving operation.

transient-map-rf clj

(transient-map-rf cons-fn)

(transient-map-rf cons-fn finalize-fn)

type-single-arg-ifn clj

(type-single-arg-ifn ifn)

Categorize the return type of a single argument ifn. May be :float64, :int64, or :object.

unary-operator clj macro

(unary-operator varname & code)

Create an implementation of java.util.function.UnaryOperator

union clj

(union s1 s2)

Union of two sets or two maps. When two maps are provided the right hand side wins in the case of an intersection.

Result is either a set or a map, depending on if s1 is a set or map.

union-reduce-java-hashmap clj

(union-reduce-java-hashmap bfn maps)

(union-reduce-java-hashmap bfn maps options)

Do an efficient union of many maps into a single java.util.HashMap.

union-reduce-maps clj

(union-reduce-maps bfn maps)

Do an efficient union reduction across many maps using bfn to update values. If the first map is mutable the union is done mutably into the first map and it is returned.

update clj

(update m k f)

(update m k f x)

(update m k f x y)

(update m k f x y z)

(update m k f x y z & more)

Version of update that produces maps from this library.

update-in clj macro

(update-in m ks f & args)

An attempt at a slightly more efficient version of update-in.

See the caveats in the readme - measure carefully before using this.

update-values clj

(update-values map bfn)

Immutably (or mutablye) update all values in the map returning a new map. bfn takes 2 arguments, k,v and returns a new v. Returns a new persistent map if input is persistent else an updated mutable map. If passed a vector, k is the index and v is the value. Will return a new vector. else map is assumed to be convertible to a sequence and this pathway works the same as map-indexed.

upgroups clj

(upgroups n-elems body-fn)

(upgroups n-elems body-fn options)

Run y groups across n-elems. Y is common pool parallelism.

body-fn gets passed two longs, startidx and endidx.

Returns a sequence of the results of body-fn applied to each group of indexes.

Before using this primitive please see if preduce will work.

Options:

• :pgroup-min - when provided n-elems must be more than this value for the computation to be parallelized.
• :batch-size - max batch size. Defaults to 64000.

upmap clj

(upmap map-fn & sequences)

Unordered pmap using the commonPool. This is useful for interacting with other primitives, namely pgroups which are also based on this pool.

Before using this primitive please see if preduce will work.

Like pmap this uses the commonPool so it composes with this api's pmap, pgroups, and dtype-next's parallelism primitives but it does not impose an ordering constraint on the results and thus may be significantly faster in some cases.

vals clj

(vals m)

Return the values of a map. This version allows parallel reduction operations on the returned sequence. Returned sequence is in same order as (keys m).

vec clj

(vec)

(vec data)

Produce a persistent vector. Optimized pathways exist for object arrays and java List implementations.

vector clj

(vector)

(vector a)

(vector a b)

(vector a b c)

(vector a b c d)

(vector a b c d e)

(vector a b c d e f)

(vector a b c d e f g)

(vector a b c d e f g h)

(vector a b c d e f g h i)

(vector a b c d e f g h i j)

(vector a b c d e f g h i j k)

(vector a b c d e f g h i j k & args)

wrap-array clj

(wrap-array ary)

Wrap an array with an implementation of IMutList

wrap-array-growable clj

(wrap-array-growable ary)

(wrap-array-growable ary ptr)

Wrap an array with an implementation of IMutList that supports add and addAllReducible. 'ptr is the numeric put ptr, defaults to the array length. Pass in zero for a preallocated but empty growable wrapper.