The thread-pool engine that runs every compute: one input, num-work identical workers and one output, all in this JVM. It replaced Onyx in clojask 2.1.
The thread-pool engine that runs every compute: one input, num-work identical workers and one output, all in this JVM. It replaced Onyx in clojask 2.1.
(run num-work open work sink)Run one job and return nil once every result has been handed to sink.
open returns {:data batches :close f}, :close optional. A reader thread walks batches and submits (work batch) to a pool of num-work threads; the calling thread hands the results to sink in input order. At most 2 * num-work + 2 batches are in flight. work must return realized data, otherwise the work moves to the calling thread.
The first exception from open, the batches, work or sink is rethrown unwrapped, after the threads are stopped and the input is closed.
Run one job and return nil once every result has been handed to sink.
open returns {:data batches :close f}, :close optional. A reader thread
walks batches and submits (work batch) to a pool of num-work threads;
the calling thread hands the results to sink in input order. At most
2 * num-work + 2 batches are in flight. work must return realized data,
otherwise the work moves to the calling thread.
The first exception from open, the batches, work or sink is rethrown
unwrapped, after the threads are stopped and the input is closed.(start-aggre num-work
batch-size
df
source
dist
exception
aggre-func
index
formatter
out)Aggregate every group on the pool and append the result rows to dist, in the order the groups are listed.
Aggregate every group on the pool and append the result rows to dist, in the order the groups are listed.
(start-aggre-only num-work
batch-size
df
dist
exception
aggre-func
index
select
out)Fold the rows that pass the filters through aggre-func, pairs of a function and a position in index, then append the result rows to dist. Rows are folded in input order.
Fold the rows that pass the filters through aggre-func, pairs of a function and a position in index, then append the result rows to dist. Rows are folded in input order.
(start-compute num-work batch-size df dist exception index melt out)Compute df into dist, appending after the header that DataFrame.compute may have written. Output is always in input order.
Compute df into dist, appending after the header that DataFrame.compute may have written. Output is always in input order.
(start-groupby num-work
batch-size
df
dist
groupby-keys
groupby-index
exception
&
{:keys [format] :or {format false}})Add every row that passes the filters to its group, a file under dist when dist is a directory path, otherwise the in-memory MGroup dist.
Add every row that passes the filters to its group, a file under dist when dist is a directory path, otherwise the in-memory MGroup dist.
(start-join num-work
batch-size
a
b
source
dist
exception
a-keys
b-keys
a-roll
b-roll
join-type
limit
a-index
b-index
b-format
write-index
out)Join every row of a that passes the filters to its matches in b, whose groups are in source when they are in memory, else in the group files under .clojask/join/b/, and append the result rows to dist in a's order.
Join every row of a that passes the filters to its matches in b, whose groups are in source when they are in memory, else in the group files under .clojask/join/b/, and append the result rows to dist in a's order.
(start-outer num-work
batch-size
a
b
mgroup-a
mgroup-b
dist
exception
a-index
b-index
a-format
b-format
write-index
out)Append to dist every group of a joined to b's group with the same key, then every group of b that had none. The groups are in mgroup-a and mgroup-b when they are in memory, else in the group files under .clojask/join/.
Append to dist every group of a joined to b's group with the same key, then every group of b that had none. The groups are in mgroup-a and mgroup-b when they are in memory, else in the group files under .clojask/join/.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |