clojask.onyx-comps — com.github.clojure-finance/clojask 2.0.5

catalog clj

catalog-aggre-gen clj

(catalog-aggre-gen num-work batch-size)

Generate the catalog for running Onyx

catalog-gen clj

(catalog-gen num-work batch-size)

Generate the catalog for running Onyx

catalog-groupby-gen clj

(catalog-groupby-gen num-work batch-size)

Generate the catalog for running Onyx

catalog-join-gen clj

(catalog-join-gen num-work batch-size)

Generate the catalog for running Onyx

close-reader clj

(close-reader event lifecycle)

config-env clj

(config-env)

Start the embedded ZooKeeper, the peer group and the peers for the current workflow. The ZooKeeper port, the Aeron port and the Aeron media-driver directory come from the JVM system properties clojask.zookeeper.port, clojask.aeron.port and clojask.aeron.dir, or the environment variables CLOJASK_ZOOKEEPER_PORT, CLOJASK_AERON_PORT and CLOJASK_AERON_DIR, so two clojask processes can share a machine.

dataframe clj

env clj

env-config clj

flow-cond-gen clj

(flow-cond-gen num-work)

Generate the flow conditions for running Onyx

flow-conditions clj

id clj

in-calls clj

inject-in-reader clj

(inject-in-reader event lifecycle)

lifecycle-aggre-gen clj

(lifecycle-aggre-gen source dist)

lifecycle-gen clj

(lifecycle-gen source dist order select)

lifecycle-groupby-gen clj

(lifecycle-groupby-gen source dist keys key-index)

lifecycle-join-gen clj

(lifecycle-join-gen source dist a b a-keys b-keys a-roll b-roll join-type)

lifecycles clj

log-path clj

n-peers clj

peer-config clj

peer-group clj

predicate-funcs clj

run-job clj

(run-job stage prepare!)

Run one job inside with-onyx-env. prepare! does the per-job setup and returns the job map for submit-and-wait. Failures are tagged with the stage they came from; with-onyx-env shuts the environment down.

shutdown clj

(shutdown)

Tear down whatever config-env managed to start, in reverse order. Every step is attempted even if an earlier one fails, so a startup that dies after ZooKeeper is up (for example in the peer group) still releases the ZooKeeper port instead of poisoning every later compute in this JVM. The first error, if any, is rethrown once all steps have run.

start-onyx clj

(start-onyx num-work batch-size dataframe dist exception order index melt out)

start the onyx cluster with the specification inside dataframe

start-onyx-aggre-only clj

(start-onyx-aggre-only num-work
                       batch-size
                       dataframe
                       dist
                       exception
                       aggre-func
                       index
                       select
                       out)

start the onyx cluster with the specification inside dataframe

start-onyx-groupby clj

(start-onyx-groupby num-work
                    batch-size
                    dataframe
                    dist
                    groupby-keys
                    groupby-index
                    exception
                    &
                    {:keys [format] :or {format false}})

start the onyx cluster with the specification inside dataframe

@format: if format the value before writing to file. For procedures that will need to compare / use the actual value of each element, should be set to false, such as aggregate, rolling join. For others, should be set to false to avoid repeated formatting

start-onyx-join clj

(start-onyx-join num-work
                 batch-size
                 dataframe
                 b
                 source
                 dist
                 exception
                 a-keys
                 b-keys
                 a-roll
                 b-roll
                 join-type
                 limit
                 a-index
                 b-index
                 b-format
                 write-index
                 out)

start the onyx cluster with the specification inside dataframe

submit-and-wait clj

(submit-and-wait job)

Submit one job to the running environment and block until it has finished, rethrowing the job's own exception if it failed. The job map supplies :catalog, :lifecycles and :flow-conditions; the workflow is the one config-env sized the peers for.

v-peers clj

with-onyx-env clj

(with-onyx-env stage num-work f)

Start the Onyx environment (embedded ZooKeeper, peer group and one peer per task) for num-work workers, call f, and shut everything down again whether or not f threw. This is the only place that starts or stops the environment, so every driver gets the same partial-startup handling.

worker-func clj

(worker-func seg)

worker-func-gen clj

(worker-func-gen df exception index)

worker-func-gen-format clj

(worker-func-gen-format df exception index)

workflow clj

workflow-gen clj

(workflow-gen num-work)

Generate workflow for running Onyx