Runs an Onyx job inside the calling JVM: no ZooKeeper, no Aeron, no peers and no JVM flags. Takes the same job map as onyx.api/submit-job, with the same plugins, task functions, lifecycles, windows and triggers.
Each task runs on its own thread. A task's :onyx/fn is applied to batches on a pool of :onyx/n-peers (else :onyx/max-peers, else :parallelism) threads, and its results are emitted in input order. A task with several parents takes one batch from each parent in turn, in workflow order. So for deterministic inputs and functions, the output is the same on every run, order included.
Windows and triggers run on Onyx's own window executors with an in-memory state store, one per task (group-by keys become groups in it). Watermarks travel with the batches instead of with barriers: an input task's watermark is the largest :onyx/assign-watermark-fn value (or WatermarkedInput watermark) read so far, any other task's is the smallest of its parents' latest, and a windowed task fires a :watermark state event after any batch that advanced it. So watermark triggers fire at the same point in the data on every run; in Onyx they fire on the next barrier, which depends on timing. Timer triggers are only evaluated when a batch arrives and at job completion.
Not supported, and rejected rather than ignored: flow conditions, Java tasks, and lifecycle phases other than :lifecycle/before-task-start and :lifecycle/after-task-stop. There is no checkpointing: an exception anywhere fails the whole run.
Runs an Onyx job inside the calling JVM: no ZooKeeper, no Aeron, no peers and no JVM flags. Takes the same job map as onyx.api/submit-job, with the same plugins, task functions, lifecycles, windows and triggers. Each task runs on its own thread. A task's :onyx/fn is applied to batches on a pool of :onyx/n-peers (else :onyx/max-peers, else :parallelism) threads, and its results are emitted in input order. A task with several parents takes one batch from each parent in turn, in workflow order. So for deterministic inputs and functions, the output is the same on every run, order included. Windows and triggers run on Onyx's own window executors with an in-memory state store, one per task (group-by keys become groups in it). Watermarks travel with the batches instead of with barriers: an input task's watermark is the largest :onyx/assign-watermark-fn value (or WatermarkedInput watermark) read so far, any other task's is the smallest of its parents' latest, and a windowed task fires a :watermark state event after any batch that advanced it. So watermark triggers fire at the same point in the data on every run; in Onyx they fire on the next barrier, which depends on timing. Timer triggers are only evaluated when a batch arrives and at job completion. Not supported, and rejected rather than ignored: flow conditions, Java tasks, and lifecycle phases other than :lifecycle/before-task-start and :lifecycle/after-task-stop. There is no checkpointing: an exception anywhere fails the whole run.
(run-job job)(run-job {:keys [catalog workflow] :as job} opts)Runs job, an onyx.api/submit-job job map, to completion in this JVM and returns {:segments {task-name number-of-segments-it-emitted}}. Blocks until every input is completed and every output has written everything; a job with a streaming input never returns. Throws the first exception any task throws, after stopping the others.
Options: :parallelism threads per task without :onyx/n-peers or :onyx/max-peers (default: available processors) :queue-size batches buffered per workflow edge (default 8) :fn-params as :onyx.peer/fn-params in a peer config
Runs job, an onyx.api/submit-job job map, to completion in this JVM and
returns {:segments {task-name number-of-segments-it-emitted}}. Blocks
until every input is completed and every output has written everything;
a job with a streaming input never returns. Throws the first exception
any task throws, after stopping the others.
Options:
:parallelism threads per task without :onyx/n-peers or :onyx/max-peers
(default: available processors)
:queue-size batches buffered per workflow edge (default 8)
:fn-params as :onyx.peer/fn-params in a peer configcljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |