Point-in-time as-of joins, built from Onyx's own windows, triggers and refinements, so they run unchanged in onyx.replay and on a cluster.
Segments of a point-in-time input carry :pit/known, the time (epoch ms or java.util.Date) their information became available: a filing's acceptance time, a data vintage, a quote's timestamp. Right-hand segments may also carry :pit/event, the time the information is about (a fiscal period end); it defaults to :pit/known.
An as-of join emits each left segment with :pit/as-of set to the right segment with the same join key that has the latest :pit/event among those with :pit/known no later than the left segment's :pit/known (ties go to the later :pit/known, so a restatement replaces the original from the moment it is known, and then to the later segment), or nil when none was known yet. It emits a left segment only once the join task's watermark has passed its :pit/known, that is, once every right segment that could match has arrived. So no result uses information that was not known at its time.
Each input must be ordered by :pit/known. A right segment that arrives after the join has emitted past its :pit/known would have changed results already emitted, so the join throws instead.
Build the inputs with input, the join with as-of-join, and check a
job with check-job before running it.
Point-in-time as-of joins, built from Onyx's own windows, triggers and refinements, so they run unchanged in onyx.replay and on a cluster. Segments of a point-in-time input carry :pit/known, the time (epoch ms or java.util.Date) their information became available: a filing's acceptance time, a data vintage, a quote's timestamp. Right-hand segments may also carry :pit/event, the time the information is about (a fiscal period end); it defaults to :pit/known. An as-of join emits each left segment with :pit/as-of set to the right segment with the same join key that has the latest :pit/event among those with :pit/known no later than the left segment's :pit/known (ties go to the later :pit/known, so a restatement replaces the original from the moment it is known, and then to the later segment), or nil when none was known yet. It emits a left segment only once the join task's watermark has passed its :pit/known, that is, once every right segment that could match has arrived. So no result uses information that was not known at its time. Each input must be ordered by :pit/known. A right segment that arrives after the join has emitted past its :pit/known would have changed results already emitted, so the join throws instead. Build the inputs with `input`, the join with `as-of-join`, and check a job with `check-job` before running it.
(as-of-join task-name key batch-size)The catalog entry, window and trigger of an as-of join task named task-name that joins its :left and :right parents on key.
The catalog entry, window and trigger of an as-of join task named task-name that joins its :left and :right parents on key.
Aggregation: per join key, the right segments so far, the left segments not yet emitted, and the time emission has reached.
Aggregation: per join key, the right segments so far, the left segments not yet emitted, and the time emission has reached.
(check-job {:keys [workflow catalog windows triggers] :as job})Returns job if each as-of join in it has what it needs to be point in
time, else throws: a group-by key, the trigger and refinement from
as-of-join, and only upstream inputs built with input, at least one
per side. An input with another watermark could let the join emit
before every right segment known by then has arrived.
Returns job if each as-of join in it has what it needs to be point in time, else throws: a group-by key, the trigger and refinement from as-of-join, and only upstream inputs built with `input`, at least one per side. An input with another watermark could let the join emit before every right segment known by then has arrived.
Refinement: drops the left segments the trigger emitted.
Refinement: drops the left segments the trigger emitted.
(emit-ready _ _ _ state-event {:keys [right pending]})Trigger emit: the pending left segments the watermark has passed, each with its :pit/as-of match.
Trigger emit: the pending left segments the watermark has passed, each with its :pit/as-of match.
(input task-map side)Makes task-map, an input catalog entry, the side (:left or :right) of an as-of join: its watermark is :pit/known and it tags segments with :pit/side.
Makes task-map, an input catalog entry, the side (:left or :right) of an as-of join: its watermark is :pit/known and it tags segments with :pit/side.
(known-ms segment)A segment's :pit/known in epoch ms. Use as :onyx/assign-watermark-fn.
A segment's :pit/known in epoch ms. Use as :onyx/assign-watermark-fn.
Trigger: fires on every watermark and at job completion.
Trigger: fires on every watermark and at job completion.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |