clj-headlights.pipeline — com.zendesk/clj-headlights 0.1.4

apply-to-value-and-output clj

(apply-to-value-and-output context value _window state clj-call)

apply-to-value-and-output-kv clj

(apply-to-value-and-output-kv context value _window state clj-call)

apply-to-value-and-output-with-timestamp clj

(apply-to-value-and-output-with-timestamp context value _window state clj-call)

apply-to-value-and-seq-outputs clj

(apply-to-value-and-seq-outputs context value _window state clj-call)

apply-to-value-and-seq-outputs-kv clj

(apply-to-value-and-seq-outputs-kv context value _window state clj-call)

co-group-by-key clj

(co-group-by-key pcolls name)

Inputs: [pcolls :- (s/cond-pre PCollection [PCollection]) name :- s/Str] Returns: PCollection

Takes a list of PCollections which are already in KV form, and joins them on their keys. Output pcollection will look like: [key [first-pcoll-vals second-pcoll-vals third-pcoll-vals &c]]

composite clj

(composite name inputs f)

Inputs: [name :- s/Str inputs :- [pcollections/PCollectionType] f :- IFn]

Nests transforms that happen in f into a composite transform

context->ms clj

(context->ms c)

create clj

(create options)

Inputs: [options :- PipelineOptions] Returns: Pipeline

Create a Pipeline object. This is the first building block of any Beam pipeline.

df-apply-dofn clj

(df-apply-dofn pcoll name clj-call)

Inputs: [pcoll name clj-call]

df-filter clj

(df-filter pcoll name clj-call)

Inputs: [pcoll :- pcollections/PCollectionType name :- s/Str clj-call :- clj-fn-call/CljCall] Returns: PCollection

Returns a PCollection containing the elements of pcoll where (f element) is truthy

df-filter-kv clj

(df-filter-kv pcoll name clj-call)

Inputs: [pcoll :- pcollections/PCollectionType name :- s/Str clj-call :- clj-fn-call/CljCall] Returns: PCollection

returns a PCollection containing the elements of pcoll where (f element) is true. coerces to KV

df-map clj

(df-map pcoll name clj-call)

Inputs: [pcoll :- pcollections/PCollectionType name :- s/Str clj-call :- clj-fn-call/CljCall] Returns: PCollection

Returns a PCollection of the return values of function clj-call being applied to the input pcoll - used for strictly 1-to-1 transformations

df-map-cat-with-side-outputs clj

(df-map-cat-with-side-outputs pcoll name clj-call tags)

Inputs: [pcoll :- pcollections/PCollectionType name :- s/Str clj-call :- clj-fn-call/CljCall tags :- [s/Keyword]] Returns: PCollectionTuple

Acts like a map-cat but returns a PCollectionTuple partitioned by the tag of the outputs of f. f must return a sequence of tuples in the form [:tag value]. The main output should use the tag :main. Nil values are filtered out

df-map-kv clj

(df-map-kv pcoll name clj-call)

Inputs: [pcoll :- pcollections/PCollectionType name :- s/Str clj-call :- clj-fn-call/CljCall]

Same as df-map but expects the output of f to be a sequence of length 2 which is converted to a Dataflow KV object. Mostly used before a GroupBy

df-map-with-side-input clj

(df-map-with-side-input pcoll name side-input-view clj-call)

Inputs: [pcoll :- pcollections/PCollectionType name :- s/Str side-input-view clj-call :- clj-fn-call/CljCall] Returns: PCollection

df-map-with-timestamp clj

(df-map-with-timestamp pcoll name clj-call)

Inputs: [pcoll :- pcollections/PCollectionType name :- s/Str clj-call :- clj-fn-call/CljCall]

Assigns a timestamp to pcoll rows

df-mapcat clj

(df-mapcat pcoll name clj-call)

Inputs: [pcoll :- pcollections/PCollectionType name :- s/Str clj-call :- clj-fn-call/CljCall] Returns: PCollection

Similar to df-map but the return-values of f are flattened in the output PCollection - used when the transformation is not strictly 1-to-1

df-mapcat-kv clj

(df-mapcat-kv pcoll name clj-call)

Inputs: [pcoll :- pcollections/PCollectionType name :- s/Str clj-call :- clj-fn-call/CljCall]

Same as df-map-kv except f returns a sequence of lenght-2-sequences to be flattened and converted into KV objects

filterer clj

(filterer context value _window state clj-call)

filterer-kv clj

(filterer-kv context value _window state clj-call)

flatten-pcollections clj

(flatten-pcollections pcolls name)

Inputs: [pcolls :- [PCollection] name :- s/Str] Returns: PCollection

Merge a list of pcolls.

get-side-output clj

get-side-outputs clj

group-by-key clj

(group-by-key pcoll name)

Inputs: [pcoll :- PCollection name :- s/Str] Returns: PCollection

make-pipeline-options clj

(make-pipeline-options)

Inputs: [] Returns: PipelineOptions

Create a PipelineOptions object.

output-is-kv clj

(output-is-kv pcoll)

process-pcoll clj

(process-pcoll pcoll name wrapping-call clj-call)

process-pcoll-kv clj

(process-pcoll-kv pcoll name wrapping-call clj-call)

side-output-function-wrapper clj

(side-output-function-wrapper context val _window state clj-call)