clojure-hadoop.flow
cleanup-step-mergeclj
(cleanup-step-merge spec)If we override source/sink key-value pairs in a step, we'll get a list back so here we take the last element to act like a normal merge. We keep the lists for :config and :params
If we override source/sink key-value pairs in a step, we'll get a list back so here we take the last element to act like a normal merge. We keep the lists for :config and :params
define-flowcljmacro
(define-flow name [& args] & body)A flow is a function which dispatches a set of jobs dictated by branching and other logic embedded around dispatch-job statements. It should return a boolean value indicating success.
A flow is a function which dispatches a set of jobs dictated by branching and other logic embedded around dispatch-job statements. It should return a boolean value indicating success.
define-shufflecljmacro
(define-shuffle name [& args] & keyvals)Shuffle simplifies flows by specifying the map-writer and corresponding reduce-reader as well as any input-output key types necessary to specify. It can be used in define-step using the :shuffle key.
Shuffle simplifies flows by specifying the map-writer and corresponding reduce-reader as well as any input-output key types necessary to specify. It can be used in define-step using the :shuffle key.
define-sinkcljmacro
(define-sink name [& arglist] & kvs)A sink is a function stored in the component registry which takes zero or more arguments and returns a map. Any keys which are valid for defjob and define-step can be included here. Any keys included in the including step will override the default key-value pairs in define-sink. Sinks are defined using the :sink key to define-step expressions.
A sink is a function stored in the component registry which takes zero or more arguments and returns a map. Any keys which are valid for defjob and define-step can be included here. Any keys included in the including step will override the default key-value pairs in define-sink. Sinks are defined using the :sink key to define-step expressions.
define-sourcecljmacro
(define-source name [& args] & kvs)A source is a named function which takes arguments and returns a map of parameter settings to be merged in with the main job definition. It is meant to be supplied to the step argument :source as a keyword name, or as a keyword/symbol in a simply application expression such as :source :text or :source (:dbfmt <jdbc-spec>)
A source is a named function which takes arguments and returns a map of parameter settings to be merged in with the main job definition. It is meant to be supplied to the step argument :source as a keyword name, or as a keyword/symbol in a simply application expression such as :source :text or :source (:dbfmt <jdbc-spec>)
define-stepcljmacro
(define-step name [& args] & keyvals)A step is a function that composes key-value pair definitions from sources, sinks and shuffles. It is otherwise identical to a job.
A step is a function that composes key-value pair definitions from sources, sinks and shuffles. It is otherwise identical to a job.
get-componentclj
(get-component type name)(get-component type name errorp)Get a component to apply when step is evaluated. Components are :source, :sink, and :shuffle
Get a component to apply when step is evaluated. Components are :source, :sink, and :shuffle
job-counter-valueclj
(job-counter-value job group-name counter-name)Utility function to help work with counters stored in completed jobs
Utility function to help work with counters stored in completed jobs
put-componentclj
(put-component type name comp)Register a component for acquisition in later macro expansions
Register a component for acquisition in later macro expansions
run-flowclj
(run-flow flow args)Run a flow. We'll put some seatbelts in here later, for now any uncaught exceptions result in a failed flow.
Run a flow. We'll put some seatbelts in here later, for now any uncaught exceptions result in a failed flow.
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close