ds-test-tools.local-save-intermediate

Liking cljdoc? Tell your friends :D

Clojure only.

*all-names-encountered*
*dir*
*names*
*prevent-loop*
apply-transform
list-transforms
save!

apply-transform^clj

(apply-transform pcoll
                 transform
                 schema
                 {:keys [coder coll-name side-outputs checkpoint] :as options})

A modified version of datasplash.core/apply-transform that will save intermediate input

A modified version of `datasplash.core/apply-transform` that will save intermediate input

source raw docstring

list-transforms^clj

(list-transforms f)

source

save!^clj

(save! f names dir)

Calls f, saving in/out of every transform in names to dir. f should run some local test pipeline. It should be blocking, ie call ds/wait-pipeline-result. names can be a collection of transform names, or it can be :all in which case every transform with a keyword name will be saved. (some internal transforms can't be saved, and these have string names, so as a workaround we filter out those that don't have a keyword name). dir will be tilde expanded for convenience. Note : it might take a while and eat your ram/cpu.

Calls `f`, saving in/out of every transform in `names` to `dir`.
`f` should run some local test pipeline. It should be blocking, ie
call `ds/wait-pipeline-result`.
`names` can be a collection of transform names, or it can be `:all`
in which case every transform with a keyword name will be saved.
(some internal transforms can't be saved, and these have string names,
so as a workaround we filter out those that don't have a keyword name).
`dir` will be tilde expanded for convenience.
Note : it might take a while and eat your ram/cpu.

source raw docstring