cascalog.cascading.tap — cascalog/cascalog-core 3.0.0

cascalog-tap clj

(cascalog-tap source sink)

fill-tap! clj

(fill-tap! tap xs)

get-sink-tuples clj

(get-sink-tuples sink)

glob-hfs clj

(glob-hfs scheme path-or-file source-pattern)

hfs clj

(hfs scheme path-or-file)

(hfs scheme path-or-file sinkmode)

hfs-seqfile clj

(hfs-seqfile path & opts)

Creates a tap on HDFS using sequence file format. Different filesystems can be selected by using different prefixes for path.

Supports keyword option for :outfields. See cascalog.cascading.tap/hfs-tap for more keyword arguments.

See http://www.cascading.org/javadoc/cascading/tap/Hfs.html and http://www.cascading.org/javadoc/cascading/scheme/SequenceFile.html

hfs-tap clj

(hfs-tap scheme
         path-or-file
         &
         {:keys [sinkmode sinkparts sink-template source-pattern templatefields]
          :as options})

Returns a Cascading Hfs tap with support for the supplied scheme, opened up on the supplied path or file object. Supported keyword options are:

:sinkmode - can be :keep, :update or :replace.

:sinkparts - used to constrain the segmentation of output files.

:source-pattern - Causes resulting tap to respond as a GlobHfs tap when used as source.

:sink-template - Causes resulting tap to respond as a TemplateTap when used as a sink.

:templatefields - When pattern is supplied via :sink-template, this option allows a subset of output fields to be used in the naming scheme.

See f.ex. the http://docs.cascading.org/cascading/2.0/javadoc/cascading/scheme/local/TextDelimited.html scheme.

hfs-textline clj

(hfs-textline path & opts)

Creates a tap on HDFS using textline format. Different filesystems can be selected by using different prefixes for path. Supported keyword options are:

:outfields - used to select the fields written to the tap

:compression - one of :enable, :disable or :default

See cascalog.cascading.tap/hfs-tap for more keyword arguments.

See http://www.cascading.org/javadoc/cascading/tap/Hfs.html and http://www.cascading.org/javadoc/cascading/scheme/TextLine.html

iter-seq clj

(iter-seq iter f)

lfs clj

(lfs scheme path-or-file)

(lfs scheme path-or-file sinkmode)

lfs-seqfile clj

(lfs-seqfile path & opts)

Creates a tap that reads data off of the local filesystem in sequence file format.

Supports keyword option for :outfields. See cascalog.cascading.tap/lfs-tap for more keyword arguments.

See http://www.cascading.org/javadoc/cascading/tap/Lfs.html and http://www.cascading.org/javadoc/cascading/scheme/SequenceFile.html

lfs-tap clj

(lfs-tap scheme
         path-or-file
         &
         {:keys [sinkmode sinkparts sink-template source-pattern templatefields]
          :as options})

Returns a Cascading Lfs tap with support for the supplied scheme, opened up on the supplied path or file object. Supported keyword options are:

:sinkmode - can be :keep, :update or :replace.

:sinkparts - used to constrain the segmentation of output files.

:source-pattern - Causes resulting tap to respond as a GlobHfs tap when used as source.

:sink-template - Causes resulting tap to respond as a TemplateTap when used as a sink.

:templatefields - When pattern is supplied via :sink-template, this option allows a subset of output fields to be used in the naming scheme.

lfs-textline clj

(lfs-textline path & opts)

Creates a tap on the local filesystem using textline format.

Supports keyword option for :outfields. See cascalog.cascading.tap/lfs-tap for more keyword arguments.

See http://www.cascading.org/javadoc/cascading/tap/Lfs.html and http://www.cascading.org/javadoc/cascading/scheme/TextLine.html

memory-source-tap clj

(memory-source-tap tuples)

(memory-source-tap fields-in tuples)

pluck-tuple clj

(pluck-tuple tap)

sequence-file clj

(sequence-file field-names)

set-sinkparts! clj

(set-sinkparts! scheme sinkparts)

If sinkparts is truthy, returns the supplied cascading scheme with the sinkparts field updated appropriately; else, acts as identity. identity.

stdout clj

(stdout)

Creates a tap that prints tuples sunk to it to standard output. Useful for experimentation in the REPL.

template-tap clj

(template-tap parent sink-template)

(template-tap parent
              sink-template
              {:keys [templatefields open-threshold]
               :or {templatefields Fields/ALL open-threshold 300}})

text-line clj

(text-line)

(text-line field-names)

(text-line source-fields sink-fields)

(text-line source-fields sink-fields compression)

valid-sinkmode? clj