hadoop-util.transfer — hadoop-util 0.3.0

Namespace responsible for recursively transferring directories from a distributed filestore (or another local filestore) to the local filesystem on the current machine.

check-in clj

(check-in throttle-agent kbs)

Report the current downloaded number of kilobytes to the supplied agent.

copy clj multimethod

file-type clj

(file-type fs path)

Accepts a hadoop filesystem object and some path and returns a namespace-qualified type keyword.

rcopy clj

(rcopy remote-fs remote-path target-path & {:keys [throttle]})

Copies information at the supplied remote-path over to the supplied local-path.

Arguments are Filesystem, remote shard path, target local path, and an optional throttling agent.

sleep-interval clj

(sleep-interval throttle-agent)

Returns the current sleep interval specified by the supplied throttling agent.

throttle clj

(throttle kb-per-second)

Returns a throttling agent. Any positive kb-per-second rate will cause throttling; if the rate is zero or negative, downloads will proceed without a throttle.

try-times clj macro

(try-times [n] & body)

Executes body. If an exception is thrown, will retry. At most n retries are done. If still some exception is thrown it is bubbled upwards in the call chain.

try-times* clj

(try-times* n thunk)

Executes thunk. If an exception is thrown, will retry. At most n retries are done. If still some exception is thrown it is bubbled upwards in the call chain.

update-limit clj

(update-limit throttle-agent new-limit)

Updates the throttling agent's rate limit (in kb/s).