zero-one.geni.streaming — zero.one/geni 0.0.33

->duration clj

(->duration value)

->java-dstream clj

(->java-dstream dstream)

->time clj

(->time value)

await-termination! clj

(await-termination! context)

await-termination-or-timeout! clj

(await-termination-or-timeout! context timeout)

cache clj

(cache dstream)

checkpoint clj multimethod

compute clj

(compute dstream t)

context clj

(context dstream)

count clj

(count dstream)

count-by-value clj

(count-by-value dstream)

(count-by-value dstream num-partitions)

disk-only clj

Flag for controlling the storage of an RDD.

DataFrame is stored only on disk and the CPU computation time is high as I/O involved.

disk-only-2 clj

Flag for controlling the storage of an RDD.

Same as disk-only storage level but replicate each partition to two cluster nodes.

dstream clj

filter clj

(filter dstream f)

flat-map clj

(flat-map dstream f)

flat-map-to-pair clj

(flat-map-to-pair dstream f)

flat-map-values clj

(flat-map-values dstream f)

get-state clj

glom clj

(glom dstream)

map clj

(map dstream f)

map-partitions clj

(map-partitions dstream f)

map-partitions-to-pair clj

(map-partitions-to-pair dstream f)

map-to-pair clj

(map-to-pair dstream f)

mapcat-to-pair clj

memory-and-disk clj

Flag for controlling the storage of an RDD.

The default behavior of the DataFrame or Dataset. In this Storage Level, The DataFrame will be stored in JVM memory as deserialized objects. When required storage is greater than available memory, it stores some of the excess partitions into a disk and reads the data from disk when it required. It is slower as there is I/O involved.

memory-and-disk-2 clj

Flag for controlling the storage of an RDD.

Same as memory-and-disk storage level but replicate each partition to two cluster nodes.

memory-and-disk-ser clj

Flag for controlling the storage of an RDD.

Same as memory-and-disk storage level difference being it serializes the DataFrame objects in memory and on disk when space not available.

memory-and-disk-ser-2 clj

Flag for controlling the storage of an RDD.

Same as memory-and-disk-ser storage level but replicate each partition to two cluster nodes.

memory-only clj

Flag for controlling the storage of an RDD.

memory-only-2 clj

Flag for controlling the storage of an RDD.

Same as memory-only storage level but replicate each partition to two cluster nodes.

memory-only-ser clj

Flag for controlling the storage of an RDD.

Same as memory-only but the difference being it stores RDD as serialized objects to JVM memory. It takes lesser memory (space-efficient) then memory-only as it saves objects as serialized and takes an additional few more CPU cycles in order to deserialize.

memory-only-ser-2 clj

Flag for controlling the storage of an RDD.

Same as memory-only-ser storage level but replicate each partition to two cluster nodes.

milliseconds clj

(milliseconds t)

minutes clj

(minutes t)

none clj

Flag for controlling the storage of an RDD.

No caching.

off-heap clj

Flag for controlling the storage of an RDD.

Off-heap refers to objects (serialised to byte array) that are managed by the operating system but stored outside the process heap in native memory (therefore, they are not processed by the garbage collector). Accessing this data is slightly slower than accessing the on-heap storage but still faster than reading/writing from a disk. The downside is that the user has to manually deal with managing the allocated memory.

persist clj

(persist dstream)

(persist dstream storage-level)

print clj

(print dstream)

(print dstream num)

reduce clj

(reduce dstream f)

reduce-by-key clj

(reduce-by-key dstream f)

(reduce-by-key dstream f partitions-or-partitioner)

remember clj

(remember context duration)

repartition clj

(repartition dstream num-partitions)

save-as-text-files! clj multimethod

seconds clj

(seconds t)

slide-duration clj

(slide-duration dstream)

socket-text-stream clj

(socket-text-stream context hostname port storage)

spark-context clj

(spark-context context)

ssc clj

(ssc context)

start! clj

(start! context)

state clj

(state context)

stop! clj

(stop! context)

streaming-context clj

(streaming-context spark duration)

text-file-stream clj

(text-file-stream context path)

union clj

(union left right)

wrap-rdd clj

(wrap-rdd dstream rdd)