zero-one.geni.spark-context — zero.one/geni 0.0.40

app-name clj

(app-name)

(app-name spark)

Params:

Result: String

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.487Z

binary-files clj multimethod

Params: (path: String, minPartitions: Int)

Result: JavaPairRDD[String, PortableDataStream]

Read a directory of binary files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI as a byte array. Each file is read as a single record and returned in a key-value pair, where the key is the path of each file, the value is the content of each file.

For example, if you have the following files:

Do

then rdd contains

A suggestion value of the minimal splitting number for input data.

Small files are preferred; very large files but may cause bad performance.

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.492Z

broadcast clj

(broadcast value)

(broadcast spark value)

Params: (value: T)

Result: Broadcast[T]

Broadcast a read-only variable to the cluster, returning a org.apache.spark.broadcast.Broadcast object for reading it in distributed functions. The variable will be sent to each cluster only once.

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.495Z

checkpoint-dir clj

(checkpoint-dir)

(checkpoint-dir spark)

Params:

Result: Optional[String]

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.509Z

conf clj

(conf)

(conf spark)

Params:

Result: SparkConf

Return a copy of this JavaSparkContext's configuration. The configuration cannot be changed at runtime.

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.511Z

default-min-partitions clj

(default-min-partitions)

(default-min-partitions spark)

Params:

Result: Integer

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.503Z

default-parallelism clj

(default-parallelism)

(default-parallelism spark)

Params:

Result: Integer

Default level of parallelism to use when not given by user (e.g. parallelize and makeRDD).

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.504Z

empty-rdd clj

(empty-rdd)

(empty-rdd spark)

Params:

Result: JavaRDD[T]

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.505Z

get-checkpoint-dir clj

(get-checkpoint-dir)

(get-checkpoint-dir spark)

Params:

Result: Optional[String]

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.509Z

get-conf clj

(get-conf)

(get-conf spark)

Params:

Result: SparkConf

Return a copy of this JavaSparkContext's configuration. The configuration cannot be changed at runtime.

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.511Z

get-local-property clj

(get-local-property k)

(get-local-property spark k)

Params: (key: String)

Result: String

Get a local property set in this thread, or null if it is missing. See org.apache.spark.api.java.JavaSparkContext.setLocalProperty.

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.512Z

get-persistent-rd-ds clj

(get-persistent-rd-ds)

(get-persistent-rd-ds spark)

Params:

Result: Map[Integer, JavaRDD[_]]

Returns a Java map of JavaRDDs that have marked themselves as persistent via cache() call.

This does not necessarily mean the caching or computation was successful.

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.513Z

get-persistent-rdds clj

(get-persistent-rdds)

(get-persistent-rdds spark)

Params:

Result: Map[Integer, JavaRDD[_]]

Returns a Java map of JavaRDDs that have marked themselves as persistent via cache() call.

This does not necessarily mean the caching or computation was successful.

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.513Z

get-spark-home clj

(get-spark-home)

(get-spark-home spark)

Params: ()

Result: Optional[String]

Get Spark's home location from either a value set through the constructor, or the spark.home Java property, or the SPARK_HOME environment variable (in that order of preference). If neither of these is set, return None.

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.518Z

is-local clj

(is-local)

(is-local spark)

Params:

Result: Boolean

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.531Z

jars clj

(jars)

(jars spark)

Params:

Result: List[String]

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.532Z

java-spark-context clj

(java-spark-context spark)

Converts a SparkSession to a JavaSparkContext.

local-property clj

(local-property k)

(local-property spark k)

Params: (key: String)

Result: String

Get a local property set in this thread, or null if it is missing. See org.apache.spark.api.java.JavaSparkContext.setLocalProperty.

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.512Z

local? clj

(local?)

(local? spark)

Params:

Result: Boolean

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.531Z

master clj

(master)

(master spark)

Params:

Result: String

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.532Z

parallelise clj

(parallelise data)

(parallelise spark data)

Params: (list: List[T], numSlices: Int)

Result: JavaRDD[T]

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.544Z

parallelise-doubles clj

(parallelise-doubles data)

(parallelise-doubles spark data)

Params: (list: List[Double], numSlices: Int)

Result: JavaDoubleRDD

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.546Z

parallelise-pairs clj

(parallelise-pairs data)

(parallelise-pairs spark data)

Params: (list: List[(K, V)], numSlices: Int)

Result: JavaPairRDD[K, V]

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.549Z

parallelize clj

(parallelize data)

(parallelize spark data)

Params: (list: List[T], numSlices: Int)

Result: JavaRDD[T]

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.544Z

parallelize-doubles clj

(parallelize-doubles data)

(parallelize-doubles spark data)

Params: (list: List[Double], numSlices: Int)

Result: JavaDoubleRDD

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.546Z

parallelize-pairs clj

(parallelize-pairs data)

(parallelize-pairs spark data)

Params: (list: List[(K, V)], numSlices: Int)

Result: JavaPairRDD[K, V]

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.549Z

persistent-rdds clj

(persistent-rdds)

(persistent-rdds spark)

Params:

Result: Map[Integer, JavaRDD[_]]

Returns a Java map of JavaRDDs that have marked themselves as persistent via cache() call.

This does not necessarily mean the caching or computation was successful.

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.513Z

resources clj

(resources)

(resources spark)

Params:

Result: Map[String, ResourceInformation]

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.550Z

sc clj

(sc)

(sc spark)

Params:

Result: SparkContext

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.550Z

spark-context clj

(spark-context)

(spark-context spark)

Params:

Result: SparkContext

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.550Z

spark-home clj

(spark-home)

(spark-home spark)

Params: ()

Result: Optional[String]

Get Spark's home location from either a value set through the constructor, or the spark.home Java property, or the SPARK_HOME environment variable (in that order of preference). If neither of these is set, return None.

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.518Z

text-file clj multimethod

Params: (path: String)

Result: JavaRDD[String]

Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings. The text files must be encoded as UTF-8.

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.570Z

value clj

memfn of value

version clj

(version)

(version spark)

Params:

Result: String

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.576Z

whole-text-files clj multimethod

Params: (path: String, minPartitions: Int)

Result: JavaPairRDD[String, String]

Read a directory of text files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI. Each file is read as a single record and returned in a key-value pair, where the key is the path of each file, the value is the content of each file. The text files must be encoded as UTF-8.

For example, if you have the following files:

Do

then rdd contains

A suggestion value of the minimal splitting number for input data.

Small files are preferred, large file is also allowable, but may cause bad performance.

Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html

Timestamp: 2020-10-19T01:56:49.582Z