sparkplug.config — amperity/sparkplug-core 0.1.7

Functions for working with Spark configuration.

app-name clj

(app-name conf name-str)

Set the Spark application name. Returns updated configuration.

contains-key? clj

(contains-key? conf k)

True if the given spark configuration contains the named parameter.

debug-str clj

(debug-str conf)

Return a string containing a representation of the configuration useful for debugging.

get-all clj

(get-all conf)

Get all configuration parameters as a map.

get-param clj

(get-param conf k)

(get-param conf k not-found)

Get a configuration parameter k in conf. If not set, this throws a NoSuchElementException or returns not-found if provided.

jars clj

(jars conf jars)

Set JAR files to distribute to the cluster. Returns updated configuration.

master clj

(master conf master)

Set the Spark master property. Returns updated configuration.

merge-params clj

(merge-params conf params)

Merge the provided parameters into the Spark configuration. Returns updated configuration.

set-executor-env clj

(set-executor-env conf env)

(set-executor-env conf k v)

Set environment variables to be used when launching executors for this application. Accepts a parameter key and value or a map of parameters. Returns an updated configuration.

set-param clj

(set-param conf k v)

(set-param conf k v & kvs)

Set a parameter to a new value in the given Spark configuration. Returns updated configuration.

set-param-default clj

(set-param-default conf k v)

Set a parameter to a new value if it is not already set in the config. Returns an updated configuration.

spark-conf clj

(spark-conf)

(spark-conf defaults?)

Construct a new Spark configuration. Optionally accepts a boolean to control whether default configuration is loaded from the system properties.

spark-home clj

(spark-home conf home)

Set the Spark home path property. Returns updated configuration.

unset-param clj

(unset-param conf k)

(unset-param conf k & ks)

Unset the given parameters on the config. Returns an updated config.