The session's runtime configs: Spark's spark.conf(), which Spark SQL
reads as queries run, over Spark Connect too. g/spark-conf returns all
the configs that are set.
The session's runtime configs: Spark's `spark.conf()`, which Spark SQL reads as queries run, over Spark Connect too. `g/spark-conf` returns all the configs that are set.
(conf-get k)(conf-get k default)(conf-get spark k)(conf-get spark k default)Returns the value of the config k as a string. Without a default, it's
the value that's set, or else Spark's own default for the config, or nil
when Spark doesn't know it. With a default, it's the value that's set, or
else default, whatever Spark's own default is.
(g/conf-get "spark.sql.shuffle.partitions")
=> "200"
(g/conf-get spark :spark.sql.ansi.enabled)
Returns the value of the config `k` as a string. Without a `default`, it's the value that's set, or else Spark's own default for the config, or nil when Spark doesn't know it. With a `default`, it's the value that's set, or else `default`, whatever Spark's own default is. ```clojure (g/conf-get "spark.sql.shuffle.partitions") => "200" (g/conf-get spark :spark.sql.ansi.enabled) ```
(conf-modifiable? k)(conf-modifiable? spark k)Returns true when the session can set the config k: Spark knows it, and
it isn't static. A config Spark doesn't know returns false, though
conf-set! still sets it.
Returns true when the session can set the config `k`: Spark knows it, and it isn't static. A config Spark doesn't know returns false, though `conf-set!` still sets it.
(conf-set! configs)(conf-set! k value)(conf-set! spark configs)(conf-set! spark k value)Sets the config k to value, or each config in a map of keys to values,
for the session. A value can be a string, a number, a boolean or a keyword.
Spark refuses a static config, such as spark.sql.warehouse.dir, and a value
of the wrong type for a config it knows.
(g/conf-set! "spark.sql.shuffle.partitions" 8)
(g/conf-set! spark {:spark.sql.ansi.enabled false})
Sets the config `k` to `value`, or each config in a map of keys to values,
for the session. A value can be a string, a number, a boolean or a keyword.
Spark refuses a static config, such as `spark.sql.warehouse.dir`, and a value
of the wrong type for a config it knows.
```clojure
(g/conf-set! "spark.sql.shuffle.partitions" 8)
(g/conf-set! spark {:spark.sql.ansi.enabled false})
```(conf-unset! k)(conf-unset! spark k)Unsets the config k, so that it goes back to Spark's own default.
Unsets the config `k`, so that it goes back to Spark's own default.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |