(active-session)Spark's active SparkSession on this thread, or else its default one, as long as it's still running. Returns nil rather than creating a session.
Spark's active SparkSession on this thread, or else its default one, as long as it's still running. Returns nil rather than creating a session.
(create-spark-session {:keys [app-name master configs log-level
checkpoint-dir]})The entry point to programming Spark with the Dataset and DataFrame API.
Like Spark's SparkSession.builder().getOrCreate(), it returns the running
session if there is one, and creates one otherwise. The options are:
:app-name and :master, which default to "Geni App" and "local[*]"
unless spark.app.name or spark.master are set already, for instance by
spark-submit.:configs, a map of Spark configs.:log-level, such as "ERROR". Without it, Geni only sets "WARN", and
only when it starts Spark and there's no log4j2 config on the classpath,
as spark-shell does.:checkpoint-dir, the SparkContext's checkpoint directory.(g/create-spark-session {:app-name "My App"
:configs {:spark.sql.shuffle.partitions 8}})
The entry point to programming Spark with the Dataset and DataFrame API.
Like Spark's `SparkSession.builder().getOrCreate()`, it returns the running
session if there is one, and creates one otherwise. The options are:
- `:app-name` and `:master`, which default to "Geni App" and "local[*]"
unless `spark.app.name` or `spark.master` are set already, for instance by
spark-submit.
- `:configs`, a map of Spark configs.
- `:log-level`, such as "ERROR". Without it, Geni only sets "WARN", and
only when it starts Spark and there's no log4j2 config on the classpath,
as `spark-shell` does.
- `:checkpoint-dir`, the SparkContext's checkpoint directory.
```clojure
(g/create-spark-session {:app-name "My App"
:configs {:spark.sql.shuffle.partitions 8}})
```(spark-conf spark-session)Params:
Result: SparkConf
Return a copy of this JavaSparkContext's configuration. The configuration cannot be changed at runtime.
Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html
Timestamp: 2020-10-19T01:56:49.511Z
Params: Result: SparkConf Return a copy of this JavaSparkContext's configuration. The configuration cannot be changed at runtime. Source: https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/api/java/JavaSparkContext.html Timestamp: 2020-10-19T01:56:49.511Z
(sql spark sql-text)Executes a SQL query using Spark, returning the result as a DataFrame.
The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.
(g/sql spark "SELECT * FROM my_table")
Executes a SQL query using Spark, returning the result as a `DataFrame`. The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'. ```clojure (g/sql spark "SELECT * FROM my_table") ```
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |