(active-session)Spark's active SparkSession on this thread, or else its default one, as long as it's still running. Returns nil rather than creating a session.
Spark's active SparkSession on this thread, or else its default one, as long as it's still running. Returns nil rather than creating a session.
(connect-only?)Whether the Spark Connect client is on the classpath without classic Spark, as it is when Spark 4's spark-connect-client-jvm takes the place of spark-sql.
Whether the Spark Connect client is on the classpath without classic Spark, as it is when Spark 4's spark-connect-client-jvm takes the place of spark-sql.
(connect-session url {:keys [configs]})A new session on a Spark Connect server, which becomes Spark's default and
active session. See g/connect, which also makes Geni use it.
A new session on a Spark Connect server, which becomes Spark's default and active session. See `g/connect`, which also makes Geni use it.
(create-spark-session {:keys [app-name master configs log-level
checkpoint-dir]})The entry point to programming Spark with the Dataset and DataFrame API.
Like Spark's SparkSession.builder().getOrCreate(), it returns the running
session if there is one, and creates one otherwise. The options are:
:app-name and :master, which default to "Geni App" and "local[*]"
unless spark.app.name or spark.master are set already, for instance by
spark-submit.:configs, a map of Spark configs.:log-level, such as "ERROR". Without it, Geni only sets "WARN", and
only when it starts Spark and there's no log4j2 config on the classpath,
as spark-shell does.:checkpoint-dir, the SparkContext's checkpoint directory.(g/create-spark-session {:app-name "My App"
:configs {:spark.sql.shuffle.partitions 8}})
The entry point to programming Spark with the Dataset and DataFrame API.
Like Spark's `SparkSession.builder().getOrCreate()`, it returns the running
session if there is one, and creates one otherwise. The options are:
- `:app-name` and `:master`, which default to "Geni App" and "local[*]"
unless `spark.app.name` or `spark.master` are set already, for instance by
spark-submit.
- `:configs`, a map of Spark configs.
- `:log-level`, such as "ERROR". Without it, Geni only sets "WARN", and
only when it starts Spark and there's no log4j2 config on the classpath,
as `spark-shell` does.
- `:checkpoint-dir`, the SparkContext's checkpoint directory.
```clojure
(g/create-spark-session {:app-name "My App"
:configs {:spark.sql.shuffle.partitions 8}})
```(spark-conf spark-session)The session's Spark configs, as a map with keyword keys: Spark's
spark.conf().getAll(). For a classic session, these are the SparkConf's
settings, plus the SQL configs set on the session since. It works for Spark
Connect sessions too.
(:spark.app.name (g/spark-conf spark))
=> "Geni App"
The session's Spark configs, as a map with keyword keys: Spark's `spark.conf().getAll()`. For a classic session, these are the SparkConf's settings, plus the SQL configs set on the session since. It works for Spark Connect sessions too. ```clojure (:spark.app.name (g/spark-conf spark)) => "Geni App" ```
(spark-context spark)The session's SparkContext. Only classic sessions have one: for a Spark Connect session, it throws an error that says so.
The session's SparkContext. Only classic sessions have one: for a Spark Connect session, it throws an error that says so.
(sql spark sql-text)Executes a SQL query using Spark, returning the result as a DataFrame.
The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.
(g/sql spark "SELECT * FROM my_table")
Executes a SQL query using Spark, returning the result as a `DataFrame`. The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'. ```clojure (g/sql spark "SELECT * FROM my_table") ```
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |