(active-session)Spark's active SparkSession on this thread, or else its default one, as long as it's still running. Returns nil rather than creating a session.
Spark's active SparkSession on this thread, or else its default one, as long as it's still running. Returns nil rather than creating a session.
(connect-only?)Whether the Spark Connect client is on the classpath without classic Spark, as it is when Spark 4's spark-connect-client-jvm takes the place of spark-sql.
Whether the Spark Connect client is on the classpath without classic Spark, as it is when Spark 4's spark-connect-client-jvm takes the place of spark-sql.
(connect-session url {:keys [configs]})A new session on a Spark Connect server, which becomes Spark's default and
active session. See g/connect, which also makes Geni use it.
A new session on a Spark Connect server, which becomes Spark's default and active session. See `g/connect`, which also makes Geni use it.
(create-spark-session {:keys [app-name master configs log-level
checkpoint-dir]})The entry point to programming Spark with the Dataset and DataFrame API.
Like Spark's SparkSession.builder().getOrCreate(), it returns the running
session if there is one, and creates one otherwise. The options are:
:app-name and :master, which default to "Geni App" and "local[*]"
unless spark.app.name or spark.master are set already, for instance by
spark-submit.:configs, a map of Spark configs.:log-level, such as "ERROR". Without it, Geni only sets "WARN", and
only when it starts Spark and there's no log4j2 config on the classpath,
as spark-shell does, before Spark logs its INFO lines as it starts.:checkpoint-dir, the SparkContext's checkpoint directory.When it starts a local session, and spark.serializer isn't set, it sets
that to zero_one.geni.rdd.ClojureSerializer: Spark's Java serialisation,
which Spark uses for RDD records, except that a false in a record stays
false. Java's own makes a new Boolean, which Clojure treats as true. A
cluster's executors load their serializer before the application's jars,
so it's left alone there.
When the JVM lacks flags that Spark's launcher sets, it names them, in a warning or in the error if Spark doesn't start.
(g/create-spark-session {:app-name "My App"
:configs {:spark.sql.shuffle.partitions 8}})
The entry point to programming Spark with the Dataset and DataFrame API.
Like Spark's `SparkSession.builder().getOrCreate()`, it returns the running
session if there is one, and creates one otherwise. The options are:
- `:app-name` and `:master`, which default to "Geni App" and "local[*]"
unless `spark.app.name` or `spark.master` are set already, for instance by
spark-submit.
- `:configs`, a map of Spark configs.
- `:log-level`, such as "ERROR". Without it, Geni only sets "WARN", and
only when it starts Spark and there's no log4j2 config on the classpath,
as `spark-shell` does, before Spark logs its INFO lines as it starts.
- `:checkpoint-dir`, the SparkContext's checkpoint directory.
When it starts a local session, and `spark.serializer` isn't set, it sets
that to `zero_one.geni.rdd.ClojureSerializer`: Spark's Java serialisation,
which Spark uses for RDD records, except that a `false` in a record stays
false. Java's own makes a new Boolean, which Clojure treats as true. A
cluster's executors load their serializer before the application's jars,
so it's left alone there.
When the JVM lacks flags that Spark's launcher sets, it names them, in a
warning or in the error if Spark doesn't start.
```clojure
(g/create-spark-session {:app-name "My App"
:configs {:spark.sql.shuffle.partitions 8}})
```(spark-conf spark-session)The session's Spark configs, as a map with keyword keys: Spark's
spark.conf().getAll(). For a classic session, these are the SparkConf's
settings, plus the SQL configs set on the session since. It works for Spark
Connect sessions too.
(:spark.app.name (g/spark-conf spark))
=> "Geni App"
The session's Spark configs, as a map with keyword keys: Spark's `spark.conf().getAll()`. For a classic session, these are the SparkConf's settings, plus the SQL configs set on the session since. It works for Spark Connect sessions too. ```clojure (:spark.app.name (g/spark-conf spark)) => "Geni App" ```
(spark-context spark)The session's SparkContext. Only classic sessions have one: for a Spark Connect session, it throws an error that says so.
The session's SparkContext. Only classic sessions have one: for a Spark Connect session, it throws an error that says so.
(sql spark sql-text)Executes a SQL query using Spark, returning the result as a DataFrame.
The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.
(g/sql spark "SELECT * FROM my_table")
Executes a SQL query using Spark, returning the result as a `DataFrame`. The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'. ```clojure (g/sql spark "SELECT * FROM my_table") ```
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |