(active-session)Spark's active SparkSession on this thread, or else its default one, as long as it's still running. Returns nil rather than creating a session.
Spark's active SparkSession on this thread, or else its default one, as long as it's still running. Returns nil rather than creating a session.
(classic-session? spark)Whether the session is classic Spark's rather than a Spark Connect one.
Whether the session is classic Spark's rather than a Spark Connect one.
(classpath-version)The version of the Spark on the classpath, such as "4.2.0": the Spark Connect client's, over Spark Connect. It says which of Spark's methods Geni can call, where the session's version says what the server runs.
The version of the Spark on the classpath, such as "4.2.0": the Spark Connect client's, over Spark Connect. It says which of Spark's methods Geni can call, where the session's version says what the server runs.
(connect-only?)Whether the Spark Connect client is on the classpath without classic Spark, as it is when Spark 4's spark-connect-client-jvm takes the place of spark-sql.
Whether the Spark Connect client is on the classpath without classic Spark, as it is when Spark 4's spark-connect-client-jvm takes the place of spark-sql.
(connect-session url {:keys [configs keep-classes]})A new session on a Spark Connect server, which becomes Spark's default and
active session. See g/connect, which also makes Geni use it.
A new session on a Spark Connect server, which becomes Spark's default and active session. See `g/connect`, which also makes Geni use it.
(create-spark-session {:keys [app-name master configs log-level
checkpoint-dir]})The entry point to programming Spark with the Dataset and DataFrame API.
Like Spark's SparkSession.builder().getOrCreate(), it returns the running
session if there is one, and creates one otherwise. The options are:
:app-name and :master, which default to "Geni App" and "local[*]"
unless spark.app.name or spark.master are set already, for instance by
spark-submit.:configs, a map of Spark configs.:log-level, such as "ERROR". Without it, Geni only sets "WARN", and
only when it starts Spark and there's no log4j2 config on the classpath,
as spark-shell does, before Spark logs its INFO lines as it starts.:checkpoint-dir, the SparkContext's checkpoint directory.When it starts a local session, and spark.serializer isn't set, it sets
that to zero_one.geni.rdd.ClojureSerializer: Spark's Java serialisation,
which Spark uses for RDD records, except that a false in a record stays
false. Java's own makes a new Boolean, which Clojure treats as true. A
cluster's executors load their serializer before the application's jars,
so it's left alone there.
When the JVM lacks flags that Spark's launcher sets, it names them, in a warning or in the error if Spark doesn't start.
(g/create-spark-session {:app-name "My App"
:configs {:spark.sql.shuffle.partitions 8}})
The entry point to programming Spark with the Dataset and DataFrame API.
Like Spark's `SparkSession.builder().getOrCreate()`, it returns the running
session if there is one, and creates one otherwise. The options are:
- `:app-name` and `:master`, which default to "Geni App" and "local[*]"
unless `spark.app.name` or `spark.master` are set already, for instance by
spark-submit.
- `:configs`, a map of Spark configs.
- `:log-level`, such as "ERROR". Without it, Geni only sets "WARN", and
only when it starts Spark and there's no log4j2 config on the classpath,
as `spark-shell` does, before Spark logs its INFO lines as it starts.
- `:checkpoint-dir`, the SparkContext's checkpoint directory.
When it starts a local session, and `spark.serializer` isn't set, it sets
that to `zero_one.geni.rdd.ClojureSerializer`: Spark's Java serialisation,
which Spark uses for RDD records, except that a `false` in a record stays
false. Java's own makes a new Boolean, which Clojure treats as true. A
cluster's executors load their serializer before the application's jars,
so it's left alone there.
When the JVM lacks flags that Spark's launcher sets, it names them, in a
warning or in the error if Spark doesn't start.
```clojure
(g/create-spark-session {:app-name "My App"
:configs {:spark.sql.shuffle.partitions 8}})
```(require-version! needed what)Throws an error that names what and the Spark version it needs, as
[major minor] or [major minor patch], when the Spark on the classpath is
older.
Throws an error that names `what` and the Spark version it needs, as `[major minor]` or `[major minor patch]`, when the Spark on the classpath is older.
(spark-conf spark-session)The session's Spark configs, as a map with keyword keys: Spark's
spark.conf().getAll(). For a classic session, these are the SparkConf's
settings, plus the SQL configs set on the session since. It works for Spark
Connect sessions too.
(:spark.app.name (g/spark-conf spark))
=> "Geni App"
The session's Spark configs, as a map with keyword keys: Spark's `spark.conf().getAll()`. For a classic session, these are the SparkConf's settings, plus the SQL configs set on the session since. It works for Spark Connect sessions too. ```clojure (:spark.app.name (g/spark-conf spark)) => "Geni App" ```
(spark-context spark)The session's SparkContext. Only classic sessions have one: for a Spark Connect session, it throws an error that says so.
The session's SparkContext. Only classic sessions have one: for a Spark Connect session, it throws an error that says so.
(sql spark sql-text)(sql spark sql-text args)Executes a SQL query using Spark, returning the result as a DataFrame.
Spark runs a command, such as CREATE TABLE, right away, and a query when
an action needs it.
With args, the query's parameters are bound to values rather than spliced
into the text: a map binds the named parameters, such as :min, and a
vector binds the ? ones in order. A value is a literal, a column such as
(g/lit ...), or a collection, which becomes an array as g/lit makes
one: of arrays when it nests, with its numbers widened as Clojure's
arithmetic widens them, and of DECIMAL(38, 18) for decimals. From Spark
4.0, a value can also be a column that builds an array, a map or a struct of
literals, such as (g/map (g/lit "k") (g/lit 1)). Spark 4.1.0 to 4.1.3 and
4.2.0 bind more than four ? parameters in the wrong order (SPARK-58341),
so on those sql throws an error for more than four, and named parameters
work instead.
(g/sql spark "SELECT * FROM my_table")
(g/sql spark "SELECT * FROM sales WHERE price > :min" {:min 1000})
(g/sql spark "SELECT ? + ?" [2 3])
Executes a SQL query using Spark, returning the result as a `DataFrame`.
Spark runs a command, such as `CREATE TABLE`, right away, and a query when
an action needs it.
With `args`, the query's parameters are bound to values rather than spliced
into the text: a map binds the named parameters, such as `:min`, and a
vector binds the `?` ones in order. A value is a literal, a column such as
`(g/lit ...)`, or a collection, which becomes an array as `g/lit` makes
one: of arrays when it nests, with its numbers widened as Clojure's
arithmetic widens them, and of DECIMAL(38, 18) for decimals. From Spark
4.0, a value can also be a column that builds an array, a map or a struct of
literals, such as `(g/map (g/lit "k") (g/lit 1))`. Spark 4.1.0 to 4.1.3 and
4.2.0 bind more than four `?` parameters in the wrong order (SPARK-58341),
so on those `sql` throws an error for more than four, and named parameters
work instead.
```clojure
(g/sql spark "SELECT * FROM my_table")
(g/sql spark "SELECT * FROM sales WHERE price > :min" {:min 1000})
(g/sql spark "SELECT ? + ?" [2 3])
```cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |