The SparkSession that Geni functions use when they aren't given one. Requiring Geni doesn't start Spark: the session is looked up, or created, when a function first needs it.
The SparkSession that Geni functions use when they aren't given one. Requiring Geni doesn't start Spark: the session is looked up, or created, when a function first needs it.
(connect)(connect url)(connect url opts)Connects to a Spark Connect server, and returns a SparkSession for it:
Spark 4's SparkSession.builder().remote(url).create(). It needs Spark's
JVM client, org.apache.spark/spark-connect-client-jvm_2.13, on the
classpath in place of spark-sql. See the Spark Connect guide.
url, such as "sc://localhost:15002", can also hold a token and other
options, as in "sc://host:443/;use_ssl=true;token=...". Without it, the
client reads the SPARK_REMOTE environment variable, or else connects to
sc://localhost:15002.:configs, a map of Spark SQL configs to set on the session.Each call starts a new session on the server, which becomes Spark's default
and active session, and the one that Geni uses, in place of any session
passed to set-default-session!. Keep it, and call .close on it when
you're done.
(g/connect "sc://localhost:15002")
(g/connect "sc://localhost:15002" {:configs {:spark.sql.shuffle.partitions 8}})
Connects to a Spark Connect server, and returns a SparkSession for it:
Spark 4's `SparkSession.builder().remote(url).create()`. It needs Spark's
JVM client, `org.apache.spark/spark-connect-client-jvm_2.13`, on the
classpath in place of spark-sql. See the Spark Connect guide.
- `url`, such as "sc://localhost:15002", can also hold a token and other
options, as in "sc://host:443/;use_ssl=true;token=...". Without it, the
client reads the `SPARK_REMOTE` environment variable, or else connects to
sc://localhost:15002.
- `:configs`, a map of Spark SQL configs to set on the session.
Each call starts a new session on the server, which becomes Spark's default
and active session, and the one that Geni uses, in place of any session
passed to `set-default-session!`. Keep it, and call `.close` on it when
you're done.
```clojure
(g/connect "sc://localhost:15002")
(g/connect "sc://localhost:15002" {:configs {:spark.sql.shuffle.partitions 8}})
```(set-default-session! spark)Makes spark the SparkSession that Geni functions use when they aren't
given one, and returns it. Pass nil to go back to Spark's active session.
(g/set-default-session! (g/create-spark-session {:app-name "My App"}))
Makes `spark` the SparkSession that Geni functions use when they aren't
given one, and returns it. Pass nil to go back to Spark's active session.
```clojure
(g/set-default-session! (g/create-spark-session {:app-name "My App"}))
```The default SparkSession, which Geni functions use when they aren't given
one. Deref it to get the session: the one passed to set-default-session!,
or else Spark's active session, or else a new local one. Geni configures
only the sessions it creates, and then only as create-spark-session
describes. With only the Spark Connect client on the classpath, the new
session connects to SPARK_REMOTE, as (connect) does.
The default SparkSession, which Geni functions use when they aren't given one. Deref it to get the session: the one passed to `set-default-session!`, or else Spark's active session, or else a new local one. Geni configures only the sessions it creates, and then only as `create-spark-session` describes. With only the Spark Connect client on the classpath, the new session connects to `SPARK_REMOTE`, as `(connect)` does.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |