What a Spark Connect server needs to run a Clojure UDF, which goes up
through the client's addArtifact, each piece once per session: Clojure's
and Geni's jars, the jar or the source directory of each namespace that the
function uses, and the classes that Clojure kept for it, with g/connect's
:keep-classes, when it was compiled at the REPL. The server loads a
namespace that has a file from that file, as a cluster's executors load it
from the application's jar. A session's server keeps the first version of
each class and file that it gets, so code that changed after it went up
throws an error, rather than running as it was.
What a Spark Connect server needs to run a Clojure UDF, which goes up through the client's `addArtifact`, each piece once per session: Clojure's and Geni's jars, the jar or the source directory of each namespace that the function uses, and the classes that Clojure kept for it, with `g/connect`'s `:keep-classes`, when it was compiled at the REPL. The server loads a namespace that has a file from that file, as a cluster's executors load it from the application's jar. A session's server keeps the first version of each class and file that it gets, so code that changed after it went up throws an error, rather than running as it was.
(check-keep-classes!)Throws when keep-classes! couldn't set Clojure's compile vars for this
thread, since it has another thread's bindings of them, as a future has
its caller's. g/connect checks before it starts a session.
Throws when `keep-classes!` couldn't set Clojure's compile vars for this thread, since it has another thread's bindings of them, as a future has its caller's. `g/connect` checks before it starts a session.
(keep-classes!)Has Clojure write the classes that it compiles from now on into a
temporary directory, as compile does, so that a UDF of a function
defined at the REPL can go to a Spark Connect server: what g/connect's
:keep-classes does. It sets the roots of *compile-files* and
*compile-path*, so Clojure compiles that way in every thread, and this
thread's bindings of them. Another thread that binds *compile-path*, as
each REPL does, writes into its own, which has to exist. Returns the
directory, which goes when the JVM exits.
Has Clojure write the classes that it compiles from now on into a temporary directory, as `compile` does, so that a UDF of a function defined at the REPL can go to a Spark Connect server: what `g/connect`'s `:keep-classes` does. It sets the roots of `*compile-files*` and `*compile-path*`, so Clojure compiles that way in every thread, and this thread's bindings of them. Another thread that binds `*compile-path*`, as each REPL does, writes into its own, which has to exist. Returns the directory, which goes when the JVM exits.
(open-sessions)The Spark Connect sessions that g/connect made, or that UDFs went to,
and that are still open.
The Spark Connect sessions that `g/connect` made, or that UDFs went to, and that are still open.
(register-session! session)Notes a session that g/connect made, so that UDFs go to it, as well as
to the default session.
Notes a session that `g/connect` made, so that UDFs go to it, as well as to the default session.
(upload-udf! session f)Uploads to session what its Spark Connect server needs to run f, a
function or a var, as a UDF, each artifact once: Clojure's and Geni's jars,
the jars or source directories of the namespaces that f uses, and the
classes that Clojure kept for the functions that f holds. Throws an error
that says what to do when the server can't get a class that f needs,
and when something that went to the session changed since.
Uploads to `session` what its Spark Connect server needs to run `f`, a function or a var, as a UDF, each artifact once: Clojure's and Geni's jars, the jars or source directories of the namespaces that `f` uses, and the classes that Clojure kept for the functions that `f` holds. Throws an error that says what to do when the server can't get a class that `f` needs, and when something that went to the session changed since.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |