Spark's SQL functions from a table, a row per function, which
zero-one.geni.core.functions holds. Each row names the Spark function, the
Spark version that added it, when that's after 3.5, and its argument lists.
A function calls Spark's functions method of that name by reflection, so
zero-one.geni.core loads on any Spark and with only the Spark Connect
client, and it takes whichever of the method's overloads on the classpath
fits its arguments best.
Spark's SQL functions from a table, a row per function, which `zero-one.geni.core.functions` holds. Each row names the Spark function, the Spark version that added it, when that's after 3.5, and its argument lists. A function calls Spark's `functions` method of that name by reflection, so `zero-one.geni.core` loads on any Spark and with only the Spark Connect client, and it takes whichever of the method's overloads on the classpath fits its arguments best.
(def-spark-functions & rows)Defines a function for each row of the table, with the row's Spark name
and version in its metadata, for table. See parse-row. Each defn is a
top-level form of its own, which AOT compilation keeps apart: more forms per
row, or the table as one literal, would be more code than a JVM method can
hold.
Defines a function for each row of the table, with the row's Spark name and version in its metadata, for `table`. See `parse-row`. Each `defn` is a top-level form of its own, which AOT compilation keeps apart: more forms per row, or the table as one literal, would be more code than a JVM method can hold.
The functions' docstrings, keyed by Spark's name, from Spark's
functions.scala, which zero-one.geni.function-docs in dev/ writes.
The functions' docstrings, keyed by Spark's name, from Spark's functions.scala, which `zero-one.geni.function-docs` in `dev/` writes.
(invoke spark-name since geni-name args)(invoke spark-name since geni-name args {:keys [arity-since columns-since]})Calls Spark's functions method spark-name with args, through the
overload that fits them best, after checking that the Spark on the
classpath is at least since, [major minor] or [major minor patch],
when that's given. checks gives the versions that some calls need:
:arity-since, by the number of arguments, the version and the arglist,
and :columns-since, the version that takes a column, rather than only a
value, after the first argument, as Spark 4.0 added to split.
Calls Spark's `functions` method `spark-name` with `args`, through the overload that fits them best, after checking that the Spark on the classpath is at least `since`, `[major minor]` or `[major minor patch]`, when that's given. `checks` gives the versions that some calls need: `:arity-since`, by the number of arguments, the version and the arglist, and `:columns-since`, the version that takes a column, rather than only a value, after the first argument, as Spark 4.0 added to `split`.
(table)The table's rows, as maps keyed by the Geni name, from the metadata of the
functions that def-spark-functions defined, in the namespaces loaded, but
not of their copies under other names, such as ->utc-timestamp.
The table's rows, as maps keyed by the Geni name, from the metadata of the functions that `def-spark-functions` defined, in the namespaces loaded, but not of their copies under other names, such as `->utc-timestamp`.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |