A DataFrame's result as Arrow IPC streams, and those streams decoded into the columns that tech.ml.dataset and dtype-next take. Loaded when it's first needed, since it needs Apache Arrow, which classic Spark brings and a Spark Connect client only has shaded. zero-one.geni.core.results has the public functions.
A DataFrame's result as Arrow IPC streams, and those streams decoded into the columns that tech.ml.dataset and dtype-next take. Loaded when it's first needed, since it needs Apache Arrow, which classic Spark brings and a Spark Connect client only has shaded. zero-one.geni.core.results has the public functions.
(classic-streams df mode)The result of df on classic Spark as complete Arrow IPC streams, one per
batch. With :all, one job collects every batch, and an empty result gives
one stream with no rows. With :lazy, the batches come one partition at a
time, a job each, as the iterator is read. Returns a java.util.Iterator.
The result of `df` on classic Spark as complete Arrow IPC streams, one per batch. With `:all`, one job collects every batch, and an empty result gives one stream with no rows. With `:lazy`, the batches come one partition at a time, a job each, as the iterator is read. Returns a java.util.Iterator.
(decode stream schema)The batches in an Arrow IPC stream from Spark, each as a map of its row
count and its columns, which schema, the result's Spark schema, names
and types.
The batches in an Arrow IPC stream from Spark, each as a map of its row count and its columns, which `schema`, the result's Spark schema, names and types.
(decode-tensors stream schema indices caller)The columns at indices of each batch in an Arrow IPC stream, as
tensor-data gives them, for the function named caller.
The columns at `indices` of each batch in an Arrow IPC stream, as tensor-data gives them, for the function named `caller`.
(refuse! col-name reason)Throws for a column that to-tensors can't take, saying why: reason
follows the column's name, as in ", whose type is string".
Throws for a column that to-tensors can't take, saying why: `reason` follows the column's name, as in ", whose type is string".
(tensor-data root schema index)A column of the batch in root, by its index, as an array and a shape:
[rows] for a numeric column, and [rows width] for arrays or dense vectors.
A column of the batch in `root`, by its index, as an array and a shape: [rows] for a numeric column, and [rows width] for arrays or dense vectors.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |