AST definition and compiler for #dt/e expressions.
#dt/e is a reader tag that produces an AST map. datajure.core interprets these ASTs when executing dt queries. This namespace handles:
Op names are stored as keywords in the AST (e.g. :and, :>, :+) rather than symbols, because the Clojure compiler tries to resolve symbols in literal data structures. Since and/or/not are macros, the compiler rejects 'Can't take value of a macro' when it encounters them as bare symbols in the map values returned by the reader tag. Keywords are self-evaluating and avoid this entirely.
Nil-safety rules (matching spec):
AST definition and compiler for #dt/e expressions.
#dt/e is a reader tag that produces an AST map. datajure.core interprets
these ASTs when executing dt queries. This namespace handles:
- AST node constructors
- compile-expr: AST -> fn of dataset -> column/scalar
- Reader tag handler registered via resources/data_readers.clj (primary)
and register-reader! / alter-var-root (AOT/script fallback)
Op names are stored as keywords in the AST (e.g. :and, :>, :+) rather than
symbols, because the Clojure compiler tries to resolve symbols in literal
data structures. Since and/or/not are macros, the compiler rejects
'Can't take value of a macro' when it encounters them as bare symbols in
the map values returned by the reader tag. Keywords are self-evaluating
and avoid this entirely.
Nil-safety rules (matching spec):
- Comparison ops with nil arg -> false column (all rows false)
- Arithmetic ops with nil arg -> nil (becomes missing when stored in dataset)
These rules only activate when a Clojure nil literal appears in an expression.
Dataset columns with missing values are handled natively by dfn.(col-max col)Column maximum, skipping nil/missing; nil for an all-missing column.
dfn/reduce-max corrupts the reduction when the column has missing values
(a missing slot poisons the running max), so we filter first and use
clojure.core/max. Backs both the #dt/e :mx op and core/max*.
Column maximum, skipping nil/missing; nil for an all-missing column. `dfn/reduce-max` corrupts the reduction when the column has missing values (a missing slot poisons the running max), so we filter first and use `clojure.core/max`. Backs both the #dt/e `:mx` op and `core/max*`.
(col-min col)Column minimum, skipping nil/missing; nil for an all-missing column.
See col-max for why dfn/reduce-min isn't used. Backs :mi and core/min*.
Column minimum, skipping nil/missing; nil for an all-missing column. See [[col-max]] for why `dfn/reduce-min` isn't used. Backs `:mi` and `core/min*`.
(col-node kw)Create a column reference AST node. kw is a keyword naming the column.
Create a column reference AST node. `kw` is a keyword naming the column.
(col-refs node)Extract the set of column keywords referenced by an AST node. If a :lit node contains an embedded expr-node (from expression composition), recursively extracts its col-refs.
Extract the set of column keywords referenced by an AST node. If a :lit node contains an embedded expr-node (from expression composition), recursively extracts its col-refs.
(compile-expr node)(compile-expr node env)Compile an AST node to a fn [ds] -> column/scalar. Column keywords resolve to dataset columns; literals pass through; ops dispatch via op-table to dfn functions.
Nil-safety: if any arg to an op evaluates to nil (e.g. a nil literal), comparison ops return an all-false boolean column; arithmetic ops return nil (which becomes a missing value when stored in a dataset column).
Compile an AST node to a fn [ds] -> column/scalar. Column keywords resolve to dataset columns; literals pass through; ops dispatch via op-table to dfn functions. Nil-safety: if any arg to an op evaluates to nil (e.g. a nil literal), comparison ops return an all-false boolean column; arithmetic ops return nil (which becomes a missing value when stored in a dataset column).
(count-distinct col)Count of distinct non-nil values in a column.
Count of distinct non-nil values in a column.
(count-non-nil col)Count of non-nil values in a column. Backs the #dt/e :ct op and core/count*.
Count of non-nil values in a column. Backs the #dt/e `:ct` op and `core/count*`.
(damerau-levenshtein s t)Damerau-Levenshtein edit distance: insertions, deletions, substitutions, and single adjacent transpositions each cost 1. Used by both the #dt/e op-name validator (suggest-op) and core's column-name validator (validate-expr-cols/validate-select-cols) to produce typo suggestions.
Single source of truth — core.clj re-uses this via expr/damerau-levenshtein rather than maintaining a duplicate implementation.
Damerau-Levenshtein edit distance: insertions, deletions, substitutions, and single adjacent transpositions each cost 1. Used by both the #dt/e op-name validator (suggest-op) and core's column-name validator (validate-expr-cols/validate-select-cols) to produce typo suggestions. Single source of truth — core.clj re-uses this via expr/damerau-levenshtein rather than maintaining a duplicate implementation.
(data->ast form)(data->ast form _ctx)Convert a runtime data-form expression to a #dt/e AST so it compiles down the same vectorized path. The form mirrors #dt/e one-for-one, as plain evaluated data — one expression language, two spellings:
qnt);ticker, and
(data->ast [:win/lag :price 1]) is exactly #dt/e (win/lag :price 1). Context
rules (e.g. win/* only in :set) are enforced by dt on the resulting AST, the
same as for #dt/e. Use sets (not vectors) for :in membership, since a
non-number-headed vector denotes an operation.The ctx argument of the 2-arity is retained for call compatibility and
ignored — the op vocabulary is uniform across contexts, exactly like #dt/e.
Convert a runtime data-form expression to a #dt/e AST so it compiles down the
same vectorized path. The form mirrors #dt/e one-for-one, as plain *evaluated*
data — one expression language, two spellings:
- a vector [op-kw & args] is an operation. op-kw is the keyword spelling of
any #dt/e op — element-wise (:>, :div0, :na2zero, …), aggregator (:mn,
:qnt, :nrow, … including every full-name/concise alias), window/row/stat
(:win/lag, :row/sum, :stat/winsorize, …);
- the special forms are vector-headed too: [:if pred then else?],
[:cond p1 v1 … :else d], [:let [:name expr …] body],
[:coalesce …]/[:coalesce-finite …], [:cut :col n]/[:cut :col n :from pred],
[:xbar :col w]/[:xbar :col w :minutes], [:win/scan :* expr],
[:win/each-prior :- expr];
- a **number-headed** vector is a literal value (e.g. a probability list
[0.2 0.5 0.8] for a multi-quantile `qnt`);
- a keyword is a column reference (or a [:let] binding reference in a body);
- anything else (number, string, set, the value of a local) is a literal.
So (data->ast [:= :tic ticker]) closes over the runtime value of `ticker`, and
(data->ast [:win/lag :price 1]) is exactly #dt/e (win/lag :price 1). Context
rules (e.g. win/* only in :set) are enforced by dt on the resulting AST, the
same as for #dt/e. Use sets (not vectors) for `:in` membership, since a
non-number-headed vector denotes an operation.
The `ctx` argument of the 2-arity is retained for call compatibility and
ignored — the op vocabulary is uniform across contexts, exactly like #dt/e.(div0 num den)Nil-safe scalar division: returns nil when the numerator or denominator is nil,
or the denominator is zero; otherwise num / den as a double. Non-numeric
inputs throw normally — this guards data absence, not programmer errors.
Backs both the #dt/e :div0 op and core/div0.
Nil-safe scalar division: returns nil when the numerator or denominator is nil, or the denominator is zero; otherwise `num` / `den` as a double. Non-numeric inputs throw normally — this guards data absence, not programmer errors. Backs both the #dt/e `:div0` op and `core/div0`.
(expr-node? x)Returns true if x is a #dt/e AST node (a map with :node/type).
Returns true if x is a #dt/e AST node (a map with :node/type).
(lit-node v)Create a literal value AST node. v is any scalar (number, string, keyword, set, etc.).
Create a literal value AST node. `v` is any scalar (number, string, keyword, set, etc.).
(op-node op args)Create an operation AST node. op is an op keyword (e.g. :+, :>), args is a seq of child AST nodes.
Create an operation AST node. `op` is an op keyword (e.g. :+, :>), `args` is a seq of child AST nodes.
(read-expr form)Reader tag handler for #dt/e. Returns an AST map. Called at read time.
Reader tag handler for #dt/e. Returns an AST map. Called at read time.
(register-reader!)Register the #dt/e reader tag via alter-var-root on data-readers. Fallback for AOT compilation or scripts where data_readers.clj is not picked up at startup. The primary registration mechanism is resources/data_readers.clj, which Clojure merges automatically for all threads at JVM startup.
Register the #dt/e reader tag via alter-var-root on *data-readers*. Fallback for AOT compilation or scripts where data_readers.clj is not picked up at startup. The primary registration mechanism is resources/data_readers.clj, which Clojure merges automatically for all threads at JVM startup.
(row-node row-op args)AST node for a row-wise function call. row-op is a keyword like :row/sum. args are parsed AST nodes.
AST node for a row-wise function call. row-op is a keyword like :row/sum. args are parsed AST nodes.
(stat-node stat-op args)AST node for a stat/* function call. stat-op is a keyword like :stat/standardize. args are parsed AST nodes.
AST node for a stat/* function call. stat-op is a keyword like :stat/standardize. args are parsed AST nodes.
(wavg w v)Weighted average. Args: weight-col value-col. Skips nil pairs. Throws :unequal-column-lengths when w and v have different lengths.
Weighted average. Args: weight-col value-col. Skips nil pairs. Throws :unequal-column-lengths when w and v have different lengths.
(win-node win-op args)AST node for a window function call. win-op is a keyword like :win/rank. args are parsed AST nodes.
AST node for a window function call. win-op is a keyword like :win/rank. args are parsed AST nodes.
(win-op-fn win-op-kw)The runtime window fn for a :win/<op> keyword, or nil. Lets callers (e.g. the
core group-set fast path) apply a single-column window op directly to a column
slice without compiling/running the whole AST.
The runtime window fn for a `:win/<op>` keyword, or nil. Lets callers (e.g. the core group-set fast path) apply a single-column window op directly to a column slice without compiling/running the whole AST.
(win-refs node)Extract the set of window op keywords (e.g. :win/rank) referenced by an AST node. If a :lit node contains an embedded expr-node (from expression composition), recursively extracts its win-refs.
Extract the set of window op keywords (e.g. :win/rank) referenced by an AST node. If a :lit node contains an embedded expr-node (from expression composition), recursively extracts its win-refs.
(wsum w v)Weighted sum. Args: weight-col value-col. Skips nil pairs. Throws :unequal-column-lengths when w and v have different lengths.
Weighted sum. Args: weight-col value-col. Skips nil pairs. Throws :unequal-column-lengths when w and v have different lengths.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |