All notable changes to Datajure will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
A performance and crash fix, reported from the Weinstein project. Upgrade if
you run dt on large or freshly derived datasets.
dt call hashed its whole input dataset. The not-a-dataset
check added in 2.7.5 built its "did you forget the dataset?" hint on every
call, and that hint tested (contains? dt-query-keys dataset), which hashes
the dataset by reading every value of every column. On a 15.2M-row,
8-column panel, (dt ds :select [:id]) took about 1.6 s instead of about
1 ms, and every step of a pipeline paid it again. Lazy columns derived by
:set were forced too. The hint is now built only when the argument is not
a dataset, and contains? is only tried on a keyword.dt threw a NullPointerException on an all-missing comparison
column. Hashing reads raw values and ignores the missing bitmap, so a
lazy column such as [:> :x :sma] over an all-missing :sma (a series
shorter than its window) crashed the next dt call on that dataset. Fixed
by the same change. Affects every version since 2.7.5.A correctness release, from reviewing 2.7.9. Aggregators now return nil
when there is nothing to aggregate, as the README and spec always said.
One result change: sm over an empty or all-missing group is now nil
instead of 0.0; use (coalesce (sm :x) 0.0) to keep the zero. Also fixes
coalesce over scalars and a crash on a one-boolean :where.
sm, sd and variance returned a number when there was nothing to
aggregate. On an empty or all-missing group, sm gave 0.0 and sd/
variance gave -0.0, although the README and spec promise nil, as
mx/mi/prod already do. sd/variance of a single value now also give
nil (they were NaN, shown as missing only once stored). Results change:
a group whose values are all missing now has a missing sum instead of 0.0;
write #dt/e (coalesce (sm :x) 0.0) to keep the zero. A real zero, such as
the variance of a constant column, is unchanged. Covers #dt/e, the data
forms, and the plain helpers core/mean, sum, stddev, variance (and the
concise mn/sm/sd), which also returned NaN/0.0/-0.0 instead of
nil.coalesce over scalars returned a column. In :agg,
#dt/e (coalesce (mn :x) 0) put a vector of the group's length into the
cell instead of a number; it also never applied, since the aggregator's
NaN is not nil. When every argument is a scalar, coalesce now returns
a scalar.:where predicate that evaluates to one boolean crashed.
:where #dt/e (> (sm :x) 1) threw a ClassCastException; it now keeps
every row when true and none when false or nil.A correctness release, from writing the datajure-website v2 pages. Results
change only where they were wrong: composed window expressions now respect
:by and :within-order, and a missing value inside a computed argument
is treated as missing everywhere. No API changes; upgrade if you compose
stored #dt/e expressions or pass arithmetic into window ops, ct/mx/
nuniq, coalesce or row/*.
#dt/e expression containing a window op, composed into
another expression, was computed over the whole dataset. With
(def ret #dt/e (- (win/ratio :price) 1)) and
(def log-ret #dt/e (log (+ 1 ret))) (the README's "Why Datajure"
example), :by [:permno] :set {:log-ret log-ret} ignored the groups: each
firm's first row got a value computed from the previous firm's last price
instead of nil. Without :by, the same composed expression ignored
:within-order. Writing the expression inline was unaffected. Affects
every version since 2.4.0 (:by) and 2.7.0 (:within-order without
:by). A composed window expression is now also correctly rejected as a
computed :by key.(def w #dt/e 20) in
(win/mavg :price w), or a stored probability in (qnt :x p), threw a
ClassCastException from the single-column fast paths.(win/scan * (+ 1 :ret)) (the README's
wealth index) was missing on every row once any return was missing, and
the same expression stored as a column first gave a different answer than
inline. Affected, when the argument was computed: win/cumsum, cummin,
cummax, cummean, mavg, msum, mdev, mmin, mmax, ema,
fills, bfill, scan; ct, mx, mi, prod, nuniq, wavg,
wsum, cov, corr, tstat; coalesce, in; all row/*;
stat/rescale. A computed argument now reaches these exactly as it would
after being stored as a column (where NaN is missing). Arithmetic,
comparisons, mn/sm/md/qnt/sd/variance and the finite-aware
cleaners already handled NaN and skip the conversion.:select [1], a typo
on a dataset with integer column names, or a malformed :within-order
entry threw a raw ClassCastException/NullPointerException while
building the suggestion; they now throw the structured :unknown-column
error.Did you mean: :height (edit distance 1) line and an Available: line
(first 20 names; the full list stays in :dt/available), as the README
already showed; before, the suggestion was only in :dt/closest.
:within-order and prepare-grouping unknown columns now get
suggestions too.Two additive syntax features from an external review of the query DSL (its remaining suggestions were documentation-only or rejected). No breaking changes — a drop-in upgrade.
lit — the explicit literal escape, in both expression spellings:
#dt/e (= :status (lit :active)) and the data-form [:lit x]. A keyword
under lit stays a value instead of becoming a column reference, and a
keyword-headed vector stays data instead of parsing as an operation — the
data-form trap where a runtime value of keyword type silently became a
column ref (worst case colliding with a real column name) now has an
unambiguous spelling. lit changes parsing only: a sequential literal is
still sequence-like to the vectorized ops (membership against collection
values stays a set + in). Wrong arity throws the structured
:wrong-arity error from both spellings.:by — a :by vector entry may be a
single-entry map naming an element-wise expression as a group key, in
either spelling: (dt sales :by [:region {:year #dt/e (year :date)}] :agg {…}) or {:year [:year :date]}. The key is derived once at the dt
boundary and substituted by name, so the grouping rides the keyword-only
fast paths; it appears as the group-key output column in :agg and is
dropped from window-mode :set output. Aggregators and win/* are
rejected as keys (:by-expr-not-elementwise — circular), as are
multi-entry maps (:by-computed-multi-entry), name collisions with an
existing column / :set target / sibling key
(:by-computed-name-collision), and non-expression values
(:by-computed-invalid-expr). cut/xbar keep their dedicated marker
forms (a cut key's breakpoint population depends on the other :by
keys, which a pre-derived column would silently change). The :expr-in-by
error now points at the named-map form.:set rule (map = independent
derivations, seq of pairs = sequential, later pairs see earlier results)
is now taught up front in "The Key Insight", instead of only via the
cross-reference error message; the stale "unknown op sqrt" error example
(stale since sqrt became a real op) was replaced.A single-fix correctness release, from the post-2.7.6 consistency audit.
Affects every version since 2.7.0 (:set) and 2.7.6 (:agg); upgrade if you
combine a prepared-grouping :by with :where in one dt call.
:where combined with a prepared-grouping :by in one
call silently computed over wrong rows (since 2.7.0 for :set, 2.7.6 for
:agg): the grouping's row-count guard ran at the dt call boundary
against the pre-:where dataset, but :where executes first, so the stale
permutation was applied to the filtered rows — a 4-row grouping over a
3-row filter even produced 4 output rows. The guard now runs at the apply
site, against the dataset the grouped step actually receives: a
row-dropping :where throws the structured :grouping-row-mismatch (with
a hint to filter in a separate dt call before prepare-grouping), while
an all-rows-kept :where remains valid (same rows, same order).The second momentum-replication batch, closing out the wrds-scratch
DATAJURE-NOTES backlog (items 6-8): the grouping amortization extended to
:agg, scalar reads as one fn, and the log-return vocabulary. No breaking
changes (one error-data key renamed: :grouping-requires-set →
:grouping-requires-set-or-agg).
dt1 — scalar reads from single-row results — same arguments as dt,
returns the single result row as a plain map, replacing the
(first (ds/mapseq-reader (dt …))) ceremony around whole-table :agg.
Throws a structured :dt1-not-single-row when the result doesn't have
exactly one row (silently taking the first of several would mask a
grouping/filter bug). Exported from datajure.concise too. (From the
wrds-scratch momentum scripts, where the ceremony appeared four times.)log1p / expm1 element-wise ops — the log-return pair:
(log1p :ret) = (log (+ 1.0 :ret)) but numerically better near zero,
expm1 inverts back to simple returns. One-for-one dfn delegations with
the same read-time checks and fast-path treatment as log/exp.:by now works for :agg (was :set-only,
rejected with :grouping-requires-set) — fast-group-agg consumes the
prebuilt perm/bounds, skipping both the per-call group-index build and the
per-group :within-order sort. Real pipelines regroup by the same keys
repeatedly; on 2M rows × 20k groups each reused :agg pass runs ~3.5–4×
faster. The guard's error key is now :grouping-requires-set-or-agg
(thrown only when a prepared grouping is passed with neither :set nor
:agg).The consumer-feedback release: structured input validation from quanajure's
smoke testing, plus the first harvest from the wrds-scratch JT93 momentum
replication (3.39M CRSP stock-months) — a column-ordering fix, the correlation
vocabulary the replication had to reconstruct by hand, and a memory-hazard
removal in the general grouped-:set path. No breaking changes.
cov, corr — bivariate aggregation ops — sample covariance and sample
Pearson correlation as 2-arg aggregations next to wavg/wsum, in #dt/e
and data-forms ((corr :x :y), [:cov :x :y]): skip pairs where either
value is nil (wavg's convention), nil when fewer than two complete pairs
remain; corr additionally returns nil when either column is constant over
the complete pairs (zero variance — guarded like div0, not thrown).
Unequal column lengths throw the structured :unequal-column-lengths;
wrong arity is a read-time :wrong-arity. Also exported from
datajure.concise. (From the wrds-scratch momentum replication — the one
statistic in the whole JT93 replication that had to leave dt.)
tstat — one-sample t-statistic aggregation — mean / (sd / sqrt n)
with sample sd over the non-nil values; nil when fewer than two non-nil
values remain or the column is constant. The most common summary line in
asset-pricing tables, now one op instead of assembled arithmetic.
Structured :not-a-dataset guard on every dataset-taking data-path entry
point — passing a non-dataset where a dataset belongs (a plain column map, a
sequence of maps, nil, anything else) previously died with a raw NPE
(dt + :where, the crash ironically inside the typo-suggestion code), a
raw protocol/cast error (:select, seq-of-maps), a misleading
:unknown-column error (nil), or — worst — silently returned the input
unchanged ((dt {:a [1 2 3]}) with no query keys). All now throw
{:dt/error :not-a-dataset :dt/fn … :dt/got …} with a shape-aware hint:
maps and seqs of maps point at tech.v3.dataset/->dataset, nil asks
whether an upstream step returned nil, and a lone query map passed to dt
gets "did you forget the dataset?". Guarded: dt, prepare-grouping,
rename, join (both sides, side-specific messages), melt, cast,
tsfill, index-by, io/write, and all util dataset fns (describe,
clean-column-names, duplicate-rows, distinct-rows, mark-duplicates,
drop-constant-columns, coerce-columns, blank->nil, parse-numeric).
The display-only clay/clerk viewers are deliberately unguarded — they fail
visibly at render time. (Reported from quanajure smoke testing.)
reshape/cast orders pivoted columns by the :from VALUES — new
columns now come out in the natural sort order of the distinct :from
values (numeric 1..10 → :1 :2 … :10), not in first-encounter row order,
which followed whatever order the input rows arrived in (a grouped :agg
feeding cast emitted deciles as :9 :1 :6 …). Value-order, not
keyword-name order — lexicographic keyword sort would put :10 between
:1 and :2. Mutually incomparable :from values fall back to the old
first-encounter order. (From the wrds-scratch momentum replication's
decade × decile table.):set path narrows to the referenced columns — when
every derivation is an introspectable expression (the cross-referencing
seq-of-pairs case), the general path now projects the dataset down to the
:by keys, :within-order columns, and the columns the derivations
reference before grouping (mirroring the :agg path's narrow-for-aggs),
re-attaching the untouched columns afterwards — removing the wide-data
memory hazard the :group-set-general-path NOTE warns about. Plain-fn
derivations (which may read any column) still see the full dataset; the
NOTE's wording now reflects both cases.The syntax-review release: a design review of the DSL against its own principles ("rich built-in primitives for daily operations", "one name per concept"), plus a second external-review round — vocabulary gaps filled, consistency warts removed, and several silent-failure modes converted to structured errors. Contains breaking renames — no deprecation aliases are kept.
abs, exp, sqrt, pow, floor, ceil,
round, signum as element-wise ops in #dt/e and data-forms, completing
the arithmetic vocabulary around the existing sq/log/asinh (log
finally has its inverse; abs unlocks nearest-date idioms like
(abs (- (day :date) 15))). One-for-one dfn delegations with the same
nil story as sq/log; round returns longs. All are window-fast-path
eligible (element-wise), guarded by read-time :wrong-arity checks (unary,
pow binary — the same checks now also cover sq/log/asinh and the
cleaners) and the :arith-nil-literal nil-literal rejection.util/distinct-rows — one-call dedup (data.table unique(DT) / dplyr
distinct): keeps the FIRST occurrence per duplicate group, preserves input
row order, keeps all columns; optional column subset restricts the key.
Complements duplicate-rows/mark-duplicates.:day ==
:days in xbar (both contexts), win/tlag {:unit …}, tsfill
:every, and join :tolerance / :window specs. Previously the
fixed-duration surfaces were plural-only (:minutes) while the calendar
surfaces were singular-only (:month) — a gratuitous spelling split.
Normalisation lives in the new shared math/canonical-unit +
math/ms-per-unit (one ms-table instead of three copies); genuinely
unsupported units (e.g. :month in a fixed-duration context) still throw
their structured errors.max/min/count in expression-head position — #dt/e (max :x)
== (mx :x), (min :x) == (mi :x), (count :x) == (ct :x), and the
data-form heads [:max :x]/[:min :x]/[:count :x] likewise (external
review follow-up). Inside #dt/e symbols are a closed vocabulary — the
clojure.core-shadowing rationale for the star names doesn't apply there —
and the SQL mapping is exact: COUNT(x)/COUNT(*)/COUNT(DISTINCT x) ≅
(count :x)/(nrow)/(count-distinct :x). Unary only: 2+ args throw a
read-time :wrong-arity pointing at row/max/row/min for the row-wise
case. Implemented as a separate expression-head alias table so the
win/scan/win/each-prior operator slots (where bare max/min mean the
element-wise binary :max/:min) and their data-form spellings
([:win/scan :max …]) are untouched. The star names remain the canonical
callable-fn spellings.win/ema domain validation — out-of-range smoothing parameters
previously produced silent plausible-looking garbage: (win/ema :x 0)
froze at the seed, a negative shorthand oscillated, {:period 0} computed
α = 2 and diverged. Now structured errors: :alpha must be a finite number
in (0, 1] (:ema-invalid-alpha), :period finite ≥ 1
(:ema-invalid-period), the numeric shorthand finite and positive
(:ema-invalid-param).when-finite on a non-numeric guard threw a raw ClassCastException;
it now throws a structured :when-finite-non-numeric error stating that
when-finite is a numeric guard and pointing at the plain-fn + pass-nil
route for non-numeric presence guards.(mx :a :b),
(mean :a :b), etc. (all spellings of mn/sm/md/sd/variance/mx/
mi/ct/nuniq/prod/first-val/last-val) previously failed at
runtime with a raw ArityException; now a read-time :wrong-arity.concise/ct drift — it aliased dtype/ecount (counting missing slots
too) while the docs and the #dt/e ct op both define ct as the non-nil
count. It now aliases core/count*.:selector-misplaced — a standalone
cut marker used in :set/:where/:agg (i.e. a forgotten #dt/e), or a
col-range/prepared-grouping map inside a :by vector, previously
look-up-missed every row silently (maps are callable): an all-nil :set
column, a zero-row :where, a one-group :by. Each now throws a structured
error pointing at the correct form — closing the guard the cut unification
would otherwise have weakened (the old standalone cut stub at least threw).cut unified across contexts; qtile removed. cut is now one name
for equal-count binning in both contexts, exactly like xbar: inside
#dt/e it derives a column of bins (unchanged); standalone in :by it is
the grouping marker that qtile used to be (was: standalone cut threw
:cut-standalone-not-supported). Everything qtile supported carries over
verbatim — per-partition breakpoints with exact keys, :from reference
subpopulations, <col>-q<n> auto-naming, :datajure/col override — and
the two contexts still share breakpoints by construction. Migration:
(qtile :mktcap 5 …) → (cut :mktcap 5 …). Error keys renamed:
:qtile-invalid-n/:qtile-invalid-col/:qtile-invalid-from →
:cut-invalid-n/:cut-invalid-col/:cut-invalid-from; the internal
marker selector is :dt/selector :cut.between (the positional :select column-range selector) renamed to
col-range. (dt ds :select (col-range :month-01 :month-12)). The old
name was one character away from the #dt/e value predicate between?
while meaning something entirely different — a near-collision with
unrelated semantics. between? is unchanged. The marker selector is
:dt/selector :col-range; datajure.concise re-exports col-range.year, month, day, dow (alias day-of-week), quarter in #dt/e and data-forms (data.table year()/month(), Polars .dt.*). Element-wise long extraction from date / date-time columns via long-temporal-field, with a nil guard: a missing date yields nil (:set stores it as a missing slot in a typed :int64 column; predicates compare it false) instead of leaking the packed Long/MIN_VALUE sentinel. dow is ISO/java.time (1 = Monday … 7 = Sunday); quarter derives 1–4 from the month. Raw :instant columns throw (zone-less — no calendar fields); convert to local dates first. Calendar grouping idiom: derive with :set, then group :by — (-> rets (dt :set {:m #dt/e (month :date)}) (dt :by [:m] :agg {…})). Wrong arity is a read-time :wrong-arity error.:object readers are nil-safe. >, <, >=, <= (and between?) over the object readers produced by nil-producing element-wise ops (date-parts, nonfin2na/neg2na cleaners) previously NPE'd inside dfn's ordering predicates on the first nil element — e.g. #dt/e (>= (nonfin2na :x) 1.0). A nil element now compares false, matching the DSL's nil rule and how the same comparison behaves after a :set round-trip. Equality was already nil-tolerant; typed columns and scalars keep the vectorized dfn fast path.:by now throw a structured :expr-in-by error. A #dt/e AST (or data-form vector) in :by previously fell into the custom-grouping-fn branch, where invoking the AST map as a function returned nil for every row — the whole dataset silently collapsed into ONE group. The error points at the :set-then-:by idiom.The mbmisc-port release: the general-purpose remainder of Mathias's decade-old R helper package, absorbed as datajure-native operations — date-value-aware lagging, panel gap-filling, bounded/backward fills, filtering joins, and a set of statistical and data-cleaning primitives.
Semi, anti, and cross joins (:how :semi / :anti / :cross). Semi/anti are filtering
joins (dplyr semi_join/anti_join): the result has left's columns only, in left row order,
each left row at most once — :semi keeps rows whose key tuple exists in right, :anti those
whose doesn't. They close the keep-groups-with-≥-N-rows idiom: :agg row counts → :where →
semi-join back. :validate/:report apply as on equi-joins (cardinality is checked even though
filtering joins never duplicate rows). :cross is the Cartesian product — no join keys (passing
:on throws), colliding right columns prefixed right. matching the as-of convention;
:validate/:report are rejected as key-based. Unknown :how errors list the new types.
reshape/tsfill — panel gap-filling (Stata tsfill; essentially tidyr's complete for panel time series). Expands each group to a regular date grid spanning that group's own min..max date, inserting missing rows (nil everywhere but the keys). Union semantics: existing rows — including off-grid ones — are always kept; gridding never drops data. Grid spec: :every (positive number for numeric date columns, default 1; :day/:week/:month/:quarter/:year for temporals, default :day) or an explicit :grid sequence (e.g. actual trading days), deduplicated/sorted/clipped per group. :carry columns are backward- then forward-filled per group (NOCB→LOCF — the mbmisc tsfill carrycols behavior). nil-date rows are dropped (documented — they cannot be placed on a grid); nil group keys throw :tsfill-nil-key; duplicate (key, date) throw :tsfill-duplicate-dates; :every type mismatches throw :tsfill-invalid-every. Output sorted by [keys, date]. Temporal matching is exact — normalise month-end-keyed panels first (xbar bucket), same caveat as win/tlag.
win/fills :limit + new win/bfill backward fill. #dt/e (win/fills :x {:limit 3}) (bare-number shorthand (win/fills :x 3)) carries the last value at most n positions into each nil run, leaving the rest nil — partial fill, matching pandas ffill(limit=n) and mbmisc h.locf, deliberately not zoo's all-or-nothing maxgap (hence the different option name). win/bfill is the symmetric backward fill (NOCB; trailing nils stay nil) with the same option. NOCB→LOCF combos compose by nesting — #dt/e (win/fills (win/bfill :x)) — so there is no combo API. A non-positive limit, a non-number, or an options map without :limit (e.g. the misspelled {:lmit 3}) throws a structured :fills-invalid-limit error instead of silently meaning "unlimited".
win/tlag — date-value-aware lag window op (mbmisc lbd / statar tlag). #dt/e (win/tlag :x :year) gives row i the value of :x at the row whose date equals date − shift — a gap in the panel yields nil instead of win/lag's silent reach-back to the wrong period. Shift defaults to 1; a negative shift is a lead. Dates may be numbers (years, xbar buckets — plain subtraction) or java.time temporals shifted per a trailing options map ({:unit :day} default, :week, :month, :quarter, :year). Temporal matching is exact, so monthly panels keyed on month-end dates should tlag on a normalised month column (e.g. an xbar bucket). Duplicate dates within a partition throw a structured :tlag-duplicate-dates error; an unknown :unit throws :tlag-unknown-unit. Works in both partitioned (:by) and whole-dataset window mode, in #dt/e and data-forms ([:win/tlag :x :date 1 {:unit :month}]).
stat/trim — quantile-based trimming. #dt/e (stat/trim :ret 0.01) nils out values below the p-th or above the (1−p)-th percentile — the removal counterpart of stat/winsorize (mbmisc trim). nil-safe; NaN/±Inf are trimmed.
stat/rescale — min-max rescaling. #dt/e (stat/rescale :x) maps a column linearly to [0,1]; (stat/rescale :x -1 1) to an explicit range (mbmisc mb.rescale / scales::rescale). A constant column returns all nil (no defined scale, matching standardize's zero-sd rule).
One-sided winsorization. stat/winsorize accepts a trailing options map: {:tail :upper} clips only the top tail, {:tail :lower} only the bottom (mbmisc winsor.tp/winsor.bm); default {:tail :both} is unchanged. Unknown :tail values throw a structured :invalid-tail error. stat/trim and stat/rescale get trim/rescale aliases in datajure.concise.
prod — product aggregation (mbmisc mb.prod). Skips nil; an all-missing column yields nil, not the empty-product 1. Available as core/prod, the #dt/e prod op, and the [:prod :col] data-form. The compounding idiom: :agg {:gross #dt/e (prod (+ 1.0 :ret))}.
util/blank->nil (mbmisc emq2na): empty and whitespace-only strings → missing, across all :string/:text columns by default or an explicit column subset. (A bare "" is already missing at dataset construction; this catches the whitespace-only strings that survive it.)
util/parse-numeric (mbmisc destring): lenient string→number column parsing — strips currency symbols, thousands separators, and trailing junk ("$1,234.50a" → 1234.5), leaves unparseable/non-finite values missing, refuses to mangle exponent-plus-junk strings ("1.23e+5x" → missing) rather than guessing, and produces an integer column when every parsed value is whole.
LICENSE file and the published pom). Code is unchanged by this; it affects only the terms
the library is distributed under from this version onward.:set with :by now rides the fast one-pass window path.
2.7.0's ":set accepts any seq of pairs" hid a performance cliff: every seq-of-pairs :set
(sequential semantics) with :by dropped to the general per-group path — per-group sub-datasets
over all columns, the pre-2.4.0 OOM shape on wide data — while only a map :set rode the
fast one-pass path. dt now detects independent pairs (all expression-valued, no pair
referencing a column derived by an earlier pair — the common generated-pass shape) and upgrades
them to the fast path; results are unchanged (independence makes sequential ≡ simultaneous) and
pair order still fixes derived-column order. Genuinely cross-referencing or plain-fn derivations
keep the general path and now print a one-time [datajure] NOTE: about its per-group cost.
(On 2.7.0, wrap generated :by window passes in (into {}) to stay on the fast path.)The syntax-consolidation release: one expression language with two spellings, queries as
pure data, and uniform window-mode semantics. Contains two breaking changes (whole-dataset
window output order; :grouping folded into :by).
#dt/e — one expression language, two spellings. The runtime
data-form vector now covers the entire #dt/e vocabulary instead of a subset: window ops
([:win/lag :price 1], [:win/mavg :x 12 {:min-periods 12}]), row/stat ops ([:row/sum …],
[:stat/winsorize …]), the special forms ([:if …], [:cond … :else …], [:let [:name expr …] body],
[:coalesce …]/[:coalesce-finite …], [:cut :col n :from pred], [:xbar :col w :minutes],
[:win/scan :* expr], [:win/each-prior :- expr]), and every full-name/concise aggregator alias
([:mean …] == [:mn …], [:fst …] == [:first-val …] — derived from the same table as #dt/e,
so the spellings can never drift). Aggregators now also compose in :where data-forms
([:> :x [:mn :x]]), matching #dt/e. Context rules are enforced on the resulting AST exactly as
for #dt/e (e.g. win/* outside :set → :win-outside-window); unknown ops throw
:unknown-data-op with Damerau-Levenshtein suggestions. Data-forms are normalised to ASTs at the
dt boundary, so they ride identical validation and dispatch (window-mode detection, map-:set
cross-reference checks, fast paths). Window-join :agg maps accept data-forms too. This retires
the read-expr-over-quoted-forms workaround for programmatic pipelines: a generated :set pass is
now just (into {} (for [c cols] [(l1key c) [:win/lag c 1]])).dt accepts a single query map — (dt ds {:where [:> :x 1] :by [:g] :agg {:n [:nrow]}}) is
equivalent to the kwargs form (kwargs + a trailing map also merge). With data-forms, a whole query
is pure EDN: storable, diffable, mergeable, and round-trippable through clojure.edn/read-string.dt query keys are rejected. A typo like :wehre previously silently did nothing;
it now throws :unknown-query-key with a suggestion (:wehre -> :where). Malformed argument lists
throw :invalid-dt-args.when-finite — NA-propagating guard (R's ifelse(NA→NA) building block).
#dt/e (when-finite :g body) (data-form [:when-finite :g […]]) yields the body value where the
guard is finite and nil (missing) where it is nil/NaN/±Inf. A bare comparison collapses a nil
operand to false before if sees it, so NA-propagating binary indicators (Piotroski-style
p-ind) were previously inexpressible in the DSL: now
#dt/e (when-finite :g (if (>= :g 0) 1.0 0.0)), with computed guards via let
((let [d (- :a :b)] (when-finite d …))). Element-wise, so it runs on the off-heap :set :by
fast path.nrow/N as a real nullary op. #dt/e (nrow) / #dt/e (N) / [:nrow] / [:N] — so row
count composes in arithmetic (#dt/e (- (nrow) 1) for peer counts) instead of needing the bare
value-marker (which still works). (nrow :col) is a read-time :wrong-arity error pointing to
count*/ct.:set/:agg accept any seq of pairs — a lazy (for [c cols] [(kw c) …]) passes directly,
no (into {})/vec wrapping. Sequential semantics as with vector-of-pairs.[:asc :col]/[:desc :col] sort-spec spelling for :order-by/:within-order/
prepare-grouping, alongside (asc :col)/(desc :col) — the EDN-friendly variant, so sort specs
fit in stored query maps. Malformed specs still throw :invalid-order-spec.:set without :by) now preserves input row order,
completing the 2.6.0 change for the :by case — the two window modes now have one contract:
:within-order governs only the order the window computation walks the rows; results are
scattered back to their original positions; passthrough columns are untouched; use :order-by
to sort output. (Previously the no-:by form returned the dataset sorted by :within-order —
the same keyword meant different output orders depending on :by.) Whole-dataset window
derivations also gain the off-heap default and the structured :unknown-column error path.:grouping is folded into :by — the keyword is removed. A prepared grouping from
prepare-grouping is now passed directly as :by:
(dt ds :set {…} :by g) instead of (dt ds :set {…} :grouping g). :by already dispatched
polymorphically (keywords, fns, xbar/qtile markers); the prepared grouping is one more shape,
and dt stays at nine query keywords instead of ten. The old keyword now throws
:unknown-query-key; the :grouping-requires-set, :grouping-conflict (with :within-order),
and :grouping-row-mismatch guards are unchanged.win/mavg/msum/mdev/mdowndev/mmin/mmax). The rolling engine now realises the column once into a primitive double[] + a present-mask and computes each trailing window with primitive arithmetic — no per-window vector allocation or boxing (the previous hot path boxed every value in every window). Results are identical (same nil/NaN/±Inf and :min-periods/:ddof semantics). On a real 2.1M-row × 45k-firm transform (40 moving averages) the window computation dropped ~3× (21 s → 7 s) and the full :set :by pass ~2.1× (27 s → 13 s).:set + :by window mode now preserves input row order. Previously a window-mode :set with :by returned rows reordered into grouped + :within-order-sorted blocks (an artifact of the per-group split + concat). Now the output keeps the original input row order — passthrough columns are untouched and each derivation's per-group results are scattered back to their original positions. :within-order consequently governs only the per-group computation order (so win/lag/win/cumsum/etc. walk in the right order within a group), not the output order; use :order-by to sort output. Both the fast and general :by paths agree. (Window-mode :set without :by is unchanged — it still sorts by :within-order.)prepare-grouping + the :grouping option — amortise grouping across multi-pass :set :by. (prepare-grouping ds [:gvkey] [(asc :datadate)]) computes the grouping + within-order permutation once; pass the result as :grouping to dt (in place of :by/:within-order) and many :set passes reuse it, skipping the per-call grouping + sort. Valid for any dataset with the same rows in the same order (adding columns between passes is fine, since :set :by preserves input row order); dt checks the row count and rejects :grouping alongside :by/:within-order. On a real 2.1M-row × 45k-firm 10-pass transform the grouping was the per-pass bottleneck once the window compute was sped up — reusing it dropped the run from 48.6 s to 18.4 s (~2.6×), identical results. Like coalesce, but skips NaN and ±Inf as well as nil, returning the first finite value (#dt/e (coalesce-finite :a :b 0.0)). This is the right primitive for fallback chains over computed columns, where an arithmetic step can yield NaN/±Inf (e.g. (+ x nil) → NaN) that plain coalesce would treat as present — previously expressed verbosely as (coalesce (nonfin2na :a) (nonfin2na :b) …). Non-numeric and non-finite arguments count as absent; if no argument is finite the result is nil. A #dt/e special form like coalesce, so it's also element-wise (works in off-heap :set :by).win/lag / win/lead :fill option. A trailing options map fills the boundary positions (those without enough history/future) with a given value: #dt/e (win/lag :price 1 {:fill 0}) → [0 …] instead of [nil …]. Collapses the common (coalesce (win/lag …) 0) two-step into one op. Source nils that get shifted stay nil; default (no map) is unchanged.win/ema accepts an options map. Alongside the numeric shorthand (>= 1 → period, < 1 → smoothing factor alpha), win/ema now takes {:alpha 0.18} or {:period 10} — a self-documenting alternative that makes the alpha-vs-period intent explicit at the call site. Same results as the equivalent numeric form.:set :by — now the default. For the keyword-only :by window-mode :set fast path, numeric derived columns are materialised in off-heap native buffers instead of on the JVM heap. Each derived column is packed normally (so its dtype + missing-set match the on-heap result exactly — output is identical by construction, and the native datatype is type-preserving: integer window ops like win/rank or a lag of an int column stay :int64, floats stay :float64), then its buffer is relocated off-heap, GC-tracked (freed when the dataset is collected, no manual cleanup) and Arrow-exportable. On a real 2.1M-row × 45k-firm transform (40 moving averages) retained JVM heap dropped from ~888 MB to ~13 MB (~68×) at the same speed; extrapolated to a ~270-column per-firm transform that's ~6 GB → ~90 MB, making datajure :set :by memory-competitive with a hand-rolled off-heap scatter. Pass :off-heap false for on-heap output; non-numeric derived columns stay on-heap regardless.win/mdowndev — moving downside-deviation window op. #dt/e (win/mdowndev :ret n) computes the trailing-window downside deviation sqrt(mean(min(r,0)²)) over width n (MAR=0, method='full', na.rm — matching R's PerformanceAnalytics::DownsideDeviation), the Sortino-style risk measure that win/mdev (std-dev) doesn't capture. nil/NaN/±Inf are skipped; a window with finite values but no downside returns 0.0; an empty / all-missing window returns nil (undefined — a deliberate divergence from R's 0, matching datajure's nil-for-undefined philosophy). Runs in :set window mode (:by + optional :within-order), expanding at the start like the other win/m* ops.
Non-expanding (R rollapplyr) windows via an options map on the rolling ops. win/mavg/win/msum/win/mdev/win/mdowndev/win/mmin/win/mmax now accept a trailing options map: #dt/e (win/mavg :ret 12 {:min-periods 12}) requires a full width-row window before emitting (leading width-1 rows → nil), matching R's zoo::rollapplyr. Default stays expanding (:min-periods 1, the q convention). win/mdev also accepts :ddof in the map ({:ddof 0 :min-periods 12}), with the positional (win/mdev :col n d) kept as a back-compat convenience.
win/mdev now defaults to sample standard deviation (ddof=1), matching R's sd and datajure's own sd aggregator. It previously computed population std-dev (ddof=0, q's mdev), which made the moving and aggregate forms of "standard deviation" inconsistent (the sd agg is sample). win/mdev gains an optional ddof arg — (win/mdev :col n 0) recovers the old population behavior (q convention). With the default, a window of a single finite value now yields nil (sample sd is undefined for n=1), where it previously returned 0.0.:set :by window mode is dramatically faster and no longer OOMs on wide datasets. For a keyword-only :by with map derivations, the window-mode group path no longer builds a materialised sub-dataset per group over all columns and concatenates them (which dragged passthrough columns through every group and masked-OOM'd on wide data). It now computes the grouped + :within-order-sorted permutation once, reorders all columns in a single ds/select-rows, and computes each derivation against the reordered base — pure single-column window ops via zero-copy dtype/sub-buffer slicing of the source column per contiguous group (no per-group sub-datasets), group aggregators broadcast per group, element-wise derivations once over the whole base. On a real 2.1M-row × 45k-firm-group transform (40 moving averages), a wide :set (528 passthrough columns) went from a masked OOM to ~30 s, and the narrowed case from ~137 s to ~27 s (and wide ≈ narrow — passthrough columns are no longer dragged through the per-group split). Same results (grouped + within-order order) as before; marker/fn :by, vector (sequential) derivations, and plain-fn derivations keep the general path.asinh in #dt/e. Four new element-wise ops, usable in #dt/e and in :where/:agg/:set data-forms: na2zero (non-finite — nil/NaN/±Inf — → 0.0), nonfin2na (non-finite → nil), neg2na (negative → nil; nil/non-finite pass through as nil), and asinh (numerically-stable inverse hyperbolic sine sign(x)·ln(|x|+√(x²+1)), nil for non-finite). These mirror mbmisc's na2zero/nonfin2na/neg2na and a stable asinh — the textbook ln(x+√(x²+1)) form silently collapses large-negative inputs to nil. The scalar primitives math/finite-double? and math/asinh back them.win/grr — inverse-hyperbolic-sine growth window op. #dt/e (win/grr :x) computes asinh(x[i]) − asinh(x[i-1]) per partition/order (mbmisc grr with IHS=TRUE): nil for the first element, and a run of zeros (x[i]==0 && x[i-1]==0) yields 0.0 rather than asinh(0)−asinh(0). Like other win/* ops it runs in :set window mode (:by + optional :within-order).qnt — one sort for a whole band set. (qnt :col [0.2 0.5 0.8]) (and core/qnt/concise/qnt, and the data-form [:qnt :col [0.2 0.5 0.8]]) returns a vector of quantiles, sorting the column once instead of three times — the q20/median/q80 idiom in one agg. Backed by math/quantiles-type7. In a data-form, a number-headed vector like [0.2 0.5 0.8] is now a literal (so the form works), while a non-number-headed vector stays an operation.:by/:agg group aggregation is dramatically faster at scale. The group-by path no longer builds a materialised sub-dataset per group (over all columns) and concatenates one-row results; it now (1) narrows the dataset to just the columns the aggs reference, (2) groups row indices in a single key-column pass, and (3) for single-column quantile aggregators (qnt/md) gathers each group's values into a primitive double[] and reduces with no boxing — assembling the result columns once. On a real 2.1M-row × 20.8k-group cross-section (69 industries × q20/median/q80), the canonical Fama-French/peer-bands aggregation went from >5 min (effectively unusable) to ~12 s. Pure internal change — same results (verified against the prior path across the suite); other aggregators and within-order groups keep the general per-group path.qnt/md on a date/temporal column now throw a structured :quantile-non-numeric error instead of a raw ClassCastException (LocalDate cannot be cast to Number) deep in the sort — the same class of guard as the as-of :asof-non-numeric-asof-key. Convert the column to epoch days/millis to rank it.median, qtile/cut, winsorize, describe now match R exactly. datajure's quantile/median estimator was tech.ml.dataset's dfn/percentiles / dfn/median (Apache Commons Math), which disagrees with R's default type-7 at the tails and, for some n, at the median (e.g. median of a 542-value column: 57.999 vs R's 57.9375). All quantile call-sites now use a single type-7 primitive (datajure.math/quantile-type7): core/median, the qtile/cut breakpoints, stat/winsorize, and util/describe quartiles. Quantiles drop nil and non-finite (NaN/±Inf, R's is.finite) before estimating. This is a deliberate behavior change for parity with R as the reference implementation (no backwards-compat shim — see the project's quantile decision).qnt — type-7 quantile aggregator in #dt/e. #dt/e (qnt :col p) returns the type-7 p-quantile (p a fraction in [0,1]) of a column's finite values, e.g. (dt panel :by [:gind] :agg {:q20 #dt/e (qnt :saleq 0.2) :med #dt/e (qnt :saleq 0.5)}). A 3-arg form (qnt :col p min-n) returns nil when fewer than min-n finite values remain (floor-free by default — min-n is opt-in for rules like peer-bands n≥11). md is now type-7 too, so (md :col) and (qnt :col 0.5) agree.:agg / :set accept the runtime data-form vector (parity with :where). A plain vector [op-kw & args] now works in :agg and :set, not just :where: (dt ds :by [:g] :agg {:q20 [:qnt :x 0.2]}), (dt ds :set {:r [:div0 [:- :a :b] :a]}). Keywords are columns, vectors are operations, anything else a literal — desugaring to the same AST #dt/e compiles, so it runs the vectorized path. This makes programmatic query generation ergonomic (e.g. generate one qnt aggregation per column in a list) without reaching for read-expr over a quoted form. :agg/:set data-forms also allow the scalar aggregators (mn/sm/md/qnt/…); :where data-forms stay element-wise-only (a predicate can't be an aggregator). Unknown ops throw :unknown-data-op.dio/read-seq now applies the string→keyword :column-allowlist / :column-blocklist normalization on Parquet (the 2.0.13 fix had reached dio/read only). Parquet applies :key-fn to column names before matching the allow/block list, so under datajure's keyword :key-fn a string allowlist entry silently matched nothing → a 0-column dataset on the streaming reader (the dashboard was unaffected — it uses dio/read; this bit only the out-of-core read-seq path). read-seq's Parquet branch now defaults :key-fn keyword and normalizes string entries to keywords just like read, so {:column-allowlist [:a :b]} and {:column-allowlist ["a" "b"]} both work on both readers. Verified end-to-end against a real prepared Parquet.:direction :nearest / :how :window on a date/temporal asof key now throw a structured error instead of a raw ClassCastException. Both paths rank or bound matches by raw arithmetic on the asof value, which a java.time key (e.g. :local-date) cannot supply, so they previously failed deep in the search with an opaque class java.time.LocalDate cannot be cast to class java.lang.Number. They now raise :dt/error :asof-non-numeric-asof-key up front, naming the offending columns/datatypes and pointing to the supported paths (:direction :backward/:forward, and :tolerance via [n unit], all of which keep working with temporal keys) or to converting the column to epoch days/millis. A packed date type (:packed-local-date) reports as both numeric and datetime but yields temporal objects, so it is correctly treated as temporal here.:where data-form predicate — vectorized filtering by a runtime value. :where now accepts a plain data vector where a keyword is a column and anything else is a literal value, so a runtime value flows straight in: (dt panel :where [:= :tic ticker]), (dt ds :where [:and [:>= :mass lo] [:< :mass hi]]). It desugars to the same AST #dt/e compiles, so it runs the vectorized dfn path — no per-row row-map (the footgun of a plain-fn :where on a wide dataset). #dt/e is read-time and can't see a runtime local; the data-form is its runtime-data twin, ideal for parameterized queries and screens built programmatically. Supports the element-wise ops > < >= <= = and or not in between? + - * / sq log div0 (use a set for :in); aggregations, window/row/stat ops, and if/cond/let/cut/xbar stay #dt/e-only. Errors: :unknown-data-op, :invalid-data-op, and the usual :unknown-column.:tolerance accepts temporal asof keys — point-in-time staleness caps. An as-of join on a date/time key (e.g. CRSP daily date × Compustat rdq) can now bound how stale a match may be with a [n unit] duration: (join crsp compustat :left-on [:gvkey :date] :right-on [:gvkey :rdq] :how :asof :tolerance [90 :days]) keeps each daily row matched to the most recent report no older than 90 days, nil beyond that. Units :seconds/:minutes/:hours/:days/:weeks (the temporal difference is taken in epoch-milliseconds). Numeric asof keys keep their plain-number :tolerance.datajure.index — keyed lookup index for repeated point-lookups. Build an index once over one or more key columns, then resolve a key to its rows in O(1) instead of scanning: (def by-tic (idx/index-by panel :tic)) then (idx/lookup by-tic "AAPL") → sub-dataset in original row order. Multi-column keys use a tuple; an absent key yields an empty dataset; lookup-indices returns the raw row indices for gathering from a row-aligned projection yourself. The index is an immutable value holding a reference to its source dataset, so a lookup can't be applied to a mismatched table — data.table's setindex(), never a mutating setkey(). Adds index? / kind / key-columns / source-dataset; structured errors :unknown-column / :invalid-key-cols / :invalid-lookup-key / :not-an-index.:asof index kind, reusable across as-of / window joins. (idx/index-by right key-cols {:kind :asof}) (or idx/asof-index) builds the prepared right-side structure that as-of and window joins use internally (last key column = asof column, the rest exact-match). The same index value can be reused across repeated merges against one right table — e.g. CRSP daily × Compustat quarterly under several reporting-lag assumptions — instead of rebuilding it each call. The build logic now lives in datajure.index; datajure.asof owns only the search and result assembly.datajure.asof's right-side index now splits each group's as-of values and original row indices (and wraps the column reader) once per group, instead of rebuilding them on every left row. asof-match / window-indices now do only a map lookup plus a binary search per probe. Pure internal change — same results, same public API. Verified depth-independent: 200k left rows hold ~140 ms as the matched group's depth grows 100→1600.asof-match / window-indices take an opts map. The canonical arity is now (asof-match left right left-keys right-keys {:direction … :tolerance … :right-index …}) (and {:lo :hi :right-index} for window-indices), avoiding a long positional signature ahead of further as-of options. Existing positional arities are kept and delegate, so callers are unaffected.long[] row-index arrays (asof-row-indices) and gathers each right column into a single object-array (build-result-arrays), instead of materialising a sequence of [left right] pair vectors and all right columns at once. Peak working memory drops from all-right-columns-wide to one-column-wide, and no per-row pair objects are allocated — meaningful on wide many-row joins (e.g. CRSP daily × Compustat quarterly). asof-match / build-result keep their pair-sequence contracts as thin wrappers. Right-column datatypes and missing values are preserved exactly (verified by regression).:tolerance on a temporal asof key now requires a [n unit] spec. Previously any :tolerance on a non-numeric (e.g. datetime) asof key threw :join-tolerance-non-numeric; temporal keys are now supported, so a bare number on a temporal key throws :join-tolerance-invalid (directing to [n unit]), an unknown unit throws :join-tolerance-unknown-unit, and :join-tolerance-non-numeric is now reserved for keys that are neither numeric nor temporal (e.g. strings).:take — row limit in dt. Positive n keeps the first n rows (head), negative keeps the last |n| (tail), 0 yields no rows, and |n| beyond the row count returns all rows. It runs last, after :order-by, so the everyday "last 20 by date" is one query: (dt ds :order-by [(asc :date)] :take -20) — no more dropping out to (ds/tail 20 …). (The signed convention follows q's #.) A non-integer :take throws :dt/error :invalid-take.dio/read accepts string :column-allowlist / :column-blocklist for Parquet/Arrow. Parquet/Arrow apply :key-fn to column names before matching the allow/block list, so under datajure's keyword :key-fn a string entry silently matched nothing → a 0-column dataset (the mirror image of the CSV/TSV bug fixed in 2.0.12). dio/read now normalises string entries to keywords for Parquet/Arrow, so {:column-allowlist [:a :b]} and {:column-allowlist ["a" "b"]} both work on every format. (Parquet reading itself was already supported; this closes the allowlist asymmetry.)core/div0 — nil-safe division as a callable function. div0 was an #dt/e-only op, so plain-fn contexts (:set/:agg with #(...), computed :by) couldn't reach it and had to roll their own zero-guard. It's now also a public scalar fn: (div0 num den) → nil when either is nil or the denominator is zero, else num/den as a double; non-numeric inputs throw normally. The #dt/e div0 op delegates to the same fn (single source of truth via expr/div0).
(div0 1 2) ;; => 0.5
(div0 1 0) ;; => nil
(dt ds :set {:pe #(div0 (:price %) (:earnings %))}) ;; plain-fn :set
dio/read accepts keyword :column-allowlist / :column-blocklist for CSV/TSV. charred matches these against the raw header strings before :key-fn is applied, so a keyword allowlist — the natural form, since datajure keyword-izes columns — silently matched nothing and returned a 0-column dataset. dio/read now normalises keyword entries to their raw names for CSV/TSV, so {:column-allowlist [:a :b]} and {:column-allowlist ["a" "b"]} both work. Scoped to CSV/TSV; Parquet/Arrow match after :key-fn, so they're left untouched.
#dt/e (div0 …) with scalar operands now returns a scalar. It always built a length-1 column, so a composed expression like #dt/e (+ :x (div0 1 2)) produced [1.5 nil nil] instead of [1.5 2.5 3.5] (the length-1 reader only covered row 0). It now returns a scalar when both operands are scalars — consistent with the other arithmetic ops — so it broadcasts correctly. Column operands are unchanged.
:order-by / :within-order now validate their columns. An unknown sort column previously surfaced a raw tech.ml.dataset Column not found exception; it now throws datajure's structured :dt/error :unknown-column (with :dt/context :order-by/:within-order and Damerau-Levenshtein suggestions), the same as :select. :order-by is validated against the post-transform dataset, so sorting by a :set-derived column works and sorting by a :select-dropped column errors. A malformed spec (missing :col, or :order that isn't :asc/:desc) throws :dt/error :invalid-order-spec.
and/or in #dt/e are now variadic. They compiled directly to dfn/and/dfn/or, which are binary-only, so #dt/e (and p1 p2 p3) threw Wrong number of args (3). They now fold over dfn/and/dfn/or, accepting any number of predicates: #dt/e (and (> :a 1) (> :b 2) (< :c 3)). (not remains unary; zero-arity (and)/(or) is rejected at read time with a :wrong-arity error, since it's meaningless in a vectorized mask.)
:order-by / :within-order are dramatically faster on wide datasets. Sorting routed through tech.ml.dataset's (sort-by dataset identity comparator), whose identity key-fn forces a full row object to be materialised for every row — all columns — even though only the sort keys are compared. On a 2.1M-row × 91-column dataset, sorting by two keys took ~326 s; it now takes ~1.6 s (~200×). The sort now reads only the key columns, stable-sorts an index permutation with clojure.core/compare (nils first, mixed :asc/:desc), and gathers via ds/select-rows. Ordering semantics are unchanged — including nil placement, tie stability, and decoded-value comparison for packed columns like dates.Max/min aggregation inside #dt/e (mx / mi). #dt/e previously had no max/min op at all, so the canonical OHLC example :hi #dt/e (mx :price) threw Unknown op at read time (uncaught because the OHLC tests only asserted open/close, never hi/lo). mx/mi aggregate a column's maximum/minimum, skipping nil and returning nil for an all-missing column.
(dt trades :by [:sym] :within-order [(asc :time)]
:agg {:open #dt/e (first-val :price) :close #dt/e (last-val :price)
:hi #dt/e (mx :price) :lo #dt/e (mi :price)})
Full-name and concise aggregation names now work inside #dt/e. Every datajure.core / datajure.concise aggregation helper is accepted as a #dt/e op, in both spellings — so #dt/e (max* :price), #dt/e (sum :size), #dt/e (count* :revenue), #dt/e (nuniq :id) etc. all work, not just the concise mn/sm/md/sd. New ops :variance and :ct (non-nil count, also backing core/count*).
core/max* and core/min* returned wrong values on columns with missing data. They aliased dfn/reduce-max / dfn/reduce-min, which corrupt the reduction when a missing value precedes the extremum (e.g. max* of [5000 3750 nil 4000] returned 4000, not 5000). They now filter nils first via the shared expr/col-max / expr/col-min, skipping missing and returning nil for an all-missing column; the #dt/e mx/mi ops use the same correct path.wavg/wsum wrong-arity error is now structured. Calling #dt/e (wavg :x) (or wsum, or the wa/ws aliases) with other than two arguments now throws a structured :wrong-arity ex-info at read time — "wavg takes exactly two arguments (weight column, value column). Got 1." — instead of a raw clojure.lang.ArityException.
Clear error for a literal nil in arithmetic. #dt/e (+ :x nil) (and - / * / / / sq / log) now throws a structured :arith-nil-literal error at read time — "Arithmetic op + received a literal nil … use coalesce … or div0 …" — instead of a cryptic "Item type null has no iterator" from deep in the dataset layer. Arithmetic requires non-nil operands (nil is ambiguous there); predicates keep their nil-literal → false rule, and coalesce/div0 remain the tools for handling nils. (Earlier docs claimed arithmetic-with-a-nil-literal returned nil; it never did — it threw.)
JSON I/O in datajure.io. .json (and .json.gz) is now a natively supported format for read and write, alongside CSV/TSV/Nippy — no optional dependency required (tech.v3.dataset parses/emits JSON via charred). Files are read/written as a JSON array of row objects; column names round-trip as keywords like every other format.
(dio/read "data.json")
(dio/write ds "output.json")
(dio/read "data.json.gz") ;; gzip auto-detected
read-seq also accepts .json: since a JSON array is a single document with no chunk boundaries, it is read whole and yielded as a one-element lazy sequence (API uniformity across formats — no streaming/memory benefit; use Parquet or JSON Lines for true out-of-core reads).
JSON Lines I/O (.jsonl / .ndjson) in datajure.io. Newline-delimited JSON — one object per line — for read, write, and read-seq (and .gz variants). Unlike a JSON array, JSON Lines is genuinely chunk-able, so read-seq streams it in batches of :batch-size rows (default 100000), suitable for files larger than memory. Parsing/encoding uses charred (the same library tech.v3.dataset uses), now declared as an explicit dependency. Missing keys read back as nil; blank lines are skipped.
(dio/read "events.jsonl")
(dio/write ds "events.jsonl")
(doseq [chunk (dio/read-seq "huge.jsonl" {:batch-size 50000})] ;; streamed
(process chunk))
A post-alpha audit pass reconciling the library with data.table-style semantics, plus a handful of correctness fixes uncovered by REPL verification of the DSL's per-partition execution paths.
qtile now uses per-partition breakpoints when combined with exact keys in :by. Previously :by [:date (qtile :mktcap 5)] computed breakpoints once from the whole dataset (and silently produced wrong answers for per-date cross-sectional sorts — the canonical CRSP / Fama-French size sort). It now partitions by the exact keys first, then resolves qtile against each sub-dataset, matching data.table / dplyr / q:
;; Per-date size quintiles — now works the obvious way
(core/dt stocks :by [:date (core/qtile :mktcap 5)]
:agg {:mean-ret #dt/e (mn :ret)})
;; Per-date NYSE-style breakpoints applied to all stocks (Fama-French size sort)
(core/dt stocks :by [:date (core/qtile :mktcap 5 :from #dt/e (= :exchcd 1))]
:agg {:mean-ret #dt/e (mn :ret)})
An audit of every other DSL feature (stat/*, aggregations inside composite #dt/e, cut :from, win/*, row/*, xbar, join :asof/:window) confirmed qtile was the only outlier — every other feature already ran per-partition by virtue of living inside apply-group-*. Pure-qtile :by (no exact keys) still resolves globally since there is nothing to partition by.
cast — long→wide reshaping. Complement to melt. For each unique combination of :id column values, pivots the :from column's distinct values into new columns filled from the :value column. New column names derived from :from values (keywords pass through; strings converted via keyword). Supports :agg for duplicate cells and :fill for missing cells (default nil). melt / cast round-trip correctly.
;; Reverse a melt
(-> ds
(melt {:id [:species :year] :measure [:mass :flipper]})
(cast {:id [:species :year] :from :variable :value :value}))
;; With aggregation for duplicate (id, from) cells
(cast ds {:id [:date :sym] :from :metric :value :val :agg dfn/mean})
cast accepts a single-keyword :id. Normalised to a one-element vector, matching melt. Previously a single-keyword :id errored with "Don't know how to create ISeq from: clojure.lang.Keyword".
row/sum, row/mean, row/min, row/max on all-nil rows. The four row-wise aggregators declared :float64 readers but claimed in their docstrings to return nil when every input is nil. Primitive float readers cannot hold nil — the value was silently coerced to NaN, contradicting the docstring. Readers are now :object, so all-nil rows honestly return nil. row-min / row-max also cast non-nil results to double to preserve the always-numeric-result convention.
wavg / wsum with mismatched column lengths. Previously silently truncated (when the weight column was shorter) or NPE'd (when the value column was shorter). Now throws a structured :unequal-column-lengths ex-info with :dt/op, :dt/weight-length, and :dt/value-length in ex-data.
join :asof :tolerance on datetime asof keys. Previously produced a raw java.lang.ClassCastException from an unguarded (double dt-value) coercion inside within-tolerance?. Now validates numerically-compatible asof key types upfront and throws a structured :join-tolerance-non-numeric ex-info with actionable guidance (convert to epoch-milliseconds). Symmetric and asymmetric join key shapes (:on vs :left-on/:right-on) both report the correct column names in ex-data.
describe on all-missing numeric columns. When every value in a numeric column was missing, dfn/standard-deviation returned -0.0 while mean, min, max, median, and percentiles correctly returned nil, producing an incoherent summary row. describe-column now routes all-missing numeric columns through the nil-filled branch (same as non-numeric columns).
parse-window-spec now strict-validates window spec shape. Previously the implementation destructured [a b c :as wspec] and silently dropped any trailing elements. Malformed specs now throw a structured :join-invalid-window ex-info — trailing junk, wrong arity, non-numeric endpoints, non-vector specs, and misplaced unit keywords are all rejected upfront. Valid [lo hi], [lo hi unit], and [lo unit hi] shapes behave exactly as before.
count-distinct now excludes nil. The fn included nil in its distinct count, contradicting its docstring ("non-nil values"). Fixed by filtering some? before distinct.
qtile and cut now use the same breakpoint algorithm. qtile's percentile-breakpoints previously used a floor-index approximation that produced different breakpoints than cut-bucket (which uses dfn/percentiles). The two now share dfn/percentiles, so qtile :mktcap 5 and #dt/e (cut :mktcap 5) produce identical bins for the same population.
Breakpoint-at-exact-value semantics unified across qtile and cut. bin-via-breakpoints now uses <= (values equal to a breakpoint go to the lower bin), matching cut-bucket's java.util.Arrays/binarySearch exact-match behaviour. The previously passing qtile-from-basic test's assertion was wrong; corrected to reflect actual semantics.
xbar / xbar-bucket now use Math/floorDiv. Previously used quot, which truncates toward zero — so negative values bucketed incorrectly relative to q's xbar semantics. E.g. (xbar -3 5) now returns -5 rather than 0.
validate-expr-cols and validate-select-cols NPE on zero-column datasets. Both helpers computed (->> avail-names (map ...) (sort-by second) first), which returned nil when avail-names was empty. (second nil) yielded nil, and (<= nil 3) threw NullPointerException. Guarded with (and closest ...) in the suggestion-emission branch.
win/scan op normalisation mirrors win/each-prior. The parser now prefers sym->op for the canonical keyword, so invalid scan ops like / resolve to :div (consistent with the rest of the codebase) rather than a keyword literally spelled with a slash. Valid scan ops (+, *, max, min) are unchanged in both AST and runtime.Damerau-Levenshtein deduplication. The edit-distance implementation used by typo suggestions was duplicated byte-for-byte in datajure.core and datajure.expr. Extracted to the public datajure.expr/damerau-levenshtein as the single source of truth; both validate-expr-cols / validate-select-cols and suggest-op now call it.
Dead win-ops set removed from datajure.expr (unused, and stale — missing win/scan and win/each-prior).
row-fns-all-nil-returns-nil-not-nan, wavg-wsum-unequal-lengths, asof-tolerance-non-numeric-error-test, describe-all-missing-numeric, wjoin-invalid-window-shape-test, qtile-per-group-breakpoints, qtile-from-with-exact-key. The existing qtile-combined-with-keyword test was strengthened from a column-names-only check to an assertion of per-group bin counts (would fail loudly against the old global-breakpoint implementation).qtile :from — reference-subpopulation breakpoints. qtile now accepts an optional :from keyword argument: a #dt/e boolean expression or a boolean column keyword that selects a reference subpopulation for computing breakpoints. Breakpoints are computed from the filtered subset and applied to all rows — the same semantics as #dt/e (cut :col n :from pred). Classic use case is NYSE-style breakpoints: (qtile :mktcap 5 :from #dt/e (= :exchcd 1)) computes quintile boundaries from NYSE stocks and applies them to all stocks.
(core/dt stocks :by [(core/qtile :mktcap 5 :from #dt/e (= :exchcd 1))]
:agg {:n core/nrow :mean-ret #dt/e (mn :ret)})
win/each-prior — generalized adjacent-element operator. Applies any binary operator to f(x[i], x[i-1]) for each element. First element → nil; nil propagates if either value is nil. Supports +, -, *, /, max, min, >, <, >=, <=, =. Generalizes win/delta (op=-) and win/ratio (op=/) without their double-casting or zero-guard:
(core/dt ds :by [:permno] :within-order [(core/asc :date)]
:set {:pw-hi #dt/e (win/each-prior max :price) ;; pairwise max with previous
:up? #dt/e (win/each-prior > :price)}) ;; did price increase?
Bounded as-of joins — :direction and :tolerance. join with :how :asof now accepts two new options:
:direction — :backward (default, last right ≤ left), :forward (first right ≥ left), or :nearest (closest by abs distance; ties prefer :backward).:tolerance — numeric max abs distance; matches beyond it produce nil. Requires a numeric asof key.;; Forward: each trade matched to next available quote
(join trades quotes :on [:sym :time] :how :asof :direction :forward)
;; Nearest: closest quote in either direction
(join trades quotes :on [:sym :time] :how :asof :direction :nearest)
;; Tolerance: reject stale quotes more than 5 seconds old
(join trades quotes :on [:sym :time] :how :asof :tolerance 5)
asof-search gains a 4-arity directional variant; asof-match gains a 6-arity variant (4-arity delegates with :backward/nil defaults — fully backward-compatible).
Window join (:how :window, q's wj). For each left row, aggregates all right rows whose asof-key falls within a window around the left row's asof-key. Window spec formats: [lo hi], [lo hi unit], or [lo unit hi] where unit is :seconds, :minutes, :hours, :days, or :weeks. Aggregation map :agg accepts #dt/e expressions (nil for empty windows) or plain fns (receive a 0-row sub-dataset for empty windows, so nrow → 0). All left rows preserved.
;; 5-minute VWAP window
(join trades quotes
:on [:sym :time]
:how :window
:window [-5 0 :minutes]
:agg {:avg-bid #dt/e (mn :bid)
:avg-ask #dt/e (mn :ask)
:n core/nrow})
asof.clj gains a public window-indices utility function for programmatic use.
qtile quantile grouping for :by. core/qtile is the :by-friendly companion to #dt/e (cut ...) -- produces an equal-count bin assignment from a column's distribution, computed once from the dataset before grouping. Use it when you want to group by quantile rather than derive a column of quantile bins:
;; Quintile buckets of market cap
(core/dt stocks :by [(core/qtile :mktcap 5)]
:agg {:n core/nrow :mean-ret #dt/e (mn :ret)})
;; Result column is auto-named :mktcap-q5
;; Per-date size quintiles combined with an exact key
(core/dt stocks :by [:date (core/qtile :mktcap 5)]
:agg {:mean-ret #dt/e (mn :ret)})
Inspired by R's cut and Stata's xtile; named to evoke quintile/decile. Companion to xbar (equal-width bins) with a symmetric API. Result column name defaults to <col>-q<n>, overridable via :datajure/col metadata (the same extension point xbar uses). nil inputs form their own group (nil key). The :from option for reference-subpopulation breakpoints was added post-release — see [2.0.8].
by->group-fn now receives the dataset. Internal refactor: the private by->group-fn helper takes the dataset as a parameter so that :by markers requiring population-level statistics (like qtile) can precompute their breakpoints before grouping. Both call sites (apply-group-agg, apply-group-set) have been updated. No user-visible behaviour change.:within-order with :agg. :within-order can now be combined with :agg (with or without :by), sorting rows within each partition (or across the whole dataset) before aggregation. This enables order-sensitive aggregations like OHLC bar construction in a single dt call:
(core/dt trades
:by [:sym]
:within-order [(core/asc :time)]
:agg {:open #dt/e (first-val :price)
:close #dt/e (last-val :price)
:vol #dt/e (sm :size)
:n core/N})
Previously this required two dt calls (pre-sort via :order-by, then aggregate). The restriction that :within-order required :set has been relaxed to ":within-order requires :set or :agg."
nrow alias for N. core/nrow is now exported alongside core/N as a more discoverable full-name alternative for row counting in :agg. Both are equivalent.clean-column-names is now Unicode-aware. The regex that strips non-identifier characters no longer removes CJK, Cyrillic, Greek, accented Latin, and other non-ASCII letters and digits. Mixed-script column names like "Some Name (HKD millions)!" combined with CJK or accented-Latin characters are now preserved intact; only punctuation, whitespace, and symbols are replaced. Pure-ASCII column names behave exactly as before. See clean-column-names-unicode in the test suite for coverage of CJK, accented Latin, and mixed-script cases.core.clj (column-name suggestions) and expr.clj (op-name suggestions) was producing incorrect edit distances (e.g., "kitten" to "sitting" returned 5 instead of 3), causing many legitimate suggestions to be dropped. The algorithm has been replaced with a correct Damerau-Levenshtein implementation that also treats single adjacent transpositions as distance 1. Typos like :hieght now correctly suggest :height.win/ratio no longer propagates Infinity on zero denominators. When the previous row's value is zero, win/ratio now returns nil rather than Infinity. This matches the div0 philosophy and gives the correct result for the canonical simple-return idiom (- (win/ratio :price) 1) -- a zero-price observation now yields nil for the next row's return, signalling "exclude" rather than contaminating downstream calculations.:agg plain-function footgun detection. Plain functions passed to :agg receive the group dataset, so #(:mass %) returns a column vector rather than a scalar -- a common mistake for users coming from :set context. Previously this silently produced nonsensical output (a column inside each result cell); now it throws a structured error with guidance pointing to either #(dfn/mean (:mass %)) or the preferred #dt/e (mn :mass).#dt/e produce structured errors with suggestions. A typo like #dt/e (sqrt :x) previously caused a raw ClassCastException at runtime. It now throws an ex-info at read time with the suggestion: Unknown op `sqrt` in #dt/e expression. Did you mean: `sq`?. Namespaced ops (win/*, row/*, stat/*) get namespace-aware suggestions: win/mvag -> win/mavg, stat/standardise -> stat/standardize.Earlier versions are not documented in this changelog. Release history is tracked in the GitHub releases page and in PROJECT_SUMMARY.md's phase-completion table.
Can you improve this documentation? These fine people already did:
Matt Buehlmaier & buehlmaierEdit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |