datajure.core — com.github.clojure-finance/datajure 2.7.4

*dt* clj

Holds the last dataset result in an interactive REPL session. Automatically bound by datajure.nrepl/wrap-dt middleware. Like Clojure's *1, but only for tech.v3.dataset results.

asc clj

(asc col)

Sort-spec helper: ascending order on col. Use in :order-by.

col-range clj

(col-range start-col end-col)

Returns a column selector that selects all columns positionally between start-col and end-col (inclusive). Both endpoints must exist in the dataset. Intended for use with :select in dt. (Named col-range — a POSITIONAL range over column names — to keep it visually distinct from the #dt/e value predicate between?.)

Example: (dt ds :select (col-range :month-01 :month-12))

count* clj

Count of non-nil values in a column. Asterisk-suffixed to avoid shadowing clojure.core/count. Distinct from N (total rows) and count-distinct (unique non-nil values). Delegates to expr/count-non-nil (shared with the #dt/e :ct op).

cut clj

(cut col-kw n & {:keys [from]})

Equal-count (quantile) binning — one name, two contexts, exactly like xbar:

• Inside #dt/e (:set/:where/:agg): a column of bin integers — (dt ds :set {:quintile #dt/e (cut :mktcap 5)}).
• Standalone in :by: a grouping marker that bins each row — (dt ds :by [(cut :mktcap 5)] :agg {...}).

Both contexts share the same breakpoints (R type-7 quantiles at 1/n .. (n-1)/n, values equal to a breakpoint fall in the lower bin) so they always bin identically for the same population. Inspired by R's cut and Stata's xtile. nil input values produce nil (their own group in :by).

Breakpoint population in :by — depends on what else is in :by:

• cut alone in :by → breakpoints from the WHOLE dataset
• cut + other exact keys in :by → breakpoints are computed PER exact-key partition

So :by [:date (cut :mktcap 5)] does what you would expect in data.table or dplyr: each date's rows are binned against that date's own quintiles. This is the canonical CRSP / Fama-French pattern — per-date cross-sectional size quintiles.

The optional :from keyword accepts a #dt/e boolean expression or a boolean column keyword selecting a reference subpopulation for breakpoint computation (same as :from inside #dt/e). When combined with other exact keys in :by, the mask is applied within each partition. Classic NYSE use case:

(dt stocks :by [:date (cut :mktcap 5 :from #dt/e (= :exchcd 1))]
    :agg {:mean-ret #dt/e (mn :ret)})

per-date NYSE quintile breakpoints applied to all stocks (NYSE + AMEX + NASDAQ) — Fama-French size sort exactly.

Companion to xbar (equal-width bins).

In :by, the result column name defaults to <col>-q<n> (e.g. :mktcap-q5). Override via :datajure/col metadata on the marker. Inside #dt/e you name the column yourself in :set.

Note on small partitions: if a partition has fewer than n non-nil values, breakpoints cannot be computed and all non-nil rows in that partition land in bin 1. Consider filtering out thin partitions upstream or using fewer bins.

Usage: ;; Column of quintile bins (dt ds :set {:size-q #dt/e (cut :mktcap 5)})

;; Global quintiles as a grouping (dt stocks :by [(cut :mktcap 5)] :agg {:n N :mean-ret #dt/e (mn :ret)})

;; Per-date size quintiles — the canonical CRSP / Fama-French pattern (dt stocks :by [:date (cut :mktcap 5)] :agg {:mean-ret #dt/e (mn :ret)})

;; Per-date NYSE quintile breakpoints applied to all stocks (dt stocks :by [:date (cut :mktcap 5 :from #dt/e (= :exchcd 1))] :agg {:mean-ret #dt/e (mn :ret)})

desc clj

(desc col)

Sort-spec helper: descending order on col. Use in :order-by.

div0 clj

Nil-safe division of two scalars: nil when either is nil or the denominator is zero, else num/den as a double. Use in plain-fn contexts (:set/:agg with #(...), computed :by) where the #dt/e div0 op isn't available; the op delegates to this same fn. Non-numeric inputs throw normally. Examples: (div0 1 2) => 0.5; (div0 1 0) => nil; (div0 1 nil) => nil.

dt clj

(dt dataset & args)

Query a dataset. Supported keywords: :where, :set, :agg, :by, :select, :order-by, :within-order, :take, :off-heap.

Arguments may be given as keyword/value pairs or as a single query map — (dt ds :where p :by [:g] :agg {...}) and (dt ds {:where p :by [:g] :agg {...}}) are equivalent. With data-form expressions, a whole query is plain EDN data that can be stored, merged, and built programmatically. Unknown query keys throw a structured :unknown-query-key error (with a typo suggestion) instead of being silently ignored.

:where - filter rows. Accepts a #dt/e expression, a runtime data-form vector (e.g. [:= :tic ticker] — keywords are columns, anything else is a literal value, so runtime values flow in without a row-map), or a plain fn of the row map. :set - derive/update columns. Accepts a map (simultaneous) or any seq of pairs (sequential — later pairs see earlier-derived columns). When :set contains win/* functions, window mode is activated — with :by, computes within groups; without :by, whole dataset is one partition. With :by, a map — or a seq of pairs whose expressions never reference a column derived by an earlier pair — runs on the fast one-pass path; genuinely cross-referencing or plain-fn derivations fall back to the per-group path (slow on wide data; a one-time NOTE says so). :agg - collapse to summary. Accepts map or seq-of-pairs. Use (nrow)/[:nrow] for row count. :by - grouping for :agg or :set (partitioned window mode). A vector of keywords, a fn of the row, or (for :set) a prepared grouping from prepare-grouping — which bundles the group keys and the :within-order sort, amortising them across multi-pass transforms. #dt/e expressions / data-form vectors are NOT accepted as entries (structured :expr-in-by error) — derive the key with :set first. :within-order - the order rows are WALKED within each partition (or across the whole dataset when :by is absent) while :set or :agg computes — for window functions (win/lag, win/cumsum, ...) and order-sensitive aggregations (first-val, last-val, OHLC). It never affects output row order: a :set query returns rows in input order regardless (use :order-by to sort output); an :agg query returns one row per group as usual. :select - keep columns. Accepts: vector of kws, single kw, [:not kw ...], regex, predicate fn, or map {old-kw new-kw} for rename-on-select. :order-by - sort rows. Accepts a vector of (asc :col)/(desc :col) specs, or bare keywords (default asc). Evaluated before :take. :take - row limit (integer). Positive n keeps the first n rows (head), negative keeps the last |n| (tail), 0 yields no rows. |n| beyond the row count returns all rows. Evaluated last, after :order-by — e.g. :order-by [(asc :date)] :take -20 is "the last 20 by date". :off-heap - boolean, default true. For window-mode :set (keyword-only :by fast path, prepared-grouping :by, or whole-dataset windows), materialise numeric derived columns in off-heap native buffers (freed on GC, type-preserving int/float) instead of on the JVM heap — for wide per-group transforms this takes the result from gigabytes of heap to ~0. Pass :off-heap false for on-heap output. No effect on other query shapes or non-numeric derived columns.

max* clj

Column maximum, skipping nil/missing; nil for an all-missing column. Asterisk-suffixed to avoid shadowing clojure.core/max. Delegates to expr/col-max — dfn/reduce-max returns a wrong value when the column has missing entries (a missing slot corrupts the reduction).

mean clj

Column mean. Full-name alias for dfn/mean.

median clj

(median col)

Column median (R type-7, matching R's median). Drops nil and non-finite values. Equivalent to (qnt col 0.5).

min* clj

Column minimum, skipping nil/missing; nil for an all-missing column. Asterisk-suffixed to avoid shadowing clojure.core/min. Delegates to expr/col-min (see max* for the dfn/reduce-min caveat).

N clj

Row count aggregation helper. Use as a value in :agg maps. Terse alias matching data.table/q convention. See also nrow for a more discoverable full name.

nrow clj

Row count aggregation helper. Use as a value in :agg maps. Full-name alias for users who prefer readability over terseness. Equivalent to N.

pass-nil clj

(pass-nil f & guard-cols)

Wraps a row-level fn to return nil if any of the specified guard columns are nil/missing in the row. Prevents crashes when plain fns encounter missing values in :set or :where.

Usage: (pass-nil #(Integer/parseInt (:x-str %)) :x-str)

prepare-grouping clj

(prepare-grouping dataset by)

(prepare-grouping dataset by within-order)

Precompute the grouping + :within-order permutation for dataset so it can be reused across many :set + :by passes (e.g. a multi-pass per-entity ETL), amortising the grouping + sort that each pass would otherwise repeat. by is a non-empty vector of keyword column names; within-order is an optional sort spec (same form as :within-order). Pass the result directly as :by to dt, in place of the column vector + :within-order:

(let [g (prepare-grouping ds [:gvkey] [(asc :datadate)])]
  (-> ds (dt :set {…} :by g) (dt :set {…} :by g) …))

The grouping stays valid for any dataset with the SAME rows in the SAME order — adding columns between passes is fine, since :set :by preserves input row order. dt checks the row count matches; reusing it on reordered rows is undefined.

prod clj

Product of the non-nil values in a column; nil for an all-missing column (an empty product of 1 would be misleading when every observation is missing). Delegates to expr/col-prod (shared with the #dt/e prod op).

qnt clj

(qnt col p)

(qnt col p min-n)

Column R type-7 p-quantile (p a fraction in [0,1]); matches R's quantile(x, p, type = 7, na.rm = TRUE). Drops nil and non-finite values. With min-n, returns nil when fewer than min-n finite values remain (floor-free by default). p may be a vector of probabilities, in which case the column is sorted once and a vector of quantiles is returned (the efficient q20/median/q80 band form). Also available as the qnt op in #dt/e and as an :agg/:set data-form [:qnt :col p].

rename clj

(rename dataset col-map)

Rename columns in a dataset without dropping any. col-map is {old-kw new-kw}.

reset-notes! clj

(reset-notes!)

Reset shown info notes. Useful for testing.

stddev clj

Column standard deviation. Full-name alias for dfn/standard-deviation.

sum clj

Column sum. Full-name alias for dfn/sum.

variance clj

Column variance. Full-name alias for dfn/variance.

xbar clj

(xbar col-kw width)

(xbar col-kw width unit)

Floor-division bucketing — floors a column value to the nearest multiple of width. Inspired by q's xbar operator.

For numeric columns: (xbar :price 10) → floor(:price / 10) * 10 For temporal columns: (xbar :time 5 :minutes) → floor to nearest 5-minute boundary

Supported temporal units: :seconds, :minutes, :hours, :days, :weeks (singular spellings — :second, :minute, … — are accepted everywhere)

Primary use case: computed :by grouping for time-series bar generation.

Usage: ;; Numeric bucketing in :by (dt ds :by [(xbar :price 10)] :agg {:n N :avg #dt/e (mn :volume)})

;; 5-minute OHLCV bars (-> trades (dt :order-by [(asc :time)]) (dt :by [(xbar :time 5 :minutes) :sym] :agg {:open #dt/e (first-val :price) :close #dt/e (last-val :price) :vol #dt/e (sm :size) :n N}))

;; Also usable inside #dt/e as a column derivation: (dt ds :set {:bucket #dt/e (xbar :price 5)})