Reshape functions for datajure: wide->long (melt), long->wide (cast), and panel gap-filling (tsfill).
Reshape functions for datajure: wide->long (melt), long->wide (cast), and panel gap-filling (tsfill).
(cast dataset {:keys [id from value agg fill] :or {fill nil}})Reshape a dataset from long to wide format. Complement to melt.
For each unique combination of :id column values, pivots the :from column's distinct values into new columns filled from the :value column. New column names are derived from the :from values (keywords passed through; strings and other types converted via keyword). They appear in order of first occurrence in the :from column.
Arguments: dataset - a tech.v3.dataset opts - map with keys: :id - (required) column keyword or vector of column keywords to use as row identifiers. A single keyword is normalised to a one-element vector, matching melt. :from - (required) column keyword whose unique values become new column names :value - (required) column keyword whose values fill the new columns :agg - aggregation fn applied to a vector of values when multiple rows share the same (id, from) combination. Default: use the first value. :fill - value for cells with no matching (id, from) row. Default: nil.
Examples: ;; Reverse a melt (-> ds (melt {:id [:species :year] :measure [:mass :flipper]}) (cast {:id [:species :year] :from :variable :value :value}))
;; Single-keyword :id also works (cast ds {:id :year :from :metric :value :val})
;; With aggregation for duplicate cells (cast ds {:id [:date :sym] :from :metric :value :val :agg dfn/mean})
Reshape a dataset from long to wide format. Complement to melt.
For each unique combination of :id column values, pivots the :from column's
distinct values into new columns filled from the :value column. New column
names are derived from the :from values (keywords passed through; strings
and other types converted via keyword). They appear in order of first
occurrence in the :from column.
Arguments:
dataset - a tech.v3.dataset
opts - map with keys:
:id - (required) column keyword or vector of column keywords to use
as row identifiers. A single keyword is normalised to a
one-element vector, matching melt.
:from - (required) column keyword whose unique values become new column names
:value - (required) column keyword whose values fill the new columns
:agg - aggregation fn applied to a vector of values when multiple rows
share the same (id, from) combination. Default: use the first value.
:fill - value for cells with no matching (id, from) row. Default: nil.
Examples:
;; Reverse a melt
(-> ds
(melt {:id [:species :year] :measure [:mass :flipper]})
(cast {:id [:species :year] :from :variable :value :value}))
;; Single-keyword :id also works
(cast ds {:id :year :from :metric :value :val})
;; With aggregation for duplicate cells
(cast ds {:id [:date :sym] :from :metric :value :val :agg dfn/mean})(melt dataset
{:keys [id measure variable-col value-col]
:or {variable-col :variable value-col :value}})Reshape a dataset from wide to long format.
Arguments: dataset - a tech.v3.dataset opts - map with keys: :id - (required) vector of column keywords to keep as identifiers :measure - vector of column keywords to stack. Defaults to all non-id columns. :variable-col - keyword for the new variable column. Defaults to :variable. :value-col - keyword for the new value column. Defaults to :value.
Returns a dataset with one row per (id, measure) combination. If no measure columns exist (all columns are id columns), returns an empty dataset with the id columns plus the variable and value columns.
Examples: ;; Basic melt (melt ds {:id [:species :year] :measure [:mass :flipper :bill]})
;; Infer measure cols (all non-id) (melt ds {:id [:species :year]})
;; Custom output column names (melt ds {:id [:species :year] :measure [:mass :flipper] :variable-col :metric :value-col :val})
Reshape a dataset from wide to long format.
Arguments:
dataset - a tech.v3.dataset
opts - map with keys:
:id - (required) vector of column keywords to keep as identifiers
:measure - vector of column keywords to stack. Defaults to all non-id columns.
:variable-col - keyword for the new variable column. Defaults to :variable.
:value-col - keyword for the new value column. Defaults to :value.
Returns a dataset with one row per (id, measure) combination.
If no measure columns exist (all columns are id columns), returns an empty dataset
with the id columns plus the variable and value columns.
Examples:
;; Basic melt
(melt ds {:id [:species :year] :measure [:mass :flipper :bill]})
;; Infer measure cols (all non-id)
(melt ds {:id [:species :year]})
;; Custom output column names
(melt ds {:id [:species :year] :measure [:mass :flipper]
:variable-col :metric :value-col :val})(tsfill dataset {:keys [by date every grid carry]})Fill gaps in a panel: expand each group to a regular date grid, inserting
missing rows (nil in every non-key column). Stata's tsfill; essentially
tidyr's complete for panel time series. Follows tidyr's union semantics:
existing rows are all KEPT (even off-grid ones) and grid rows are added
where no exact date match exists — no data is ever dropped by gridding.
Arguments:
dataset - a tech.v3.dataset
opts - map with keys:
:date - (required) the date column keyword.
:by - group key column(s), keyword or vector. Omit to treat the whole
dataset as one series.
:every - grid step: a positive number for numeric date columns
(default 1), or :day/:week/:month/:quarter/:year for temporal
columns (default :day). Temporal grids are anchored at each
group's min date; matching is exact, so panels keyed on
month-end dates should be normalised first (e.g. an xbar
bucket) — the same caveat as win/tlag.
:grid - explicit sequence of date values to use as the grid instead of
:every (e.g. actual trading days); it is deduplicated, sorted,
and clipped to each group's [min, max] date range.
:carry - column(s) to carry across the inserted rows after expansion,
per group: backward-fill then forward-fill (NOCB→LOCF,
mbmisc tsfill carrycols) — e.g. a company name column.
The grid spans each group's own min..max date (no rows are invented before a group's first or after its last observation). Rows whose date is nil are dropped (they cannot be placed on a grid); a group with only nil dates vanishes. nil group keys throw :tsfill-nil-key; duplicate (key, date) combinations throw :tsfill-duplicate-dates (aggregate duplicates first, e.g. :agg by the keys). Output rows are sorted by [keys…, date] — unlike :set, input order is not meaningful once rows are inserted.
Example: (tsfill ds {:by [:gvkey] :date :fyear}) ;; yearly panel (tsfill ds {:by [:permno] :date :month :every :month :carry [:ticker]})
Fill gaps in a panel: expand each group to a regular date grid, inserting
missing rows (nil in every non-key column). Stata's `tsfill`; essentially
tidyr's `complete` for panel time series. Follows tidyr's union semantics:
existing rows are all KEPT (even off-grid ones) and grid rows are added
where no exact date match exists — no data is ever dropped by gridding.
Arguments:
dataset - a tech.v3.dataset
opts - map with keys:
:date - (required) the date column keyword.
:by - group key column(s), keyword or vector. Omit to treat the whole
dataset as one series.
:every - grid step: a positive number for numeric date columns
(default 1), or :day/:week/:month/:quarter/:year for temporal
columns (default :day). Temporal grids are anchored at each
group's min date; matching is exact, so panels keyed on
month-end dates should be normalised first (e.g. an xbar
bucket) — the same caveat as win/tlag.
:grid - explicit sequence of date values to use as the grid instead of
:every (e.g. actual trading days); it is deduplicated, sorted,
and clipped to each group's [min, max] date range.
:carry - column(s) to carry across the inserted rows after expansion,
per group: backward-fill then forward-fill (NOCB→LOCF,
mbmisc `tsfill` carrycols) — e.g. a company name column.
The grid spans each group's own min..max date (no rows are invented before
a group's first or after its last observation). Rows whose date is nil are
dropped (they cannot be placed on a grid); a group with only nil dates
vanishes. nil group keys throw :tsfill-nil-key; duplicate (key, date)
combinations throw :tsfill-duplicate-dates (aggregate duplicates first,
e.g. :agg by the keys). Output rows are sorted by [keys…, date] — unlike
:set, input order is not meaningful once rows are inserted.
Example:
(tsfill ds {:by [:gvkey] :date :fyear}) ;; yearly panel
(tsfill ds {:by [:permno] :date :month :every :month
:carry [:ticker]})cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |