scicloj.ml.metamorph — scicloj/scicloj.ml 0.1.0-beta1

This namespace contains functions, which operate on a metamorph context. They all return the context as well.

So all functions in this namespace are metamorph compliant and can be placed in a metamorph pipeline.

Most functions here are only manipulating the dataset, which is in the ctx map under the key :metamorph/data. And they behave the same in pipeline mode :fit and :transform.

A few functions manipulate other keys inside the ctx map, and/or behave different in :fit and :transform.

This is documented per function in this form:

metamorph	.
Behaviour in mode :fit	.
Behaviour in mode :transform	.
Reads keys from ctx	.
Writes keys to ctx	.

The namespaces scicloj.ml.metamorph and scicloj.ml.dataset contain functions with the same name. But they operate on either a context map (ns metamorph) or on a dataset (ns dataset)

The functions in this namesspaces are re-exported from :

• tablecloth.pipeline
• tech.v3.libs.smile.metamorph
• scicloj.metamorph.ml
• tech.v3.dataset.metamorph

->array clj

(->array colname)

(->array colname datatype)

Convert numerical column(s) to java array

add-column clj

(add-column column-name column)

(add-column column-name column size-strategy)

Add or update (modify) column under column-name.

column can be sequence of values or generator function (which gets ds as input).

add-columns clj

(add-columns columns-map)

(add-columns columns-map size-strategy)

Add or updade (modify) columns defined in columns-map (mapping: name -> column)

add-or-replace-column clj

(add-or-replace-column column-name column)

(add-or-replace-column column-name column size-strategy)

add-or-replace-columns clj

(add-or-replace-columns columns-map)

(add-or-replace-columns columns-map size-strategy)

add-or-update-column clj

(add-or-update-column column)

(add-or-update-column colname column)

aggregate clj

(aggregate aggregator)

(aggregate aggregator options)

Aggregate dataset by providing:

• aggregation function
• map with column names and functions
• sequence of aggregation functions

Aggregation functions can return:

• single value
• seq of values
• map of values with column names

aggregate-columns clj

(aggregate-columns columns-selector column-aggregators)

(aggregate-columns columns-selector column-aggregators options)

Aggregates each column separately

anti-join clj

(anti-join ds-right columns-selector)

(anti-join ds-right columns-selector options)

append clj

(append & datasets)

append-columns clj

(append-columns column-seq)

as-regular-dataset clj

(as-regular-dataset)

Remove grouping tag

asof-join clj

(asof-join ds-right colname)

(asof-join ds-right colname options)

assoc-ds clj

(assoc-ds cname cdata & args)

assoc-metadata clj

(assoc-metadata filter-fn-or-ds k v & args)

bind clj

(bind & datasets)

bow->something-sparse clj

(bow->something-sparse bow-col indices-col bow->sparse-fn options)

Converts a bag-of-word column bow-col to a sparse data column indices-col. The exact transformation to the sparse representtaion is given by bow->sparse-fn

metamorph	.
Behaviour in mode :fit	normal
Behaviour in mode :transform	normal
Reads keys from ctx	none
Writes keys to ctx	:scicloj.ml.smile.metamorph/bow->sparse-vocabulary

bow->sparse-array clj

(bow->sparse-array bow-col indices-col)

(bow->sparse-array bow-col indices-col options)

Converts a bag-of-word column bow-col to sparse indices column indices-col, as needed by the Maxent model. vocab size is the size of vocabluary used, sorted by token frequency

metamorph	.
Behaviour in mode :fit	normal
Behaviour in mode :transform	normal
Reads keys from ctx	none
Writes keys to ctx	:scicloj.ml.smile.metamorph/count-vectorize-vocabulary

bow->SparseArray clj

(bow->SparseArray bow-col indices-col)

(bow->SparseArray bow-col indices-col options)

Converts a bag-of-word column bow-col to sparse indices column indices-col, as needed by the discrete naive bayes model. vocab size is the size of vocabluary used, sorted by token frequency

metamorph	.
Behaviour in mode :fit	normal
Behaviour in mode :transform	normal
Reads keys from ctx	none
Writes keys to ctx	:scicloj.ml.smile.metamorph/count-vectorize-vocabulary

bow->tfidf clj

(bow->tfidf bow-column tfidf-column)

Calculates the tfidf score from bag-of-words (as token frequency maps) in column bow-column and stores them in a new column tfid-column as maps of token->tfidf-score.

metamorph	.
Behaviour in mode :fit	normal
Behaviour in mode :transform	normal
Reads keys from ctx	none
Writes keys to ctx	none

brief clj

(brief)

(brief options)

by-rank clj

(by-rank columns-selector rank-predicate)

(by-rank columns-selector rank-predicate options)

Select rows using rank on a column, ties are resolved using :dense method.

See R docs. Rank uses 0 based indexing.

Possible :ties strategies: :average, :first, :last, :random, :min, :max, :dense. :dense is the same as in data.table::frank from R

:desc? set to true (default) order descending before calculating rank

categorical->number clj

(categorical->number filter-fn-or-ds)

(categorical->number filter-fn-or-ds table-args)

(categorical->number filter-fn-or-ds table-args result-datatype)

categorical->one-hot clj

(categorical->one-hot filter-fn-or-ds)

(categorical->one-hot filter-fn-or-ds table-args)

(categorical->one-hot filter-fn-or-ds table-args result-datatype)

clone clj

(clone)

Clone an object. Can clone anything convertible to a reader.

column clj

(column colname)

column->dataset clj

(column->dataset colname transform-fn)

(column->dataset colname transform-fn options)

column-cast clj

(column-cast colname datatype)

column-count clj

(column-count)

column-labeled-mapseq clj

(column-labeled-mapseq value-colname-seq)

column-map clj

(column-map result-colname map-fn)

(column-map result-colname map-fn filter-fn-or-ds)

(column-map result-colname map-fn res-dtype filter-fn-or-ds)

column-names clj

(column-names)

(column-names columns-selector)

(column-names columns-selector meta-field)

column-values->categorical clj

(column-values->categorical src-column)

columns clj

(columns)

(columns result-type)

columns-with-missing-seq clj

(columns-with-missing-seq)

columnwise-concat clj

(columnwise-concat colnames)

(columnwise-concat colnames options)

concat clj

(concat & datasets)

concat-copying clj

(concat-copying & datasets)

concat-inplace clj

(concat-inplace & datasets)

convert-types clj

(convert-types coltype-map-or-columns-selector)

(convert-types columns-selector new-types)

Convert type of the column to the other type.

count-vectorize clj

(count-vectorize text-col bow-col)

(count-vectorize text-col bow-col options)

Transforms the text column text-col into a map of token frequencies in column bow-col

metamorph	.
Behaviour in mode :fit	normal
Behaviour in mode :transform	normal
Reads keys from ctx	none
Writes keys to ctx	none

data->dataset clj

(data->dataset)

dataset->categorical-xforms clj

(dataset->categorical-xforms)

dataset->data clj

(dataset->data)

dataset->str clj

(dataset->str)

(dataset->str options)

Convert a dataset to a string. Prints a single line header and then calls dataset-data->str.

For options documentation see dataset-data->str.

dataset-name clj

(dataset-name)

dataset? clj

(dataset?)

Is ds a dataset type?

descriptive-stats clj

(descriptive-stats)

(descriptive-stats options)

difference clj

(difference ds-right)

(difference ds-right options)

drop clj

(drop columns-selector rows-selector)

Drop columns and rows.

drop-columns clj

(drop-columns)

(drop-columns columns-selector)

(drop-columns columns-selector meta-field)

Drop columns by (returns dataset):

• name
• sequence of names
• map of names with new names (rename)
• function which filter names (via column metadata)

drop-missing clj

(drop-missing)

(drop-missing columns-selector)

Drop rows with missing values

columns-selector selects columns to look at missing values

drop-rows clj

(drop-rows)

(drop-rows rows-selector)

(drop-rows rows-selector options)

Drop rows using:

• row id
• seq of row ids
• seq of true/false
• fn with predicate

empty-ds? clj

(empty-ds?)

ensure-array-backed clj

(ensure-array-backed)

(ensure-array-backed options)

feature-ecount clj

(feature-ecount)

fill-range-replace clj

(fill-range-replace colname max-span)

(fill-range-replace colname max-span missing-strategy)

(fill-range-replace colname max-span missing-strategy missing-value)

filter clj

(filter predicate)

filter-column clj

(filter-column colname predicate)

filter-dataset clj

(filter-dataset filter-fn-or-ds)

first clj

(first)

fold-by clj

(fold-by columns-selector)

(fold-by columns-selector folding-function)

full-join clj

(full-join ds-right columns-selector)

(full-join ds-right columns-selector options)

group-by clj

(group-by grouping-selector)

(group-by grouping-selector options)

Group dataset by:

• column name
• list of columns
• map of keys and row indexes
• function getting map of values

Options are:

• select-keys - when grouping is done by function, you can limit fields to a select-keys seq.
• result-type - return results as dataset (:as-dataset, default) or as map of datasets (:as-map) or as map of row indexes (:as-indexes) or as sequence of (sub)datasets
• other parameters which are passed to dataset fn

When dataset is returned, meta contains :grouped? set to true. Columns in dataset:

• name - group name
• group-id - id of the group (int)
• data - group as dataset

group-by->indexes clj

(group-by->indexes key-fn)

group-by-column clj

(group-by-column colname)

group-by-column->indexes clj

(group-by-column->indexes colname)

grouped? clj

(grouped?)

Is dataset represents grouped dataset (result of group-by)?

groups->map clj

(groups->map)

Convert grouped dataset to the map of groups

groups->seq clj

(groups->seq)

has-column? clj

(has-column? column-name)

head clj

(head)

(head n)

inference-column? clj

(inference-column?)

inference-target-column-names clj

(inference-target-column-names)

inference-target-ds clj

(inference-target-ds)

inference-target-label-inverse-map clj

(inference-target-label-inverse-map & [label-columns])

inference-target-label-map clj

(inference-target-label-map & [label-columns])

info clj

(info)

(info result-type)

inner-join clj

(inner-join ds-right columns-selector)

(inner-join ds-right columns-selector options)

intersect clj

(intersect ds-right)

(intersect ds-right options)

join-columns clj

(join-columns target-column columns-selector)

(join-columns target-column columns-selector options)

labels clj

(labels)

last clj

(last)

left-join clj

(left-join ds-right columns-selector)

(left-join ds-right columns-selector options)

map-columns clj

(map-columns column-name map-fn)

(map-columns column-name columns-selector map-fn)

(map-columns column-name new-type columns-selector map-fn)

mapseq-reader clj

(mapseq-reader)

mark-as-group clj

(mark-as-group)

Add grouping tag

missing clj

(missing)

model clj

(model options)

Executes a machine learning model in train/predict (depending on :mode) from the metamorph.ml model registry.

The model is passed between both invocation via the shared context ctx in a key (a step indentifier) which is passed in key :metamorph/id and guarantied to be unique for each pipeline step.

The function writes and reads into this common context key.

Options:

• :model-type - Keyword for the model to use

Further options get passed to train functions and are model specific.

See here for an overview for the models build into scicloj.ml:

https://scicloj.github.io/scicloj.ml/userguide-models.html

Other libraries might contribute other models, which are documented as part of the library.

metamorph	.
Behaviour in mode :fit	Calls scicloj.metamorph.ml/train using data in :metamorph/data and optionsand stores trained model in ctx under key in :metamorph/id
Behaviour in mode :transform	Reads trained model from ctx and calls scicloj.metamorph.ml/predict with the model in $id and data in :metamorph/data
Reads keys from ctx	In mode :transform : Reads trained model to use for prediction from key in :metamorph/id.
Writes keys to ctx	In mode :fit : Stores trained model in key $id and writes feature-ds and target-ds before prediction into ctx at :scicloj.metamorph.ml/target-ds /:scicloj.metamorph.ml/target-ds

See as well:

• scicloj.metamorph.ml/train
• scicloj.metamorph.ml/predict

model-type clj

(model-type & [column-name-seq])

new-column clj

(new-column data)

(new-column data metadata)

(new-column data metadata missing)

new-dataset clj

(new-dataset)

(new-dataset column-seq)

(new-dataset ds-metadata column-seq)

num-inference-classes clj

(num-inference-classes)

order-by clj

(order-by columns-or-fn)

(order-by columns-or-fn comparators)

(order-by columns-or-fn comparators options)

Order dataset by:

• column name
• columns (as sequence of names)
• key-fn
• sequence of columns / key-fn Additionally you can ask the order by:
• :asc
• :desc
• custom comparator function

order-column-names clj

(order-column-names colname-seq)

pivot->longer clj

(pivot->longer)

(pivot->longer columns-selector)

(pivot->longer columns-selector options)

tidyr pivot_longer api

pivot->wider clj

(pivot->wider columns-selector value-columns)

(pivot->wider columns-selector value-columns options)

print-dataset clj

(print-dataset)

(print-dataset options)

probability-distributions->label-column clj

(probability-distributions->label-column dst-colname)

process-group-data clj

(process-group-data f)

(process-group-data f parallel?)

rand-nth clj

(rand-nth)

(rand-nth options)

random clj

(random)

(random n)

(random n options)

read-nippy clj

(read-nippy)

remove-column clj

(remove-column col-name)

remove-columns clj

(remove-columns colname-seq-or-fn)

remove-rows clj

(remove-rows row-indexes)

rename-columns clj

(rename-columns columns-mapping)

(rename-columns columns-selector columns-map-fn)

Rename columns with provided old -> new name map

reorder-columns clj

(reorder-columns columns-selector & columns-selectors)

Reorder columns using column selector(s). When column names are incomplete, the missing will be attached at the end.

replace-missing clj

(replace-missing)

(replace-missing strategy)

(replace-missing columns-selector strategy)

(replace-missing columns-selector strategy value)

replace-missing-value clj

(replace-missing-value scalar-value)

(replace-missing-value filter-fn-or-ds scalar-value)

right-join clj

(right-join ds-right columns-selector)

(right-join ds-right columns-selector options)

row-count clj

(row-count)

rows clj

(rows)

(rows result-type)

sample clj

(sample)

(sample n)

(sample n options)

select clj

(select columns-selector rows-selector)

Select columns and rows.

select-by-index clj

(select-by-index col-index row-index)

select-columns clj

(select-columns)

(select-columns columns-selector)

(select-columns columns-selector meta-field)

Select columns by (returns dataset):

• name
• sequence of names
• map of names with new names (rename)
• function which filter names (via column metadata)

select-columns-by-index clj

(select-columns-by-index col-index)

select-missing clj

(select-missing)

(select-missing columns-selector)

Select rows with missing values

columns-selector selects columns to look at missing values

select-rows clj

(select-rows)

(select-rows rows-selector)

(select-rows rows-selector options)

Select rows using:

• row id
• seq of row ids
• seq of true/false
• fn with predicate

select-rows-by-index clj

(select-rows-by-index row-index)

semi-join clj

(semi-join ds-right columns-selector)

(semi-join ds-right columns-selector options)

separate-column clj

(separate-column column separator)

(separate-column column target-columns separator)

(separate-column column target-columns separator options)

set-dataset-name clj

(set-dataset-name ds-name)

set-inference-target clj

(set-inference-target target-name-or-target-name-seq)

shape clj

(shape)

Returns shape of the dataset [rows, cols]

shuffle clj

(shuffle)

(shuffle options)

sort-by clj

(sort-by key-fn)

(sort-by key-fn compare-fn)

sort-by-column clj

(sort-by-column colname)

(sort-by-column colname compare-fn)

tail clj

(tail)

(tail n)

take-nth clj

(take-nth n-val)

ungroup clj

(ungroup)

(ungroup options)

Concat groups into dataset.

When add-group-as-column or add-group-id-as-column is set to true or name(s), columns with group name(s) or group id is added to the result.

Before joining the groups groups can be sorted by group name.

union clj

(union & datasets)

unique-by clj

(unique-by)

(unique-by columns-selector)

(unique-by columns-selector options)

unique-by-column clj

(unique-by-column colname)

(unique-by-column options colname)

unmark-group clj

(unmark-group)

Remove grouping tag

unordered-select clj

(unordered-select colname-seq index-seq)

unroll clj

(unroll columns-selector)

(unroll columns-selector options)

unroll-column clj

(unroll-column column-name)

(unroll-column column-name options)

update clj

(update filter-fn-or-ds update-fn & args)

update-column clj

(update-column col-name update-fn)

update-columns clj

(update-columns columns-map)

(update-columns columns-selector update-functions)

update-columnwise clj

(update-columnwise filter-fn-or-ds cwise-update-fn & args)

update-elemwise clj

(update-elemwise map-fn)

(update-elemwise filter-fn-or-ds map-fn)

value-reader clj

(value-reader)

write! clj

(write! output-path)

(write! output-path options)

Write a dataset out to a file. Supported forms are:

(ds/write! test-ds "test.csv")
(ds/write! test-ds "test.tsv")
(ds/write! test-ds "test.tsv.gz")
(ds/write! test-ds "test.nippy")
(ds/write! test-ds out-stream)

Options:

• :max-chars-per-column - csv,tsv specific, defaults to 65536 - values longer than this will cause an exception during serialization.
• :max-num-columns - csv,tsv specific, defaults to 8192 - If the dataset has more than this number of columns an exception will be thrown during serialization.
• :quoted-columns - csv specific - sequence of columns names that you would like to always have quoted.
• :file-type - Manually specify the file type. This is usually inferred from the filename but if you pass in an output stream then you will need to specify the file type.
• :headers? - if csv headers are written, defaults to true.

write-nippy! clj

(write-nippy! filename)