tech.ml.dataset.pipeline — techascent/tech.ml.dataset 1.0-alpha2

A set of common 'pipeline' operations you probably will want to run on a dataset.

->datatype clj

(->datatype dataset)

(->datatype dataset column-filter)

(->datatype dataset column-filter datatype)

Marshall columns to be the etl datatype. This changes numeric columns to be a unified backing store datatype.

assoc-metadata clj

(assoc-metadata dataset column-filter att-name att-value)

Assoc a new value into the metadata.

col clj

(col & [column-name])

Return a column. Only works during 'm=' and the default column is the current operating column.

filter clj

(filter dataset column-filter filter-fn)

Filter out indexes for which filter-fn produces a 0 or false value.

impute-missing clj

(impute-missing dataset)

(impute-missing dataset column-filter)

(impute-missing dataset column-filter k)

Group columns into K groups and impute missing values from the means calculated from those groups.

int-map clj

(int-map table col-data & {:keys [not-strict?]})

Perform an integer->integer conversion of a column using a static map. The map must be complete; missing entries are errors.

m= clj

(m= dataset column-filter operation)

Perform some math. Sets up variables such that the 'col' operator works.

new-column clj

(new-column dataset result-colname dataset-column-fn)

Create a new column. fn takes dataset and returns a reader, an iterable, or a new column.

one-hot clj

(one-hot dataset)

(one-hot dataset column-filter)

(one-hot dataset
         column-filter
         table-value-list
         &
         {:keys [datatype] :as op-args})

Replace string columns with one-hot encoded columns. table value list Argument can be nothing or a map containing keys representing the new derived column names and values representing which original values to encode to that particular column. The special keyword :rest indicates any remaining unencoded columns. example argument: {:main ["apple" "mandarin"] :other :rest}

pca clj

(pca dataset)

(pca dataset
     column-filter
     &
     {:keys [method variance n-components]
      :or {method :svd variance 0.95}
      :as op-args})

pif clj

(pif dataset bool-expr pipe-when-true pipe-when-false)

pipeline-inference-context clj macro

(pipeline-inference-context train-context & body)

pipeline-train-context clj macro

(pipeline-train-context & body)

pwhen clj

(pwhen dataset bool-expr pipe-when-true)

range-scale clj

(range-scale dataset)

(range-scale dataset column-filter)

(range-scale dataset column-filter value-range & {:keys [datatype] :as op-args})

Range-scale a set of columns to be within either [-1 1] or the range provided by the first argument. Will fail if columns have missing values.

read-var clj

(read-var varname)

remove-columns clj

(remove-columns dataset column-filter)

Remove columns selected by column-filter

remove-missing clj

(remove-missing dataset)

Remove any missing values from the dataset

replace clj

(replace dataset column-filter replace-value-or-fn & {:keys [result-datatype]})

Map a function across a column or set of columns. Map-fn may be a map?. Result column names are identical to src column names but metadata like a label map is removed. If map-setup-fn is provided, map-fn must be nil and map-setup-fn will be called with the dataset and column name to produce map-fn.

replace-missing clj

(replace-missing dataset column-filter missing-value)

Replace all the missing values in the dataset. Can take a sclar missing value or a callable fn. If callable fn, the fn is passed the dataset and column-name

std-scale clj

(std-scale dataset)

(std-scale dataset
           column-filter
           &
           {:keys [use-mean? use-std? datatype]
            :or {use-mean? true use-std? true}
            :as op-args})

Scale columns to have 0 mean and 1 std deviation. Will fail if columns contain missing values.

store-variables clj

(store-variables dataset varmap-fn)

string->number clj

(string->number dataset)

(string->number dataset column-filter)

(string->number dataset
                column-filter
                table-value-list
                &
                {:keys [datatype] :as op-args})

Convert all string columns to numeric recording the lookup table in the column metadata.

Replace any string values with numeric values. Updates the label map of the options. Arguments may be notion or a vector of either expected strings or tuples of expected strings to their hardcoded values.

training? clj

(training?)

update-column clj

(update-column dataset column-filter column-fn)

Update a column via a function. Function takes a column and returns a either a column, an iterable, or a reader.

update-dataset-column clj

(update-dataset-column dataset column-filter dataset-column-fn)

Update a column via a function. Function takes a dataset and a column and returns either a column, an iterable, or a reader.

with-column-name clj macro

(with-column-name colname & body)

with-column-name-seq clj macro

(with-column-name-seq colname-seq & body)

with-datatype clj macro

(with-datatype datatype & body)

with-ds clj macro

(with-ds dataset & body)

without-recording clj macro

(without-recording & body)