clj-djl.dataframe — clj-djl/dataframe 0.1.1

->dataframe clj

->dataset clj

(->dataset dataset)

(->dataset dataset {:keys [table-name dataset-name] :as options})

Create a dataset from either csv/tsv or a sequence of maps.

• A String be interpreted as a file (or gzipped file if it ends with .gz) of tsv or csv data. The system will attempt to autodetect if this is csv or tsv and then engineering around detecting datatypes all of which can be overridden.

• InputStreams have no file type and thus a file-type must be provided in the options.

• A sequence of maps may be passed in in which case the first N maps are scanned in order to derive the column datatypes before the actual columns are created.

Parquet, xlsx, and xls formats require that you require the appropriate libraries which are tech.v3.libs.parquet for parquet, tech.v3.libs.fastexcel for xlsx, and tech.v3.libs.poi for xls.

Arrow support is provided via the tech.v3.libs.Arrow namespace not via a file-type overload as the Arrow project current has 3 different file types and it is not clear what their final suffix will be or which of the three file types it will indicate. Please see documentation in the tech.v3.libs.arrow namespace for further information on Arrow file types.

Options:

• :dataset-name - set the name of the dataset.
• :file-type - Override filetype discovery mechanism for strings or force a particular parser for an input stream. Note that parquet must have paths on disk and cannot currently load from input stream. Acceptible file types are: #{:csv :tsv :xlsx :xls :parquet}.
• :gzipped? - for file formats that support it, override autodetection and force creation of a gzipped input stream as opposed to a normal input stream.
• :column-whitelist - either sequence of string column names or sequence of column indices of columns to whitelist.
• :column-blacklist - either sequence of string column names or sequence of column indices of columns to blacklist.
• :num-rows - Number of rows to read
• :header-row? - Defaults to true, indicates the first row is a header.
• :key-fn - function to be applied to column names. Typical use is: :key-fn keyword.
• :separator - Add a character separator to the list of separators to auto-detect.
• :csv-parser - Implementation of univocity's AbstractParser to use. If not provided a default permissive parser is used. This way you parse anything that univocity supports (so flat files and such).
• :bad-row-policy - One of three options: :skip, :error, :carry-on. Defaults to :carry-on. Some csv data has ragged rows and in this case we have several options. If the option is :carry-on then we either create a new column or add missing values for columns that had no data for that row.
• :skip-bad-rows? - Legacy option. Use :bad-row-policy.
• :max-chars-per-column - Defaults to 4096. Columns with more characters that this will result in an exception.
• :max-num-columns - Defaults to 8192. CSV,TSV files with more columns than this will fail to parse. For more information on this option, please visit: https://github.com/uniVocity/univocity-parsers/issues/301
• :n-initial-skip-rows - Skip N rows initially. This currently may include the header row. Works across both csv and spreadsheet datasets.
• :parser-fn -
  • keyword? - all columns parsed to this datatype. For example: {:parser-fn :string}
  • map? - {column-name parse-method} parse each column with specified parse-method. The parse-method can be:
    • keyword? - parse the specified column to this datatype. For example: {:parser-fn {:answer :boolean :id :int32}}
    • tuple - pair of [datatype parse-data] in which case container of type [datatype] will be created. parse-data can be one of:
      • :relaxed? - data will be parsed such that parse failures of the standard parse functions do not stop the parsing process. :unparsed-values and :unparsed-indexes are available in the metadata of the column that tell you the values that failed to parse and their respective indexes.
      • fn? - function from str-> one of :tech.ml.dataset.parser/missing, :tech.ml.dataset.parser/parse-failure, or the parsed value. Exceptions here always kill the parse process. :missing will get marked in the missing indexes, and :parse-failure will result in the index being added to missing, the unparsed the column's :unparsed-values and :unparsed-indexes will be updated.
      • string? - for datetime types, this will turned into a DateTimeFormatter via DateTimeFormatter/ofPattern. For encoded-text, this has to be a valid argument to Charset/forName.
      • DateTimeFormatter - use with the appropriate temporal parse static function to parse the value.
• map? - the header-name-or-idx is used to lookup value. If not nil, then value can be any of the above options. Else the default column parser is used.

Returns a new dataset

->ndarray clj

(->ndarray ndm dataframe)

Convert dataframe to NDArray

add-column clj

(add-column dataset column)

Add a new column. Error if name collision

add-or-update-column clj

(add-or-update-column dataset column)

(add-or-update-column dataset colname column)

If column exists, replace. Else append new column.

assoc-ds clj

(assoc-ds dataset cname cdata & args)

If dataset is not nil, calls clojure.core/assoc. Else creates a new empty dataset and then calls clojure.core/assoc. Guaranteed to return a dataset (unlike assoc).

brief clj

(brief ds)

(brief ds options)

Get a brief description, in mapseq form of a dataset. A brief description is the mapseq form of descriptive stats.

categorical->one-hot clj

(categorical->one-hot dataset filter-fn-or-ds)

(categorical->one-hot dataset filter-fn-or-ds table-args)

(categorical->one-hot dataset filter-fn-or-ds table-args result-datatype)

Convert string columns to numeric columns. See tech.v3.dataset.categorical/fit-one-hot

column clj

(column dataset colname)

column-count clj

(column-count dataset)

column-names clj

(column-names dataset)

In-order sequence of column names

columns clj

(columns dataset)

Return sequence of all columns in dataset.

columns-with-missing-seq clj

(columns-with-missing-seq dataset)

Return a sequence of:

  {:column-name column-name
   :missing-count missing-count
  }

or nil of no columns are missing data.

concat clj

(concat dataset & datasets)

Concatenate datasets in place. See also concat-copying as it may be more efficient for your use case.

concat-copying clj

(concat-copying dataset & datasets)

Concatenate datasets into a new dataset copying data. Respects missing values. Datasets must all have the same columns. Result column datatypes will be a widening cast of the datatypes.

concat-inplace clj

(concat-inplace dataset & datasets)

Concatenate datasets in place. Respects missing values. Datasets must all have the same columns. Result column datatypes will be a widening cast of the datatypes.

data->dataset clj

(data->dataset {:keys [metadata columns] :as input})

Convert a data-ized dataset created via dataset->data back into a full dataset

dataset->data clj

(dataset->data ds)

Convert a dataset to a pure clojure datastructure. Returns a map with two keys: {:metadata :columns}. :columns is a vector of column definitions appropriate for passing directly back into new-dataset. A column definition in this case is a map of {:name :missing :data :metadata}.

dataset-name clj

(dataset-name dataset)

drop-columns clj

(drop-columns dataset col-name-seq)

Same as remove-columns

drop-missing clj

(drop-missing dataset-or-col)

Remove missing entries by simply selecting out the missing indexes

drop-rows clj

(drop-rows dataset-or-col row-indexes)

Drop rows from dataset or column

ensure-array-backed clj

(ensure-array-backed ds)

(ensure-array-backed ds {:keys [unpack?] :or {unpack? true}})

Ensure the column data in the dataset is stored in pure java arrays. This is sometimes necessary for interop with other libraries and this operation will force any lazy computations to complete. This also clears the missing set for each column and writes the missing values to the new arrays.

Columns that are already array backed and that have no missing values are not changed and retuned.

The postcondition is that dtype/->array will return a java array in the appropriate datatype for each column.

options - :unpack? - unpack packed datetime types. Defaults to true

filter clj

(filter dataset predicate)

dataset->dataset transformation. Predicate is passed a map of colname->column-value.

filter-column clj

(filter-column dataset colname predicate)

Filter a given column by a predicate. Predicate is passed column values. If predicate is not an instance of Ifn it is treated as a value and will be used as if the predicate is #(= value %). Returns a dataset.

group-by clj

(group-by dataset key-fn)

Produce a map of key-fn-value->dataset. key-fn is a function taking a map of colname->column-value. Selecting which columns are used in the key-fn using column-name-seq is optional but will greatly improve performance.

group-by->indexes clj

(group-by->indexes dataset key-fn)

(Non-lazy) - Group a dataset and return a map of key-fn-value->indexes where indexes is an in-order contiguous group of indexes.

group-by-column clj

(group-by-column dataset colname)

Return a map of column-value->dataset.

group-by-column->indexes clj

(group-by-column->indexes dataset colname)

(Non-lazy) - Group a dataset by a column return a map of column-val->indexes where indexes is an in-order contiguous group of indexes.

has-column? clj

(has-column? dataset column-name)

head clj

(head dataset)

(head dataset n)

Get the first n row of a dataset. Equivalent to `(select-rows ds (range n)). Arguments are reversed, however, so this can be used in ->> operators.

missing clj

(missing dataset-or-col)

Given a dataset or a column, return the missing set as a roaring bitmap

new-column clj

(new-column name data)

(new-column name data metadata)

(new-column name data metadata missing)

Create a new column. Data will scanned for missing values unless the full 4-argument pathway is used.

order-column-names clj

(order-column-names dataset colname-seq)

Order a sequence of columns names so they match the order in the original dataset. Missing columns are placed last.

remove-column clj

(remove-column dataset col-name)

Same as:

(dissoc dataset col-name)

remove-columns clj

(remove-columns dataset colname-seq)

Same as drop-columns

remove-rows clj

(remove-rows dataset-or-col row-indexes)

Same as drop-rows.

rename-columns clj

(rename-columns dataset colname-map)

Rename columns using a map. Does not reorder columns.

replace-missing clj

(replace-missing ds)

(replace-missing ds strategy)

(replace-missing ds columns-selector strategy)

(replace-missing ds columns-selector strategy value)

Replace missing values in some columns with a given strategy. The columns selector may be any legal argument to select-columns. Strategies may be:

• :down - take value from previous non-missing row if possible else use next non-missing row.
• :up - take value from next non-missing row if possible else use previous non-missing row.
• :mid - Use midpoint of averaged values between previous and next nonmissing rows.
• :lerp - Linearly interpolate values between previous and next nonmissing rows.
• :value - Value will be provided - see below. value may be provided which will then be used. Value may be a function in which case it will be called on the column with missing values elided and the return will be used to as the filler.

row-count clj

(row-count dataset-or-col)

select clj

(select dataset colname-seq index-seq)

Reorder/trim dataset according to this sequence of indexes. Returns a new dataset. colname-seq - one of:

• :all - all the columns
• sequence of column names - those columns in that order.
• implementation of java.util.Map - column order is dictate by map iteration order selected columns are subsequently named after the corresponding value in the map. similar to rename-columns except this trims the result to be only the columns in the map. index-seq - either keyword :all or list of indexes. May contain duplicates.

select-by-index clj

(select-by-index dataframe row-index col-index)

select-columns clj

(select-columns dataset col-name-seq)

Select columns from the dataset by seq of column names or :all.

select-columns-by-index clj

(select-columns-by-index dataset col-index)

Select columns from the dataset by seq of index(includes negative) or :all.

See documentation for select-by-index.

select-rows clj

(select-rows dataset-or-col row-indexes)

Select rows from the dataset or column.

select-rows-by-index clj

(select-rows-by-index dataset-or-col row-index)

Select rows from the dataset or column by seq of index(includes negative) or :all.

See documentation for select-by-index.

set-dataset-name clj

(set-dataset-name dataset ds-name)

shape clj

(shape dataframe)

Get the shape of dataset, row count first

sort-by clj

(sort-by dataset key-fn)

(sort-by dataset key-fn compare-fn)

Sort a dataset by a key-fn and compare-fn.

sort-by-column clj

(sort-by-column dataset colname)

(sort-by-column dataset colname compare-fn)

Sort a dataset by a given column using the given compare fn.

tail clj

(tail dataset)

(tail dataset n)

Get the last n rows of a dataset. Equivalent to `(select-rows ds (range ...)). Argument order is dataset-last, however, so this can be used in ->> operators.

take-nth clj

(take-nth dataset n-val)

unique-by clj

(unique-by dataset map-fn)

(unique-by dataset
           {:keys [keep-fn]
            :or {keep-fn (fn* [p1__25510# p2__25509#] (first p2__25509#))}
            :as _options}
           map-fn)

Map-fn function gets passed map for each row, rows are grouped by the return value. Keep-fn is used to decide the index to keep.

:keep-fn - Function from key,idx-seq->idx. Defaults to #(first %2).

unique-by-column clj

(unique-by-column dataset colname)

(unique-by-column dataset
                  {:keys [keep-fn]
                   :or {keep-fn (fn* [p1__25523# p2__25522#]
                                     (first p2__25522#))}
                   :as _options}
                  colname)

Map-fn function gets passed map for each row, rows are grouped by the return value. Keep-fn is used to decide the index to keep.

:keep-fn - Function from key, idx-seq->idx. Defaults to #(first %2).

unordered-select clj

(unordered-select dataset colname-seq index-seq)

Perform a selection but use the order of the columns in the existing table; do not reorder the columns based on colname-seq. Useful when doing selection based on sets or persistent hash maps.

update-column clj

(update-column dataset col-name update-fn)

Update a column returning a new dataset. update-fn is a column->column transformation. Error if column does not exist.

update-columns clj

(update-columns dataset column-name-seq update-fn)

Update a sequence of columns.