hfds-clj.core — io.github.zmedelis/hfds-clj v2023.11.05

Dataset fetching and storing from HuggingFace.

HF Datasets provide rich functionality https://huggingface.co/docs/datasets/index

download-ds clj

(download-ds {:keys [dataset offset length split] :as params}
             {:hfds/keys [cache-dir limit] :or {cache-dir default-cache-dir}})

Download data set from the HuggingFace. See load-dataset for params documentation

load-dataset clj

(load-dataset ds-params-or-ds-name)

(load-dataset {:keys [dataset split config offset length]
               :or {split "train" config "default" offset 0 length 100}}
              {:hfds/keys [cache-dir download-mode]
               :or {download-mode :reuse-dataset-if-exists
                    cache-dir default-cache-dir}
               :as read-params})

Download a dataset from HuggingFace. Dataset name is usually specified in HuggingFace dataset webpage. Usually in a form of org-name/ds-name

First argument can be

1. a map specifying HuggingFace HTTP call parameters and is used as is for HF REST API HTTP calls b) a string with dataset name, it will be converted into {:dataset ds-name} param map

Second argument is a map specifying how to read the ds.

read-ds clj

(read-ds {:keys [dataset split]}
         {:hfds/keys [limit cache-dir] :or {cache-dir default-cache-dir}})

Read data set from the cache. It is assumed that it is there. See load-dataset for params documentation.