bosquet.dataset.huggingface — io.github.zmedelis/bosquet v2023.11.21

Dataset fetching and storing from HuggingFace.

HF Datasets provide rich functionality https://huggingface.co/docs/datasets/index

Replicating it all would be a sizeable effort, here I have only functionality needed by Bosquet. TODO extract it to a separate OSS lib (even if it only covers basics of HF DS functionality)

download-ds clj

(download-ds {:keys [dataset offset length] :as params}
             {:hfds/keys [use-cache cache-dir record-limit]
              :or {use-cache false cache-dir ".cache/hugging-face-datasets"}})

hf-dataset clj

(hf-dataset hf-params)

(hf-dataset {:keys [dataset] :as hf-params}
            {:hfds/keys [use-cache cache-dir]
             :or {use-cache false cache-dir default-cache-dir}})

load-ds clj

(load-ds dataset)

(load-ds dataset ds-dir record-limit)