tokenizers.core — net.clojars.savya/tokenizers-clj 0.2.0

Idiomatic Clojure wrapper over DJL's HuggingFace tokenizers, which bind the native Rust tokenizers library via JNI. Build a tokenizer with from-file / from-pretrained / from-stream, then encode, decode, or count-tokens.

A tokenizer holds a native handle: close it (with-open works) to free it.

assert-compatible-native-runtime! clj

(assert-compatible-native-runtime!)

(assert-compatible-native-runtime! os-name os-arch)

Fail early for known DJL native tokenizer runtime mismatches.

batch-decode clj

(batch-decode t id-seqs)

(batch-decode t
              id-seqs
              {:keys [skip-special-tokens?] :or {skip-special-tokens? true}})

Decode many id sequences. Opts: :skip-special-tokens? (default true).

batch-encode clj

(batch-encode t texts)

(batch-encode t
              texts
              {:keys [add-special-tokens? with-overflowing-tokens?]
               :or {add-special-tokens? true with-overflowing-tokens? false}})

Encode many texts at once, returning a vector of encode-shaped maps. Accepts the same options as encode.

batch-encode-pairs clj

(batch-encode-pairs t pairs)

(batch-encode-pairs t
                    pairs
                    {:keys [add-special-tokens? with-overflowing-tokens?]
                     :or {add-special-tokens? true
                          with-overflowing-tokens? false}})

Encode [text text-pair] pairs, returning encode-shaped maps.

count-tokens clj

(count-tokens t text)

(count-tokens t text opts)

Number of token ids text encodes to (see encode for opts).

decode clj

(decode t id-seq)

(decode t id-seq {:keys [skip-special-tokens?] :or {skip-special-tokens? true}})

Decode a seq of token id-seq back to text. Opts: :skip-special-tokens? (default true).

encode clj

(encode t text)

(encode t text pair-or-opts)

(encode t
        text
        text-pair
        {:keys [add-special-tokens? with-overflowing-tokens?]
         :or {add-special-tokens? true with-overflowing-tokens? false}})

Encode text, optionally paired with a second text, into a token-data map. Opts: :add-special-tokens? (default true), :with-overflowing-tokens? (default false).

from-file clj

(from-file path)

(from-file path opts)

Tokenizer from a tokenizer.json (path string, File, or Path). Constructor options include :truncation, :max-length, :stride, :padding, :pad-to-multiple-of, :add-special-tokens?, :lowercase?, and :tokenizer-config.

from-pretrained clj

(from-pretrained id)

(from-pretrained id opts)

Tokenizer by HuggingFace Hub id. Options include :revision, :auth-token, :cache-dir, and :local-only? / :offline?, plus constructor options.

from-stream clj

(from-stream is)

(from-stream is opts)

Tokenizer from an InputStream over a tokenizer.json, with constructor opts.

ids clj

(ids t text)

(ids t text opts)

Token ids for text (see encode for opts).

tokens clj

(tokens t text)

(tokens t text opts)

Token strings for text (see encode for opts).