tokenizers.core — net.clojars.savya/tokenizers-clj 0.3.0

Idiomatic Clojure wrapper over DJL's HuggingFace tokenizers, which bind the native Rust tokenizers library via JNI. Build a tokenizer with from-file / from-pretrained / from-stream, then encode, decode, or count-tokens.

A tokenizer holds a native handle: close it (with-open works) to free it.

assert-compatible-native-runtime! clj

(assert-compatible-native-runtime!)

(assert-compatible-native-runtime! os-name os-arch)

Fail early for known DJL native tokenizer runtime mismatches.

batch-decode clj

(batch-decode t id-seqs)

(batch-decode t
              id-seqs
              {:keys [skip-special-tokens?] :or {skip-special-tokens? true}})

Decode many id sequences. Opts: :skip-special-tokens? (default true).

batch-encode clj

(batch-encode t texts)

(batch-encode t
              texts
              {:keys [add-special-tokens? with-overflowing-tokens?]
               :or {add-special-tokens? true with-overflowing-tokens? false}})

Encode many texts at once, returning a vector of encode-shaped maps. Accepts the same options as encode.

batch-encode->ndlist clj

(batch-encode->ndlist t texts manager)

(batch-encode->ndlist t
                      texts
                      manager
                      {:keys [add-special-tokens? with-overflowing-tokens?
                              with-token-type-ids? int32?]
                       :or {add-special-tokens? true
                            with-overflowing-tokens? false
                            with-token-type-ids? false
                            int32? false}})

Encode texts directly to one batched DJL NDList owned by manager. Opts include the batch-encode opts plus :with-token-type-ids? and :int32? (both default false). The caller owns and must close the NDManager.

batch-encode-pairs clj

(batch-encode-pairs t pairs)

(batch-encode-pairs t
                    pairs
                    {:keys [add-special-tokens? with-overflowing-tokens?]
                     :or {add-special-tokens? true
                          with-overflowing-tokens? false}})

Encode [text text-pair] pairs, returning encode-shaped maps.

build-sentence clj

(build-sentence t token-strings)

Reconstruct a sentence from token strings, applying the tokenizer's native token-joining rules. This is distinct from decoding token IDs.

builder clj

(builder)

(builder opts)

Create a DJL tokenizer builder configured from wrapper opts. Entries in :options or :raw-options pass through verbatim by DJL option name, such as modelMaxLength, stripAccents, and addPrefixSpace; keyword keys are converted with name. Raw entries override translated wrapper opts. :manager attaches the built tokenizer to a caller-supplied NDManager.

count-tokens clj

(count-tokens t text)

(count-tokens t text opts)

Number of token ids text encodes to (see encode for opts).

decode clj

(decode t id-seq)

(decode t id-seq {:keys [skip-special-tokens?] :or {skip-special-tokens? true}})

Decode a seq of token id-seq back to text. Opts: :skip-special-tokens? (default true).

effective-config clj

(effective-config t)

Effective native truncation and padding configuration as a Clojure map.

encode clj

(encode t text)

(encode t text pair-or-opts)

(encode t
        text
        text-pair
        {:keys [add-special-tokens? with-overflowing-tokens?]
         :or {add-special-tokens? true with-overflowing-tokens? false}})

Encode text, optionally paired with a second text, into a token-data map. Opts: :add-special-tokens? (default true), :with-overflowing-tokens? (default false).

encode->ndlist clj

(encode->ndlist t text manager)

(encode->ndlist t
                text
                manager
                {:keys [add-special-tokens? with-overflowing-tokens?
                        with-token-type-ids? int32?]
                 :or {add-special-tokens? true
                      with-overflowing-tokens? false
                      with-token-type-ids? false
                      int32? false}})

Encode text directly to a DJL NDList owned by manager. Opts include the encode opts plus :with-token-type-ids? and :int32? (both default false). The caller owns and must close the NDManager.

encode-pretokenized clj

(encode-pretokenized t words)

(encode-pretokenized t
                     words
                     {:keys [add-special-tokens? with-overflowing-tokens?]
                      :or {add-special-tokens? true
                           with-overflowing-tokens? false}})

Encode an already-split sequence of word strings, preserving native word IDs. Opts: :add-special-tokens? (default true), :with-overflowing-tokens? (default false).

from-bpe-files clj

(from-bpe-files vocab-path merges-path)

(from-bpe-files vocab-path merges-path opts)

BPE tokenizer from separate vocab.json and merges.txt paths. Accepts the constructor options documented by from-file, including raw :options / :raw-options.

from-file clj

(from-file path)

(from-file path opts)

Tokenizer from a tokenizer.json (path string, File, or Path). Constructor options include :truncation, :max-length, :stride, :padding, :pad-to-multiple-of, :add-special-tokens?, :lowercase?, and :tokenizer-config. See builder for raw options and :manager.

from-pretrained clj

(from-pretrained id)

(from-pretrained id opts)

Tokenizer by HuggingFace Hub id. Options include :revision, :auth-token, :cache-dir, and :local-only? / :offline?, plus constructor options.

from-stream clj

(from-stream is)

(from-stream is opts)

Tokenizer from an InputStream over a tokenizer.json, with constructor opts.

ids clj

(ids t text)

(ids t text opts)

Token ids for text (see encode for opts).

max-length clj

(max-length t)

Effective native maximum sequence length.

native-version clj

(native-version t)

Version of the loaded native tokenizers runtime.

pad-to-multiple-of clj

(pad-to-multiple-of t)

Effective native padding multiple.

padding clj

(padding t)

Effective native padding strategy as a keyword.

stride clj

(stride t)

Effective native truncation stride.

tokens clj

(tokens t text)

(tokens t text opts)

Token strings for text (see encode for opts).

truncation clj

(truncation t)

Effective native truncation strategy as a keyword.