tokenizers.core — net.clojars.savya/tokenizers-clj 0.10.0

Idiomatic Clojure wrapper over DJL's HuggingFace tokenizers, which bind the native Rust tokenizers library via JNI. Build a tokenizer with from-file / from-pretrained / from-stream, then encode, decode, or count-tokens.

A tokenizer holds a native handle: close it (with-open works) to free it.

assert-compatible-native-runtime! clj

(assert-compatible-native-runtime!)

(assert-compatible-native-runtime! os-name os-arch)

Fail early for known DJL native tokenizer runtime mismatches.

batch-count-tokens clj

(batch-count-tokens t texts)

(batch-count-tokens t
                    texts
                    {:keys [add-special-tokens? with-overflowing-tokens?]
                     :or {add-special-tokens? true
                          with-overflowing-tokens? false}})

Token-id counts for texts via native batch encoding.

batch-decode clj

(batch-decode t id-seqs)

(batch-decode t
              id-seqs
              {:keys [skip-special-tokens?] :or {skip-special-tokens? true}})

Decode many id sequences. Opts: :skip-special-tokens? (default true).

batch-encode clj

(batch-encode t texts)

(batch-encode t
              texts
              {:keys [add-special-tokens? with-overflowing-tokens?]
               :or {add-special-tokens? true with-overflowing-tokens? false}})

Encode many texts at once, returning a vector of encode-shaped maps. Accepts the same options as encode.

batch-encode->ndlist clj

(batch-encode->ndlist t texts manager)

(batch-encode->ndlist t
                      texts
                      manager
                      {:keys [add-special-tokens? with-overflowing-tokens?
                              with-token-type-ids? int32?]
                       :or {add-special-tokens? true
                            with-overflowing-tokens? false
                            with-token-type-ids? false
                            int32? false}})

Encode texts directly to one batched DJL NDList owned by manager. Opts include the batch-encode opts plus :with-token-type-ids? and :int32? (both default false). The caller owns and must close the NDManager.

batch-encode-pairs clj

(batch-encode-pairs t pairs)

(batch-encode-pairs t
                    pairs
                    {:keys [add-special-tokens? with-overflowing-tokens?]
                     :or {add-special-tokens? true
                          with-overflowing-tokens? false}})

Encode [text text-pair] pairs, returning encode-shaped maps.

batch-encode-pretokenized clj

(batch-encode-pretokenized t batches)

(batch-encode-pretokenized t batches opts)

Encode batches of already-split word strings while preserving native word ids. DJL 0.36.0 exposes no batch pretokenized operation, so each item uses the native pretokenized encode overload and retains its own overflow entries.

batch-ids clj

(batch-ids t texts)

(batch-ids t
           texts
           {:keys [add-special-tokens? with-overflowing-tokens?]
            :or {add-special-tokens? true with-overflowing-tokens? false}})

Token ids for texts via one native batch encoding.

batch-tokens clj

(batch-tokens t texts)

(batch-tokens t
              texts
              {:keys [add-special-tokens? with-overflowing-tokens?]
               :or {add-special-tokens? true with-overflowing-tokens? false}})

Token strings for texts via one native batch encoding.

build-sentence clj

(build-sentence t token-strings)

Reconstruct a sentence from token strings with the tokenizer's native token-joining rules. This is different from a decode of token ids.

builder clj

(builder)

(builder opts)

Create a DJL tokenizer builder configured from wrapper opts. Entries in :options or :raw-options pass through verbatim by DJL option name, such as modelMaxLength, stripAccents, and addPrefixSpace; keyword keys are converted with name. Raw entries override translated wrapper opts. :manager attaches the built tokenizer to a caller-supplied NDManager.

char->token clj

(char->token enc char-idx)

Token index containing char-idx in an encode result map, or nil.

count-tokens clj

(count-tokens t text)

(count-tokens t text opts)

Number of token ids text encodes to (see encode for opts).

decode clj

(decode t id-seq)

(decode t id-seq {:keys [skip-special-tokens?] :or {skip-special-tokens? true}})

Decode a seq of token id-seq back to text. Opts: :skip-special-tokens? (default true).

effective-config clj

(effective-config t)

Effective native truncation and padding configuration as a Clojure map.

encode clj

(encode t text)

(encode t text pair-or-opts)

(encode t text text-pair opts)

Encode text, optionally paired with a second text, into a token-data map. Opts: :add-special-tokens? (default true), :with-overflowing-tokens? (default false).

encode->ndlist clj

(encode->ndlist t text manager)

(encode->ndlist t
                text
                manager
                {:keys [add-special-tokens? with-overflowing-tokens?
                        with-token-type-ids? int32?]
                 :or {add-special-tokens? true
                      with-overflowing-tokens? false
                      with-token-type-ids? false
                      int32? false}})

Encode text directly to a DJL NDList owned by manager. Opts include the encode opts plus :with-token-type-ids? and :int32? (both default false). The caller owns and must close the NDManager.

encode-pretokenized clj

(encode-pretokenized t words)

(encode-pretokenized t
                     words
                     {:keys [add-special-tokens? with-overflowing-tokens?]
                      :or {add-special-tokens? true
                           with-overflowing-tokens? false}})

Encode an already-split sequence of word strings. This keeps the native word ids. Opts: :add-special-tokens? (default true), :with-overflowing-tokens? (default false).

from-bpe-files clj

(from-bpe-files vocab-path merges-path)

(from-bpe-files vocab-path merges-path opts)

BPE tokenizer from separate vocab.json and merges.txt paths. Accepts the constructor options documented by from-file, including raw :options / :raw-options.

from-file clj

(from-file path)

(from-file path opts)

Tokenizer from a tokenizer.json (path string, File, or Path). Constructor options include :truncation, :max-length, :stride, :padding, :pad-to-multiple-of, :add-special-tokens?, :lowercase?, and :tokenizer-config. See builder for raw options and :manager.

from-pretrained clj

(from-pretrained id)

(from-pretrained id opts)

Tokenizer by HuggingFace Hub id. Options include :revision, :auth-token, :cache-dir, :local-only? / :offline?, and :download-timeout-ms. :max-download-bytes limits each Hub response (100 MiB by default). Wrapper-managed Hub downloads cache both tokenizer.json and the optional tokenizer_config.json; DJL applies the latter's model and special-token metadata during construction. Wrapper-managed Hub downloads use a 30,000 ms connect and read timeout by default; :download-timeout-ms must be a positive integer.

from-stream clj

(from-stream is)

(from-stream is opts)

Tokenizer from an InputStream over a tokenizer.json, with constructor opts.

ids clj

(ids t text)

(ids t text opts)

Token ids for text (see encode for opts).

max-length clj

(max-length t)

Effective native maximum sequence length.

native-version clj

(native-version t)

Version of the loaded native tokenizers runtime.

pad-to-multiple-of clj

(pad-to-multiple-of t)

Effective native padding multiple.

padding clj

(padding t)

Effective native padding strategy as a keyword.

split-by-token-budget clj

(split-by-token-budget t text token-budget)

(split-by-token-budget t text token-budget opts)

Split text into native-token windows of at most token-budget tokens.

Returns encode-shaped maps with :text, :offset (Java character indexes), :ids, and overflow metadata. Special tokens count against the budget by default. Set :count-special-tokens? to false to exclude them while keeping tokenizer-added prefix/suffix specials in the first/last window. Other opts are the encode options, notably :add-special-tokens?. DJL 0.36.0 has no per-call budget API, so the full native encoding is measured before windows are partitioned.

split-by-tokens clj

stride clj

(stride t)

Effective native truncation stride.

token->chars clj

(token->chars enc token-idx)

Character span for token-idx in an encode result map, or nil.

token->word clj

(token->word enc token-idx)

Word index for token-idx in an encode result map, or nil.

tokenize clj

(tokenize t text)

Token strings for text without constructing an encode result.

tokenizer-config clj

(tokenizer-config path)

Load tokenizer configuration metadata from a tokenizer_config.json path. Returns nil when path is nil and preserves optional metadata as nil.

tokens clj

(tokens t text)

(tokens t text opts)

Token strings for text (see encode for opts).

truncate-by-token-budget clj

(truncate-by-token-budget t text token-budget)

(truncate-by-token-budget t text token-budget opts)

Return the first split-by-token-budget window, including overflow metadata. Returns nil when text produces no tokens. Special-token accounting and encode options match split-by-token-budget.

truncate-by-tokens clj

truncation clj

(truncation t)

Effective native truncation strategy as a keyword.

word->tokens clj

(word->tokens enc word-id)

Token indices for word-id in an encode result map.