tokenizers.core — net.clojars.savya/tokenizers-clj 0.1.0

Idiomatic Clojure wrapper over DJL's HuggingFace tokenizers, which bind the native Rust tokenizers library via JNI. Build a tokenizer with from-file / from-pretrained / from-stream, then encode, decode, or count-tokens.

A tokenizer holds a native handle: close it (with-open works) to free it.

batch-encode clj

(batch-encode t texts)

Encode many texts at once, returning a vector of encode-shaped maps.

count-tokens clj

(count-tokens t text)

(count-tokens t text opts)

Number of token ids text encodes to (see encode for opts).

decode clj

(decode t id-seq)

(decode t id-seq {:keys [skip-special-tokens?] :or {skip-special-tokens? true}})

Decode a seq of token id-seq back to text. Opts: :skip-special-tokens? (default true).

encode clj

(encode t text)

(encode t
        text
        {:keys [add-special-tokens? with-overflowing-tokens?]
         :or {add-special-tokens? true with-overflowing-tokens? false}})

Encode text into a map of :ids :tokens :type-ids :word-ids :attention-mask :special-tokens-mask. Opts: :add-special-tokens? (default true), :with-overflowing-tokens? (default false).

from-file clj

(from-file path)

Tokenizer from a tokenizer.json (path string, File, or Path).

from-pretrained clj

(from-pretrained id)

Tokenizer by HuggingFace hub id, e.g. "bert-base-uncased". Downloads then caches. Needs network on first use.

from-stream clj

(from-stream is)

Tokenizer from an InputStream over a tokenizer.json.

ids clj

(ids t text)

(ids t text opts)

Token ids for text (see encode for opts).

tokens clj

(tokens t text)

(tokens t text opts)

Token strings for text (see encode for opts).