tokenizers-clj

Idiomatic Clojure tokenization: tokenize, encode, decode, and count tokens against any HuggingFace tokenizer.json, backed by the native Rust tokenizers library.

Stack

A thin Clojure wrapper over DJL's ai.djl.huggingface/tokenizers. DJL binds the same Rust tokenizers library that HuggingFace ships for Python. The wrapper gives you exact token counts and ids for BERT, GPT, Llama, Qwen, and any other model that publishes a tokenizer.json.

Install

deps.edn:

net.clojars.savya/tokenizers-clj {:mvn/version "0.7.0"}

Leiningen / Boot:

[net.clojars.savya/tokenizers-clj "0.7.0"]

Usage

(require '[tokenizers.core :as tok])

;; From a local tokenizer.json ...
(with-open [t (tok/from-file "bert-base-uncased/tokenizer.json")]
  (tok/count-tokens t "Hello, world!"))          ;=> 6

;; Get token strings directly, without an encode result map:
(with-open [t (tok/from-file "bert-base-uncased/tokenizer.json")]
  (tok/tokenize t "Hello, world!"))              ;=> ["[CLS]" "hello" "," "world" "!" "[SEP]"]

;; ... or straight from the HuggingFace hub (downloads + caches both tokenizer
;; files once, including tokenizer_config.json when the model provides it).
(with-open [t (tok/from-pretrained "bert-base-uncased")]
  (tok/encode t "Hello, world!"))
;=> {:ids [101 7592 1010 2088 999 102]
;    :tokens ["[CLS]" "hello" "," "world" "!" "[SEP]"]
;    :attention-mask [1 1 1 1 1 1]
;    :type-ids [0 0 0 0 0 0] :word-ids [...]
;    :special-tokens-mask [1 0 0 0 0 1]
;    :offsets [[0 0] [0 5] [5 6] [7 12] [12 13] [0 0]]
;    :sequence-ids [...] :overflow [] :exceed-max-length? false}

;; Drop the framing special tokens for a raw count:
(with-open [t (tok/from-pretrained "bert-base-uncased")]
  (tok/count-tokens t "Hello, world!" {:add-special-tokens? false}))  ;=> 4

;; Round-trip:
(with-open [t (tok/from-pretrained "bert-base-uncased")]
  (tok/decode t (tok/ids t "hello there" {:add-special-tokens? false})))  ;=> "hello there"

Construction options

from-file, from-stream, and from-pretrained accept an options map:

• :truncation: :longest-first, :only-first, :only-second, or :none (booleans are also accepted).
• :max-length and :stride: truncation size and overlap.
• :padding: :longest, :max-length, or :none (booleans are also accepted).
• :pad-to-multiple-of: pad encoded lengths to a multiple.
• :add-special-tokens?, :with-overflowing-tokens?, and :lowercase?: tokenizer behavior flags.
• :tokenizer-config: path, File, or Path to a tokenizer_config.json.

from-pretrained also accepts :revision, :auth-token, :cache-dir, and :local-only? / :offline?. If you supply a revision, cache, or offline option, the library uses a revision-specific local cache. In an offline mode the library fails without a network request when the tokenizer is absent.

Hub loading also downloads tokenizer_config.json when available and passes it to DJL. Its model_max_length is reflected by max-length, effective-config, and :exceed-max-length?; BOS/EOS/UNK/PAD and other special token metadata is applied by DJL during tokenization and padding. A missing tokenizer_config.json remains compatible with repositories that publish only tokenizer.json.

(with-open [t (tok/from-pretrained
               "bert-base-uncased"
               {:revision "main"
                :cache-dir ".cache/tokenizers"
                :truncation :longest-first
                :max-length 128
                :stride 16
                :padding :max-length
                :pad-to-multiple-of 8})]
  (tok/encode t "A question" "A paired answer"))

Encode results and batches

Every encode, batch-encode, and batch-encode-pairs result contains :ids, :tokens, :type-ids, :word-ids, :attention-mask, :special-tokens-mask, :offsets, :sequence-ids, :overflow, and :exceed-max-length?. Overflow entries have the same shape.

encode accepts a second string for paired-sequence encoding. Its four-argument form also accepts :add-special-tokens? and :with-overflowing-tokens?.

(with-open [t (tok/from-pretrained "bert-base-uncased")]
  (tok/encode t "Question" "Answer"
              {:add-special-tokens? true
               :with-overflowing-tokens? false})
  (tok/batch-encode t ["first" "second"]
                    {:add-special-tokens? false})
  (tok/batch-encode-pairs t [["question 1" "answer 1"]
                             ["question 2" "answer 2"]])
  (tok/batch-decode t [[101 2034 102] [101 2117 102]]
                    {:skip-special-tokens? true}))

Count native batch results without encode maps:

(with-open [t (tok/from-pretrained "bert-base-uncased")]
  (tok/batch-count-tokens t ["hello" "world"]))
;=> [3 3]

Span helpers operate directly on an encode result:

(with-open [t (tok/from-pretrained "bert-base-uncased")]
  (let [enc (tok/encode t "unaffordable cat")]
    [(tok/token->chars enc 2)
     (tok/token->word enc 2)
     (tok/char->token enc 3)
     (tok/word->tokens enc 0)]))
;=> [[3 5] 0 2 [1 2 3 4]]

Batch encode options are the same as encode options. batch-decode accepts :skip-special-tokens?, which defaults to true. Padding set at construction can make batch results rectangular. You can get the real token counts from each :attention-mask.

Requirements

• JDK 21
• A JVM that matches your CPU architecture. DJL loads a native library for the JVM's reported os.arch. On Apple Silicon, use an arm64 JDK. An x86_64 JVM under Rosetta cannot resolve the native tokenizer and fails with Unexpected flavor: cpu. Check the JVM with java -XshowSettings:properties -version 2>&1 | grep 'os.arch\|java.home'.
• Network access the first time DJL gets the native library. DJL then caches it under ~/.djl.ai/. from-pretrained also needs network access to download the model file.

License

Copyright © 2026 Savyasachi

Distributed under the Eclipse Public License 2.0.

Wraps Deep Java Library (Apache-2.0) and the HuggingFace tokenizers library (Apache-2.0).