Idiomatic Clojure wrapper over DJL's HuggingFace tokenizers, which bind the native
Rust tokenizers library via JNI. Build a tokenizer with from-file /
from-pretrained / from-stream, then encode, decode, or count-tokens.
A tokenizer holds a native handle: close it (with-open works) to free it.
Concurrent wrapper operations, including concurrent encode, are not
guaranteed by DJL 0.36.0. Never close a tokenizer while any operation is in
flight; close can free the native handle during JNI execution, causing a
native crash rather than a catchable Clojure or Java exception.
Idiomatic Clojure wrapper over DJL's HuggingFace tokenizers, which bind the native Rust `tokenizers` library via JNI. Build a tokenizer with `from-file` / `from-pretrained` / `from-stream`, then `encode`, `decode`, or `count-tokens`. A tokenizer holds a native handle: close it (`with-open` works) to free it. Concurrent wrapper operations, including concurrent `encode`, are not guaranteed by DJL 0.36.0. Never close a tokenizer while any operation is in flight; `close` can free the native handle during JNI execution, causing a native crash rather than a catchable Clojure or Java exception.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |