Idiomatic Clojure wrapper over DJL's HuggingFace tokenizers, which bind the native
Rust tokenizers library via JNI. Build a tokenizer with from-file /
from-pretrained / from-stream, then encode, decode, or count-tokens.
A tokenizer holds a native handle: close it (with-open works) to free it.
Idiomatic Clojure wrapper over DJL's HuggingFace tokenizers, which bind the native Rust `tokenizers` library via JNI. Build a tokenizer with `from-file` / `from-pretrained` / `from-stream`, then `encode`, `decode`, or `count-tokens`. A tokenizer holds a native handle: close it (`with-open` works) to free it.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |