This namespace splits text recursively into chunks for RAG and LLM pipelines.
split breaks text into overlapping chunks. No chunk is larger than a target size.
split tries an ordered list of separators, from coarsest (paragraph) to finest
(character), so chunks use natural boundaries. :length-fn measures the size. Its
default is count, which measures characters. Pass a token counter, for example
tokenizers-clj's count-tokens, to chunk by tokens. Tokens are the correct unit for
a model with a token limit.
This namespace splits text recursively into chunks for RAG and LLM pipelines. `split` breaks text into overlapping chunks. No chunk is larger than a target size. `split` tries an ordered list of separators, from coarsest (paragraph) to finest (character), so chunks use natural boundaries. `:length-fn` measures the size. Its default is `count`, which measures characters. Pass a token counter, for example tokenizers-clj's `count-tokens`, to chunk by tokens. Tokens are the correct unit for a model with a token limit.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |