Recursive text splitting (chunking) for RAG / LLM pipelines.
split breaks text into overlapping chunks no larger than a target size, trying an
ordered list of separators from coarsest (paragraph) to finest (character) so chunks
land on natural boundaries. Size is measured by :length-fn (default count =
characters); pass a token counter (e.g. tokenizers-clj's count-tokens) to chunk by
tokens instead - the correct unit when feeding a model with a token limit.
Recursive text splitting (chunking) for RAG / LLM pipelines. `split` breaks text into overlapping chunks no larger than a target size, trying an ordered list of separators from coarsest (paragraph) to finest (character) so chunks land on natural boundaries. Size is measured by `:length-fn` (default `count` = characters); pass a token counter (e.g. tokenizers-clj's `count-tokens`) to chunk by tokens instead - the correct unit when feeding a model with a token limit.
Coarsest-to-finest split boundaries (the empty string splits into characters).
Coarsest-to-finest split boundaries (the empty string splits into characters).
Language-aware split boundaries, coarsest first.
Language-aware split boundaries, coarsest first.
(separators-for lang)Return the separator vector for language keyword lang.
Return the separator vector for language keyword `lang`.
(split text)(split text opts)Split text into a vector of chunk strings.
Options:
:chunk-size max size of a chunk, in :length-fn units (default 1000):overlap size of trailing context repeated at the start of the next chunk (default 0):separators ordered split boundaries, coarsest first (default default-separators):language keyword selecting language-separators; conflicts with :separators:keep-separator :start (default), :end, or false; attach separators to the
following or preceding piece, or drop them:length-fn measures a string's size; default count (characters). Pass a token
counter to chunk by tokens.A piece with no admissible finer separator (an "atom" longer than :chunk-size) is
emitted whole rather than dropped.
Split `text` into a vector of chunk strings.
Options:
- `:chunk-size` max size of a chunk, in `:length-fn` units (default 1000)
- `:overlap` size of trailing context repeated at the start of the next chunk (default 0)
- `:separators` ordered split boundaries, coarsest first (default `default-separators`)
- `:language` keyword selecting `language-separators`; conflicts with `:separators`
- `:keep-separator` `:start` (default), `:end`, or `false`; attach separators to the
following or preceding piece, or drop them
- `:length-fn` measures a string's size; default `count` (characters). Pass a token
counter to chunk by tokens.
A piece with no admissible finer separator (an "atom" longer than `:chunk-size`) is
emitted whole rather than dropped.(split-with-offsets text)(split-with-offsets text opts)Split text into maps with :text, :start, and :end offsets.
Options match split. Offsets are character indices into the original text. With
:keep-separator false, a chunk that cannot be located as an exact substring has
nil offsets.
Split `text` into maps with `:text`, `:start`, and `:end` offsets. Options match `split`. Offsets are character indices into the original text. With `:keep-separator false`, a chunk that cannot be located as an exact substring has nil offsets.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |