This namespace splits text recursively into chunks for RAG and LLM pipelines.
split breaks text into overlapping chunks. No chunk is larger than a target size.
split tries an ordered list of separators, from coarsest (paragraph) to finest
(character), so chunks use natural boundaries. :length-fn measures the size. Its
default is count, which measures characters. Pass a token counter, for example
tokenizers-clj's count-tokens, to chunk by tokens. Tokens are the correct unit for
a model with a token limit.
This namespace splits text recursively into chunks for RAG and LLM pipelines. `split` breaks text into overlapping chunks. No chunk is larger than a target size. `split` tries an ordered list of separators, from coarsest (paragraph) to finest (character), so chunks use natural boundaries. `:length-fn` measures the size. Its default is `count`, which measures characters. Pass a token counter, for example tokenizers-clj's `count-tokens`, to chunk by tokens. Tokens are the correct unit for a model with a token limit.
(chunk-document document)(chunk-document {:keys [id text metadata]} opts)Split a document into chunks carrying its id, metadata, and source offsets.
The document must be a map with :id, :text, and :metadata keys. Options
match split, including :chunk-size, :overlap, :separators, :language,
:keep-separator, and :length-fn. Each returned map has :id, :index,
:start, :end, :text, and :metadata keys. Offsets are character indices
into the document's original :text value.
With :keep-separator false, offsets may be nil when a chunk is not an exact
source substring, matching split-with-offsets.
Split a document into chunks carrying its id, metadata, and source offsets. The document must be a map with `:id`, `:text`, and `:metadata` keys. Options match `split`, including `:chunk-size`, `:overlap`, `:separators`, `:language`, `:keep-separator`, and `:length-fn`. Each returned map has `:id`, `:index`, `:start`, `:end`, `:text`, and `:metadata` keys. Offsets are character indices into the document's original `:text` value. With `:keep-separator false`, offsets may be nil when a chunk is not an exact source substring, matching `split-with-offsets`.
Split boundaries from coarsest to finest. The empty string splits into characters.
Split boundaries from coarsest to finest. The empty string splits into characters.
Split boundaries for each language, with the coarsest first.
Split boundaries for each language, with the coarsest first.
(separators-for lang)Return the separator vector for language keyword lang.
Return the separator vector for language keyword `lang`.
(split text)(split text opts)Split text into a vector of chunk strings.
Options:
:chunk-size max size of a chunk, in :length-fn units (default 1000):overlap size of trailing context repeated at the start of the next chunk (default 0):separators ordered split boundaries, coarsest first (default default-separators):language keyword selecting language-separators; conflicts with :separators:keep-separator :start (default), :end, or false; attach separators to the
following or preceding piece, or drop them:length-fn measures a string's size; default count (characters). Pass a token
counter to chunk by tokens.A piece with no admissible finer separator is emitted whole. This includes an "atom"
longer than :chunk-size.
Split `text` into a vector of chunk strings.
Options:
- `:chunk-size` max size of a chunk, in `:length-fn` units (default 1000)
- `:overlap` size of trailing context repeated at the start of the next chunk (default 0)
- `:separators` ordered split boundaries, coarsest first (default `default-separators`)
- `:language` keyword selecting `language-separators`; conflicts with `:separators`
- `:keep-separator` `:start` (default), `:end`, or `false`; attach separators to the
following or preceding piece, or drop them
- `:length-fn` measures a string's size; default `count` (characters). Pass a token
counter to chunk by tokens.
A piece with no admissible finer separator is emitted whole. This includes an "atom"
longer than `:chunk-size`.(split-with-offsets text)(split-with-offsets text opts)Split text into maps with :text, :start, and :end offsets.
Options match split. Offsets are character indices in the original text. With
:keep-separator false, a chunk that is not an exact substring has nil offsets.
Split `text` into maps with `:text`, `:start`, and `:end` offsets. Options match `split`. Offsets are character indices in the original text. With `:keep-separator false`, a chunk that is not an exact substring has nil offsets.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |