chunk-clj

This library splits text recursively into chunks for RAG and LLM pipelines. It splits text on natural boundaries into overlapping chunks by characters or tokens.

Stack

To prepare documents for retrieval, split them into chunks that fit a model's context. Split on suitable boundaries and use a small overlap. With overlap, text that crosses a chunk boundary stays in both chunks. chunk-clj is the Clojure equivalent of LangChain's RecursiveCharacterTextSplitter. It tries a list of separators, from coarsest (paragraph) to finest (character), until each chunk fits. Then it packs the chunks and adds the overlap.

You set the size limit. :length-fn defaults to characters. Models limit you by tokens, so pass a token counter and chunk by a real token budget.

Install

tools.deps (deps.edn):

net.clojars.savya/chunk-clj {:mvn/version "0.3.1"}

Leiningen (project.clj):

[net.clojars.savya/chunk-clj "0.3.1"]

Development

clojure -M:test
clojure -T:build jar
clojure -T:build deploy

Usage

(require '[chunk.core :as chunk])

;; Character-sized chunks with overlap (default):
(chunk/split long-text {:chunk-size 1000 :overlap 200})
;=> ["first ~1000-char chunk ..." "next chunk, sharing ~200 chars ..." ...]

;; Short text is returned whole:
(chunk/split "hello world" {:chunk-size 100})
;=> ["hello world"]

;; Custom separators (e.g. split markdown on headings first):
(chunk/split doc {:chunk-size 800 :separators ["\n## " "\n\n" "\n" " " ""]})

;; Or use a built-in language preset:
(chunk/split doc {:chunk-size 800 :language :markdown})

;; Keep source locations for indexing or highlighting:
(chunk/split-with-offsets doc {:chunk-size 800 :language :markdown})
;=> [{:text "...", :start 0, :end 42} ...]

By default, :keep-separator :start keeps each separator and attaches it to the piece that follows it. This keeps language and markdown content such as def and ##. Use :keep-separator :end to attach a separator to the piece before it. Use :keep-separator false for the separator-dropping behavior from 0.2.2.

Language presets

:language selects a separator preset from chunk.core/language-separators. Chunks then land on structural boundaries: headings, function definitions, and tags. The preset then falls back to paragraphs, lines, words, and characters. The presets are :markdown, :python, :clojure, :javascript, :typescript, :java, :go, :rust, :html, and :latex.

(chunk/split source {:chunk-size 512 :language :clojure})

(chunk/separators-for :python)
;=> ["\nclass " "\ndef " "\n\tdef " "\n\n" "\n" " " ""]

All presets are literal strings, not regexes. Each preset ends with the default paragraph, line, word, and character tail. If you pass both :language and :separators, the function throws. An unknown language keyword also throws, with the known set in ex-data.

Chunk by tokens

Models limit input by tokens, not characters. Use a real tokenizer to size the chunks:

(require '[chunk.core :as chunk]
         '[tokenizers.core :as tok])

(with-open [t (tok/from-pretrained "bert-base-uncased")]
  (chunk/split long-text {:chunk-size 256          ; 256 tokens, not chars
                          :overlap    32
                          :length-fn  #(tok/count-tokens t %)}))

Any String -> number function works as :length-fn. You can target an embedding model's exact token limit.

Options

Key	Default	Meaning
:chunk-size	1000	Maximum chunk size, in :length-fn units
:overlap	0	Trailing context repeated at the start of the next chunk
:separators	["\n\n" "\n" " " ""]	Ordered split boundaries, coarsest first
:keep-separator	:start	Keep separators on the following piece (:end attaches them to the preceding piece; false drops them)
:language	-	Select a built-in language separator preset
:length-fn	count	Measures a string's size (use a token counter)

If an "atom" is longer than :chunk-size and has no finer separator available, the splitter emits it whole. It does not drop it. An example is one very long word when "" is not in :separators.

split-with-offsets has the same options. It returns {:text s :start i :end j} maps, where the offsets index the original input. With :keep-separator false, a chunk that is not an exact source substring has nil offsets. The splitter caches token-mode measurements for each split call. It does not tokenize a joined candidate again after it measures the candidate.

License

Copyright © 2026 Savyasachi

Distributed under the Eclipse Public License 2.0.