chunk-clj

Recursive text splitting (chunking) for RAG and LLM pipelines - split text on natural boundaries into overlapping chunks sized by characters or tokens.

Stack

The standard way to prepare documents for retrieval is to split them into chunks that fit a model's context, on sensible boundaries, with a little overlap so a thought split across two chunks survives in both. chunk-clj is the Clojure equivalent of LangChain's RecursiveCharacterTextSplitter: it tries a list of separators from coarsest (paragraph) to finest (character) until each chunk fits, then packs and overlaps them.

The size limit is whatever you want it to be - :length-fn defaults to characters, but since models limit you by tokens, pass a token counter and chunk by real token budgets.

Install

Leiningen / Boot:

[net.clojars.savya/chunk-clj "0.2.0"]

deps.edn:

net.clojars.savya/chunk-clj {:mvn/version "0.2.0"}

Usage

(require '[chunk.core :as chunk])

;; Character-sized chunks with overlap (default):
(chunk/split long-text {:chunk-size 1000 :overlap 200})
;=> ["first ~1000-char chunk ..." "next chunk, sharing ~200 chars ..." ...]

;; Short text is returned whole:
(chunk/split "hello world" {:chunk-size 100})
;=> ["hello world"]

;; Custom separators (e.g. split markdown on headings first):
(chunk/split doc {:chunk-size 800 :separators ["\n## " "\n\n" "\n" " " ""]})

;; Or use a built-in language preset:
(chunk/split doc {:chunk-size 800 :language :markdown})

Language presets

:language selects a separator preset from chunk.core/language-separators so chunks land on structural boundaries (headings, function definitions, tags) before falling back to paragraphs, lines, words, and characters. Available: :markdown, :python, :clojure, :javascript, :typescript, :java, :go, :rust, :html, :latex.

(chunk/split source {:chunk-size 512 :language :clojure})

(chunk/separators-for :python)
;=> ["\nclass " "\ndef " "\n\tdef " "\n\n" "\n" " " ""]

All presets are literal strings (no regexes) ending in the default paragraph/line/word/character tail. Passing both :language and :separators throws; an unknown language keyword throws with the known set in ex-data.

Chunk by tokens

Models cap input by tokens, not characters, so size chunks with a real tokenizer:

(require '[chunk.core :as chunk]
         '[tokenizers.core :as tok])

(with-open [t (tok/from-pretrained "bert-base-uncased")]
  (chunk/split long-text {:chunk-size 256          ; 256 tokens, not chars
                          :overlap    32
                          :length-fn  #(tok/count-tokens t %)}))

Any String -> number function works as :length-fn, so you can target an embedding model's exact token limit.

Options

Key	Default	Meaning
:chunk-size	1000	Max chunk size, in :length-fn units
:overlap	0	Trailing context repeated at the start of the next chunk
:separators	["\n\n" "\n" " " ""]	Ordered split boundaries, coarsest first
:length-fn	count	Measures a string's size (swap in a token counter)

An "atom" longer than :chunk-size with no admissible finer separator (e.g. one huge word when "" is not in :separators) is emitted whole rather than dropped.

License

Copyright © 2026 Savyasachi

Distributed under the Eclipse Public License 2.0.