bosquet.nlp.splitter — io.github.zmedelis/bosquet 2025.03.28

character clj

Text splitter by individual characters.

chunk-size clj

Since of the chunks in which the text gets split.

chunk-text clj

(chunk-text {:splitter/keys [unit] :as opts} text)

Chunk text into chunk-size blocks using specified splitter. Optionaly overlap can be specified by how many text units chunks can overap (defaults to 0).

Supported text splitters:

• sentence-splitter
• character-splitter
• token-splitter

model clj

Model used for tokenization.

overlap clj

Number of units by which chunks can overlap.

sentence clj

Text splitter by sentences. It will use OpenNLP sentnce splitter to partition the text.

split-handlers clj

Split handlers are needed to turn text into specified text units via encode function. decode function will turn those units back into single text string.

split-unit clj

Lexical units in which the text gets split: character, token, sentence.

text-splitter clj

(text-splitter {:splitter/keys [chunk-size overlap] :or {overlap 0}} text-units)

token clj

Text splitter by tokens. Tokenization is done based on the provided model.