scicloj.metamorph.ml.text — org.scicloj/metamorph.ml 1.7.0

Large-scale text processing and TF-IDF feature engineering for NLP pipelines.

This namespace provides efficient tools for converting raw text documents into machine learning-ready features using TF-IDF (Term Frequency-Inverse Document Frequency) scoring. Designed to handle large text corpora with flexible memory management strategies.

Core Functions:

->tidy-text Parses text files or datasets into tidy-text format (one token per row). Line-by-line processing enables handling of files larger than available RAM. Supports custom tokenization and metadata extraction.

Output format: tech.v3.dataset with columns:

• :document (int): Document/line identifier
• :token-idx (int): Token as indexed integer (maps to lookup table)
• :token-pos (int): Position of token within document
• :meta (optional): Arbitrary metadata from line-split-fn

->tfidf Transforms tidy-text into TF-IDF vector representation for bag-of-words models. Calculates term frequency (TF) and inverse document frequency (IDF) for each token.

Output columns:

• :document
• :token-idx
• :token-count
• :tf
• :tfid

Memory Optimization:

The namespace provides flexible memory control for large texts via options:

Container Types:

• :jvm-heap (default): Java heap storage (fast, limited by heap)
• :native-heap: Off-heap native memory via tech.v3
• :mmap: Memory-mapped files (disk-backed, bypasses heap limits)

Processing Options:

• container-type: Storage for intermediate results during processing
• column-container-type: Storage for final output dataset
• combine-method: :coalesce-blocks! or :concat-buffers (tradeoffs)
• compacting-document-interval: Batch size for consolidating data
• datatype-document/token-pos/idx: Memory datatype selection (:int16 vs :int32)

Token Management:

• token->index-map: Custom token lookup table (can reuse across runs)
• new-token-behaviour: :store (default), :fail, or :as-unknown

Performance Characteristics:

• Typical text requires ~1.5x the original file size in RAM
• A 8GB text file typically needs ≥12GB total memory
• Scaling strategy: Use off-heap or mmap for large corpora

Typical Workflow:

1. Use ->tidy-text to create tidy text format from raw documents
2. Use ->tfidf to create TF-IDF feature vectors
3. Pass vectors to classification/regression models

See also: scicloj.metamorph.ml.column-metric for evaluation, scicloj.metamorph.ml/train for model training

->tfidf clj

(->tfidf tidy-text
         &
         {:keys [container-type column-container-type combine-method
                 datatype-meta]
          :or {combine-method :coalesce-blocks!
               column-container-type :jvm-heap
               container-type :jvm-heap
               datatype-meta :object}})

Transforms a dataset in tidy text format in the bag-of-words representation including TFIDF calculation of the the tokens.

tidy-text needs to be a dataset with columns

• :document
• :token-idx
• :token-pos

The following three can be used to move data off heap during calculations. They can make dramatic differences in performance (faster and slower) and memory usage.

• container-type decides if the intermidiate results are stored on-heap (:jvm-heap, the default) or off-heap (:native-heap) or :mmap (as mmaped file)
• column-container-type same decides if the resulting dataset os store on-hep (:jvm-heap, the default) or off-heap (:native-heap) or :mmap (as mmaped file)
• combine-method How to combine the intermidiate containers, either :concat-bufders or :coalesce-buffers!

Returns a dataset with columns:

• :document document id
• :token-idx The token as id
• :token-count How often the token appears in a 'document'
• :tf :token-count divided by document length
• :tfidf tfidf value for token

->tidy-text clj

(->tidy-text lines-source
             line-seq-fn
             line-split-fn
             line-tokenizer-fn
             &
             {:keys [skip-lines max-lines container-type datatype-document
                     datatype-token-pos datatype-meta datatype-token-idx
                     compacting-document-intervall combine-method
                     token->index-map column-container-type new-token-behaviour]
              :or {datatype-token-idx :int16
                   max-lines Integer/MAX_VALUE
                   datatype-document :int16
                   container-type :jvm-heap
                   datatype-meta :object
                   datatype-token-pos :int16
                   compacting-document-intervall 10000
                   skip-lines 0
                   column-container-type :jvm-heap
                   combine-method :coalesce-blocks!
                   new-token-behaviour :store
                   token->index-map (Object2IntOpenHashMap. 10000)}})

Reads, parses and tokenizes a text file or a TMD dataset into a seq of tech.v3.dataset in the tidy-text format, so one word per row. It does the parsing and conversion strictly line based, so it should work for large documents.

Initial tests show that each byte of text size need 1.5 byte on average So a 8 GB text file can be sucessfully loaded when having at least 12 GB.

• lines-source Either a buffered reader or a TMD dadaset

• line-seq-fn A function which return a lazy-list of lines , given the lines-source

• line-split-fn A fn which should seperate a single line of input in text and other Supposed to return a seq of size 2, where the first is the 'text' of the line and meta can be anything non-nil (map, vector, scalar). It's value will be returned in column meta and is supposed to be further processed later. meta can be nil always, so no column meta will be created

• text-tokenizer-fn A function which will be called for any text as obtained by line-split-fn It should split the text by word boundaries and return the obtained tokens as a seq of strings. It can do any text normalisation desired.

Optional options are:

• skip-lines 0 Lines to skip at beginning
• max-lines MAX_INT max lines to return

The following can be used to optimize the heap usage for larger texts. It can be tune depending on how may documents, how many words per document, and how many tokens overall are in the text corpus.

• datatype-document :int16 Datatype of :document column (:int16 or :int32)
• datatype-token-pos :int16 Datatype of :token-pos column (:int16 or :int32)
• datatype-meta :object Datatype of :meta column (anything, need to match what line-split-fn returns as 'meta')
• datatype-token-idx :int16 Datatype of :token-idx column (:int16 or :int32)

The following options can be used to move data off heap during calculations. They can make dramatic differences in performance (faster and slower) and memory usage.

• column-container-type :jvm-heap If the resulting table is created on heap (:jvm-heap ) of off heap (:native-heap)
• container-type :jvm-heap as column-container-type but for intermidiate reuslts, per interval compacting-document-intervall 10000 After how many lines the data is written into a continous block
• combine-method :coalesce-blocks! Which method to use to combine blocks (:coalesce-blocks! or :concat-buffers) One or the other might need less RAM in ceratin scenarious.
• token->index-map Object2IntOpenHashMap Can be overriden with a own object->int map implementation, (maybe off-heap). Can as well be a map obtained from a prevoius run in order to guranty same mappings.
• new-token-behaviour :store How to react when new tokens appear , which are no in token->id-map Either :store (default), :fail (throw exception) or :as-unknown (use specific token [UNKNOWN])

The following three can be used to move data off heap during calculations. They can make dramatic differences in performance (faster and slower) and memory usage.

• container-type decides if the intermidiate results are stored on-heap (:jvm-heap, the default) or off-heap (:native-heap) or :mmap (as mmaped file)
• column-container-type same decides if the resulting dataset os store on-hep (:jvm-heap, the default) or off-heap (:native-heap) or :mmap (as mmaped file)
• combine-method How to combine the intermidiate containers, either :concat-bufders or :coalesce-buffers!

Function returns a map of :datasets and :token-lookup-table

:datasets is a seq of TMD datasets each having 4 columns which represent the input text in the tidy-text format:

• :document The 'document/line' a token is comming from
• :token-idx The token/word (as int) , which is present as well in the token->int look up table returned
• :token-pos The position of the token in the document
• :meta The meta values if return by line-split-fn

Assuming that the text-tokenizer-fn does no text normalisation, the table is a exact representation of the input text. I contains as well the word order in column :token-pos, so resorting the table keeps the original text.

libsvm->tidy clj

(libsvm->tidy reader)

Reads LIBSVM format data into a tidy dataset.

reader - Reader (typically from a file) containing LIBSVM formatted text

Returns a dataset with columns:

• :instance - Document/instance ID (0-indexed)
• :label - Class label from the LIBSVM file
• :index - Feature index
• :value - Feature value

Each line in LIBSVM format is parsed: <label> <index>:<value> ...

The reader is automatically closed after reading.

See also: tidy->libsvm!, ->tidy-text

tidy->libsvm! clj

(tidy->libsvm! tidy-ds writer column)

Writes a tidy dataset to LIBSVM text format.

• tidy-ds - Dataset with TF-IDF features (usually from ->tidy or ->tfidf) including columns:token-idx :token-pos :document
• writer - BufferedWriter for output
• column - Column name containing feature values (e.g., :meta or :tfidf)

Writes the dataset in LIBSVM sparse format: <label> <index>:<value> <index>:<value> ... where label comes from the column column. Groups by :document and writes one line per :document with tokens sorted by token-idx.

The writer is automatically closed after writing.

See also: [->tfidf], [libsvm->tidy], [->tidy]