tech.ml.dataset.text — techascent/tech.ml.dataset 2.13

default-charset clj

default-tokenizer clj

encode-text clj

(encode-text)

(encode-text item)

(encode-text data encoding)

encoded-text-builder clj

(encoded-text-builder)

(encoded-text-builder n-elems)

(encoded-text-builder charset n-elems)

make-tokenized-text-builder clj

(make-tokenized-text-builder)

(make-tokenized-text-builder str-table)

(make-tokenized-text-builder str-table options)

string->tokenized-text! clj

(string->tokenized-text! str-data)

(string->tokenized-text! str-table str-data)

(string->tokenized-text! str-table
                         {:keys [token tokenizer]
                          :or {token " " tokenizer default-tokenizer}}
                         str-data)

Mutates the string table adding tokens and records offset and length in a text object.