tech.ml.dataset.text — techascent/tech.ml.dataset 2.15

data->enc-builder clj

(data->enc-builder data)

default-charset clj

default-tokenizer clj

enc-builder->data clj

(enc-builder->data enc)

encoded-text-builder clj

(encoded-text-builder)

(encoded-text-builder encoding)

Encoding clj

make-tokenized-text-builder clj

(make-tokenized-text-builder)

(make-tokenized-text-builder str-table)

(make-tokenized-text-builder str-table options)

PEncodingToFn clj protocol

encoding->decode-fn clj

(encoding->decode-fn encoding)

encoding->encode-fn clj

(encoding->encode-fn encoding)

string->tokenized-text! clj

(string->tokenized-text! str-data)

(string->tokenized-text! str-table str-data)

(string->tokenized-text! str-table
                         {:keys [token tokenizer]
                          :or {token " " tokenizer default-tokenizer}}
                         str-data)

Mutates the string table adding tokens and records offset and length in a text object.