tech.ml.dataset.text.bag-of-words — techascent/tech.ml.dataset 2.15

parse-token-column clj

(parse-token-column token-table tokenizer col-data)

path->dataset-master-token-table clj

(path->dataset-master-token-table path bag-of-words-colname)

(path->dataset-master-token-table path
                                  bag-of-words-colname
                                  {:keys [tokenizer]
                                   :or {tokenizer (simple-tokenizer-fn)}})

Parse a file returning a map of {:dataset :token-table} where token-table is a map of tokens to counts. Dataset has a sha-256-hash where the original text once was.

path-token-map->bag-of-words clj

(path-token-map->bag-of-words path bag-of-words-colname token->idx-map)

(path-token-map->bag-of-words path
                              bag-of-words-colname
                              token->idx-map
                              {:keys [tokenizer]
                               :or {tokenizer (simple-tokenizer-fn)}})

sha256 clj

(sha256 string)

simple-tokenizer-fn clj

(simple-tokenizer-fn)

sum-bifun clj