peco.core — peco 0.1.6

alpha-numeric-string? clj

(alpha-numeric-string? s)

Matches a string that is a combination of digits and letters. Useful for matching many part numbers and strings like 3am etc.

concat-ngrams clj

(concat-ngrams n pred)

(concat-ngrams n pred coll)

Repeatedley takes a sequence of length n from terms and passes it to pred. If pred returns a truthy value, the sequence is concatenated. Useful for normalising n-grams e.g. 'user name' -> 'username'. Returns a transducer if no collection of tokens is supplied.

concat-singles clj

(concat-singles)

(concat-singles coll)

Concatenates consecutive single characters. Useful for normalising names etc. For example, after tokenization, [A D B Widgets] | [ADB Widgets] -> [ADB Widgets]. Returns a transducer if no collection of tokens is provided.

dimension-string? clj

(dimension-string? s)

Matches a string that looks like a dimension e.g. 3cm. The list of unit suffixes can defintely be improved upon.

lancaster-stem clj

(lancaster-stem word)

lovins-stem clj

(lovins-stem word)

numeric-string? clj

(numeric-string? s)

operation-table clj

porter-stem clj

(porter-stem word)

slurp-from-classpath clj

(slurp-from-classpath path)

Slurps a file from the classpath. Lifted from https://github.com/krisajenkins/yesql

slurp-words clj

(slurp-words file-path)

(slurp-words file-path tokenizer)

token-pipeline clj

(token-pipeline operation-table operations)

tokenize clj

(tokenize text)

tokenizer clj

(tokenizer)

(tokenizer operations)

(tokenizer operations & ud-operations)

tokenizer-fn clj

(tokenizer-fn operation-table operations)