A full-text index of documents, and queries of it ranked by BM25F, on both platforms.
A document is a map of its :id, its :fields, and what a result gives back, under :stored. A field holds text, or anything whose text counts, such as a number or a set of keywords, but a vector holds terms to take as they are. The text of an instant is its date and time in UTC, e.g. 2024-03-01 09:30:00, and that of anything else is what str gives.
Most of the code behind these functions is in these namespaces under dk.simongray.drop-in-search:
indexing: adding and removing documents queries: the query language terms: the terms that the words of a query stand for matching: the documents that match a query scoring: the BM25F score of a document that matches ranking: the order of the results hits: the documents that match in a segment, and their scores, as arrays like: the terms that set documents apart, for more-like-this
A full-text index of documents, and queries of it ranked by BM25F, on
both platforms.
A document is a map of its :id, its :fields, and what a result gives
back, under :stored. A field holds text, or anything whose text counts,
such as a number or a set of keywords, but a vector holds terms to take
as they are. The text of an instant is its date and time in UTC, e.g.
2024-03-01 09:30:00, and that of anything else is what str gives.
Most of the code behind these functions is in these namespaces under
dk.simongray.drop-in-search:
indexing: adding and removing documents
queries: the query language
terms: the terms that the words of a query stand for
matching: the documents that match a query
scoring: the BM25F score of a document that matches
ranking: the order of the results
hits: the documents that match in a segment, and their scores,
as arrays
like: the terms that set documents apart, for more-like-this(add index)(add index doc & docs)The index with the document doc, and the docs after it, each a map
of :id, :fields and :stored. A document replaces any earlier one with
the same id.
The words are mapped by the :term-fn of the index. An index read back from EDN or CIFF has none until restore or ciff/read-files gives it one, and without it the words of a new document stay as they are, unlike those already in the index.
The `index` with the document `doc`, and the `docs` after it, each a map of :id, :fields and :stored. A document replaces any earlier one with the same id. The words are mapped by the :term-fn of the index. An index read back from EDN or CIFF has none until restore or ciff/read-files gives it one, and without it the words of a new document stay as they are, unlike those already in the index.
The limits that keep the cost of a query that anyone can type within bounds, when its options don't set them: those of search.queries/parse, and these.
The limits that keep the cost of a query that anyone can type within bounds, when its options don't set them: those of search.queries/parse, and these. - :max-completions, the terms that a prefix stands for, those of the most documents, as in Xapian - :max-expansions, the terms that a word with typos stands for, the closest, as in Lucene's FuzzyQuery
(document index id)The document id of index as a map of its :id and :stored, as a
result of query has them, or nil when the index has none of that id.
The document `id` of `index` as a map of its :id and :stored, as a result of query has them, or nil when the index has none of that id.
(ids index)The ids of the documents in index.
The ids of the documents in `index`.
(index)(index docs)(index docs {:keys [term-fn] :as opts})An index of the documents docs, or an empty one, with opts as the
defaults of its queries, e.g. the :boosts of its fields, and its
:term-fn, :without-positions and :max-field-terms.
The :term-fn is a function of each term of a text that gives the term
to index and to look for in its place, e.g. a stemmer's, so that posts
finds post. The other opts print with the index, so they must be data,
but a function can't, so give the :term-fn to restore and
ciff/read-files again.
The :without-positions is a set of fields to index without the positions of their words, e.g. long notes, which then take far less memory. A phrase finds its words anywhere in such a field, as Xapian treats a phrase where it has no positions.
The :max-field-terms is a map of fields to the most terms of each to index, e.g. of long transcripts, as Lucene's LimitTokenCountFilter indexes them: the terms after them are left out, as if the field ended there. A field is indexed up to max-field-terms of its terms by default.
The options of every function are one set, search.queries/option-keys, and an option outside it throws, e.g. :limt for :limit.
An index of the documents `docs`, or an empty one, with `opts` as the defaults of its queries, e.g. the :boosts of its fields, and its :term-fn, :without-positions and :max-field-terms. The :term-fn is a function of each term of a text that gives the term to index and to look for in its place, e.g. a stemmer's, so that posts finds post. The other `opts` print with the index, so they must be data, but a function can't, so give the :term-fn to restore and ciff/read-files again. The :without-positions is a set of fields to index without the positions of their words, e.g. long notes, which then take far less memory. A phrase finds its words anywhere in such a field, as Xapian treats a phrase where it has no positions. The :max-field-terms is a map of fields to the most terms of each to index, e.g. of long transcripts, as Lucene's LimitTokenCountFilter indexes them: the terms after them are left out, as if the field ended there. A field is indexed up to max-field-terms of its terms by default. The options of every function are one set, search.queries/option-keys, and an option outside it throws, e.g. :limt for :limit.
(matches? index text q)(matches? index text q opts)Whether text matches the query q, read as index reads it, with the
opts of search.queries/matches? over the defaults of the index.
Whether `text` matches the query `q`, read as `index` reads it, with the `opts` of search.queries/matches? over the defaults of the index.
The most terms of a field that an index holds, when its :max-field-terms doesn't set fewer for the field.
The most terms of a field that an index holds, when its :max-field-terms doesn't set fewer for the field.
(more-like-this index like)(more-like-this index like opts)The documents of index most like those of like, best first, as
query gives them, with the opts below.
The like is a document, as add takes it, a text, or a collection of
them, e.g. the last few posts that someone read. The query looks for any
of the terms that set them apart, as Lucene's MoreLikeThis does, and
leaves the documents of like out of its results. The metadata of the
results has the :terms, pairs of a term and its weight, the best first:
(meta (more-like-this idx post))
;; => {:total 1 :terms [["clojure" 0.8] ["the" 0.6…]]}
The opts are those of query, and these, over the defaults of the
index and like/defaults:
like among the resultslike, by :unlike-weight, 0.2 by defaultlike or :unlike that gives
how much it weighs, 1 by default, e.g. more for one read recentlyA term's weight in a document is its frequency times its inverse
document frequency, and the weights of each document are scaled to a
length of 1. Those of the documents of like are averaged, less those
of :unlike, as in the Rocchio algorithm. The comments of the like
namespace say where this departs from Lucene and Elasticsearch.
The documents of `index` most like those of `like`, best first, as
query gives them, with the `opts` below.
The `like` is a document, as add takes it, a text, or a collection of
them, e.g. the last few posts that someone read. The query looks for any
of the terms that set them apart, as Lucene's MoreLikeThis does, and
leaves the documents of `like` out of its results. The metadata of the
results has the :terms, pairs of a term and its weight, the best first:
(meta (more-like-this idx post))
;; => {:total 1 :terms [["clojure" 0.8] ["the" 0.6…]]}
The `opts` are those of query, and these, over the defaults of the
index and like/defaults:
- :max-query-terms, the most terms to look for, 25 by default
- :min-term-freq, the fewest times that the documents hold a term, 1 by
default, where a term counts the boost of its field each time
- :min-doc-freq, the fewest documents of the index that hold a term, 2
by default
- :max-doc-freq, the most documents that hold a term, or the share of
them when below 1, e.g. 0.05 for 5%, to leave out common words
- :max-group-freq, the most groups of :group-fn that hold a term, or the
share of them when below 1, 0.4 by default, which leaves out the words
of more than 40% of the authors of a blog, e.g., however many posts
one author writes, and nil for no cap. A share is never fewer than 2
documents or groups, since a term of one links it to nothing.
- :min-word-length and :max-word-length, the fewest and most characters
of a term
- :stop-words, words to leave out
- :boost-terms, a number to boost each term by, times its weight over
the best one's, which is off by default
- :min-match, the fewest of the terms that a result holds, or the share
of them when below 1
- :include?, true to keep the documents of `like` among the results
- :unlike, documents or texts whose terms count against those of
`like`, by :unlike-weight, 0.2 by default
- :weight-fn, a function of a document of `like` or :unlike that gives
how much it weighs, 1 by default, e.g. more for one read recently
A term's weight in a document is its frequency times its inverse
document frequency, and the weights of each document are scaled to a
length of 1. Those of the documents of `like` are averaged, less those
of :unlike, as in the Rocchio algorithm. The comments of the like
namespace say where this departs from Lucene and Elasticsearch.(query index q)(query index q opts)The documents of index that match the query q, best first, as maps
of :id, :score and :stored, with the opts below.
The query is a string, the same query as data, or a vector of terms
taken as they are. The opts are those of search.queries/parse, and
these, over the defaults of the index:
Equal scores are ordered by id, numbers first and ascending, and other ids by hash, so the order is stable on both platforms.
The results have the :total of the documents that match in their metadata, e.g. for "1 to 10 of 243", unless a :filter-pred or :max-per-group had no need to read them all. When words matched others with typos, the metadata also has :typos, a map of each such word to those it matched, the closest first, and so does each result that holds a word only with typos:
(meta (query idx "intervew"))
;; => {:total 2 :typos {"intervew" ["interview" "interviews"]}}
The documents of `index` that match the query `q`, best first, as maps
of :id, :score and :stored, with the `opts` below.
The query is a string, the same query as data, or a vector of terms
taken as they are. The `opts` are those of search.queries/parse, and
these, over the defaults of the index:
- :limit, the most results, and :offset, how many to skip first
- :fuzzy, which words match words with typos too: by default those that
no document holds, true for all, a number for all with at most that
many edits, up to 2, and false for none but those marked with ~
- :typo-lengths, the shortest words with one edit and with two, [3 6]
by default, as Elasticsearch's AUTO
- :boosts, the weight of a match in each field, 1 by default, where 0
leaves a field out unless the query names it
- :fields, the set of fields to search, all by default, which gives the
other fields the weight 0
- :k1 and :b, the saturation and length normalization of BM25F
- :filter-pred, a predicate of a result to keep it
- :rank-fn, a function of a result to order by, ascending, instead of
the score
- :group-fn, a function of a document's id that gives its group, and
:max-per-group, the most results of each group, its best ones, as
Elasticsearch's field collapsing keeps them
- :max-completions and :max-expansions, the most terms that a prefix
and a word with typos stand for
Equal scores are ordered by id, numbers first and ascending, and other
ids by hash, so the order is stable on both platforms.
The results have the :total of the documents that match in their
metadata, e.g. for "1 to 10 of 243", unless a :filter-pred or
:max-per-group had no need to read them all. When words matched others
with typos, the metadata also has :typos, a map of each such word to
those it matched, the closest first, and so does each result that holds
a word only with typos:
(meta (query idx "intervew"))
;; => {:total 2 :typos {"intervew" ["interview" "interviews"]}}(query-opts index)(query-opts index opts)The opts over the defaults of index, to read a query as the index
does: with the names of its fields among the :aliases, its :term-fn,
and a :known-pred that tells the terms its documents hold.
The `opts` over the defaults of `index`, to read a query as the index does: with the names of its fields among the :aliases, its :term-fn, and a :known-pred that tells the terms its documents hold.
(remove index)(remove index id & ids)The index without the document id, and the ids after it.
The `index` without the document `id`, and the `ids` after it.
(restore index)(restore index {:keys [term-fn] :as opts})The index read back from EDN with its arrays made again, which a query
of it otherwise does each time, and the :term-fn of opts that it was
built with, or nil for a nil index.
The `index` read back from EDN with its arrays made again, which a query of it otherwise does each time, and the :term-fn of `opts` that it was built with, or nil for a nil `index`.
(size index)The number of documents in index.
The number of documents in `index`.
(snippet index text q)(snippet index text q opts)A window of text around the words that match the query q, read as
index reads it, with the opts of search.queries/snippet over the
defaults of the index, e.g. :field for the field that the text is from.
(snippet idx (:notes post) "notes:clojure" {:field :notes})
A window of `text` around the words that match the query `q`, read as
`index` reads it, with the `opts` of search.queries/snippet over the
defaults of the index, e.g. :field for the field that the text is from.
(snippet idx (:notes post) "notes:clojure" {:field :notes})cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |