Core entry CRUD, query, search, expiry, and status helpers.
Core entry CRUD, query, search, expiry, and status helpers.
(add-entries! config-atom entries)IMemoryStoreBatchWrite: write ENTRIES grouped by routed collection. Per collection: one ensure-collection!, then per chunk of :batch-records (default 256) one batch embed and one milvus upsert, under the resilient circuit/retry wrapper. Returns one add-entry! result per entry in input order; a failing record or chunk never loses the others' results.
IMemoryStoreBatchWrite: write ENTRIES grouped by routed collection. Per collection: one ensure-collection!, then per chunk of :batch-records (default 256) one batch embed and one milvus upsert, under the resilient circuit/retry wrapper. Returns one add-entry! result per entry in input order; a failing record or chunk never loses the others' results.
Write an entry under the write time budget; refuse oversize before embedding.
Write an entry under the write time budget; refuse oversize before embedding.
What callers may rely on beyond the port. :embed-text: an entry's transient :embed-text is embedded in place of its :content and never stored.
What callers may rely on beyond the port. :embed-text: an entry's transient :embed-text is embedded in place of its :content and never stored.
(collection-outcome fetch coll-name filter-expr limit)One collection's FanOutOutcome for a scalar query: up to limit rows matching
filter-expr, read through FETCH ((fn [filter-expr page-limit] rows)).
A limit above Milvus's per-query cap is served in pages that each stay
within it (enumerate/rows-matching), so a large limit returns rows, never
a failure dressed as an empty result. A collection that cannot be read
yields :rows [] with its :error.
One collection's FanOutOutcome for a scalar query: up to `limit` rows matching `filter-expr`, read through FETCH (`(fn [filter-expr page-limit] rows)`). A `limit` above Milvus's per-query cap is served in pages that each stay within it (`enumerate/rows-matching`), so a large limit returns rows, never a failure dressed as an empty result. A collection that cannot be read yields `:rows []` with its `:error`.
(count-entries config-atom {:keys [content-hash-prefix] :as opts})IMemoryStoreCount: the exact number of entries matching opts (the
query-entries filter keys; :limit :order-by :output-fields are ignored),
summed over the known collections with a server-side count(*). Raises when
a collection cannot answer; a failure is never 0.
IMemoryStoreCount: the exact number of entries matching `opts` (the query-entries filter keys; :limit :order-by :output-fields are ignored), summed over the known collections with a server-side count(*). Raises when a collection cannot answer; a failure is never 0.
Records per milvus upsert in add-entries! unless the store config
carries :batch-records.
Records per milvus upsert in `add-entries!` unless the store config carries :batch-records.
(delete-entry! config-atom id)Delete id from every known collection. true when every collection took
the delete (one that was never created has nothing to take). A collection
that failed means the delete did not land, so it raises inside
resilient (deleted-value): a transient cause is retried once and then
answered as the hive-spi Failure map ({:success? false ...}), a fatal
cause propagates as the ex-info. Never a true for a lost delete.
Delete `id` from every known collection. true when every collection took
the delete (one that was never created has nothing to take). A collection
that failed means the delete did not land, so it raises inside
`resilient` (`deleted-value`): a transient cause is retried once and then
answered as the hive-spi Failure map (`{:success? false ...}`), a fatal
cause propagates as the ex-info. Never a `true` for a lost delete.(delete-everywhere delete colls id)Delete id from every one of colls through DELETE ((fn [coll id])).
r/ok true every collection took the delete (an unknown id included;
a collection that was never created holds nothing to
delete, see lookup/missing-collection?)
r/err :milvus/write-incomplete {:id :failed [WriteFailure ...] :cause e}
some collection did not, so the entry may survive there
Every collection is attempted even after a failure. Never throws.
Delete `id` from every one of `colls` through DELETE (`(fn [coll id])`).
r/ok true every collection took the delete (an unknown id included;
a collection that was never created holds nothing to
delete, see `lookup/missing-collection?`)
r/err :milvus/write-incomplete {:id :failed [WriteFailure ...] :cause e}
some collection did not, so the entry may survive there
Every collection is attempted even after a failure. Never throws.(deleted-value res)The value a delete-everywhere result answers: true, or a raised ex-info
when some collection did not take the delete. Like located-value, the
ex-info wraps the first failing collection's exception as its cause (so
resilient classifies it) and carries :hive-milvus/failed-collections.
The value a `delete-everywhere` result answers: true, or a raised ex-info when some collection did not take the delete. Like `located-value`, the ex-info wraps the first failing collection's exception as its cause (so `resilient` classifies it) and carries `:hive-milvus/failed-collections`.
(fan-out outcomes)Fold per-collection OUTCOMES into gathered rows and the failures that make an empty result unreliable.
Pure, and the whole of the isolation policy: a failing collection contributes
whatever rows it managed and is recorded in :failed, rather than sinking
the query or vanishing.
Fold per-collection OUTCOMES into gathered rows and the failures that make an empty result unreliable. Pure, and the whole of the isolation policy: a failing collection contributes whatever rows it managed and is recorded in `:failed`, rather than sinking the query or vanishing.
The rows a fan-out gathered, and the collections whose silence is unexplained
by the data. :failed empty means every collection answered.
The rows a fan-out gathered, and the collections whose silence is unexplained by the data. `:failed` empty means every collection answered.
What one collection contributed to a fan-out: its rows, and — when it could not be reached — the message saying so. Rows and an error are not exclusive; a partial read may carry both.
What one collection contributed to a fan-out: its rows, and — when it could not be reached — the message saying so. Rows and an error are not exclusive; a partial read may carry both.
(get-entry config-atom id)The entry id, or nil when every known collection answered and none
holds it (a configured collection that was never created answers 'none').
A collection that cannot be read is NOT absence: when no collection holds
id and one failed, the read raises inside resilient (see
located-value). A transient cause is retried once, then answered as
{:success? false :errors [..] :reconnecting? true}; a fatal cause
propagates as the ex-info. Either way, never nil.
The entry `id`, or nil when every known collection answered and none
holds it (a configured collection that was never created answers 'none').
A collection that cannot be read is NOT absence: when no collection holds
`id` and one failed, the read raises inside `resilient` (see
`located-value`). A transient cause is retried once, then answered as
`{:success? false :errors [..] :reconnecting? true}`; a fatal cause
propagates as the ex-info. Either way, never nil.(locate-entry fetch colls id)Ask each of colls, in order, for id through FETCH
((fn [coll id] value-or-nil)), stopping at the first value.
r/ok value some collection answered with it
r/ok nil EVERY collection answered and none holds id - absence.
A configured collection that was never created counts as
answered (lookup/missing-collection?): it holds nothing.
r/err :milvus/read-incomplete {:id :failed [ReadFailure ...] :cause e}
no value was found AND at least one collection failed, so
absence cannot be told from an outage
A hit wins over a failing neighbour. Never throws: the decision is data,
and located-value is the boundary that raises it.
Ask each of `colls`, in order, for `id` through FETCH
(`(fn [coll id] value-or-nil)`), stopping at the first value.
r/ok value some collection answered with it
r/ok nil EVERY collection answered and none holds `id` - absence.
A configured collection that was never created counts as
answered (`lookup/missing-collection?`): it holds nothing.
r/err :milvus/read-incomplete {:id :failed [ReadFailure ...] :cause e}
no value was found AND at least one collection failed, so
absence cannot be told from an outage
A hit wins over a failing neighbour. Never throws: the decision is data,
and `located-value` is the boundary that raises it.(located-value res)The value a locate-entry result answers to a caller: the entry, nil for
a true absence, or a raised ex-info for an incomplete read - never nil for
a failure. ex-data carries :hive-milvus/failed-collections, the key
query-entries uses.
The ex-info wraps the first failing collection's exception as its cause,
so resilient classifies it by that cause: a TRANSIENT cause is retried
once and, if still failing, answered as the legacy
{:success? false ...} map; a FATAL cause is re-thrown by resilient
(retry/classify-err), so the caller sees this ex-info.
The value a `locate-entry` result answers to a caller: the entry, nil for
a true absence, or a raised ex-info for an incomplete read - never nil for
a failure. ex-data carries `:hive-milvus/failed-collections`, the key
`query-entries` uses.
The ex-info wraps the first failing collection's exception as its cause,
so `resilient` classifies it by that cause: a TRANSIENT cause is retried
once and, if still failing, answered as the legacy
`{:success? false ...}` map; a FATAL cause is re-thrown by `resilient`
(`retry/classify-err`), so the caller sees this ex-info.(page-step fetch colls base limit keep? cursor)One step of a page walk over colls, through FETCH
((fn [coll filter-expr limit] rows)), for a cursor
{:coll i :after id-or-nil} (nil starts at the first collection).
Rows are walked per collection in ascending id order, keyset-style
(id > after), so a page never holds more than limit rows and no offset
window caps the walk. The cursor moves past the last id the page READ, not
the last it kept: :keep? may drop rows (an exact prefix test) while the
walk still advances. A page shorter than limit closes its collection and
moves to the next. Returns {:rows [raw-row ..] :next-cursor cursor-or-nil},
:next-cursor nil exactly when the last collection is exhausted.
Raises when a fetched page is not in ascending id order - the keyset walk would silently skip rows - and when the cursor is malformed.
One step of a page walk over `colls`, through FETCH
(`(fn [coll filter-expr limit] rows)`), for a cursor
`{:coll i :after id-or-nil}` (nil starts at the first collection).
Rows are walked per collection in ascending id order, keyset-style
(`id > after`), so a page never holds more than `limit` rows and no offset
window caps the walk. The cursor moves past the last id the page READ, not
the last it kept: `:keep?` may drop rows (an exact prefix test) while the
walk still advances. A page shorter than `limit` closes its collection and
moves to the next. Returns `{:rows [raw-row ..] :next-cursor cursor-or-nil}`,
`:next-cursor` nil exactly when the last collection is exhausted.
Raises when a fetched page is not in ascending id order - the keyset walk
would silently skip rows - and when the cursor is malformed.(query-entries config-atom opts)Scalar fan-out. The deadline of each page scales with the rows it asks for.
A collection that cannot answer (timeout included) makes the whole result an
error Result (read-outcome), never a match vector.
Scalar fan-out. The deadline of each page scales with the rows it asks for. A collection that cannot answer (timeout included) makes the whole result an error Result (`read-outcome`), never a match vector.
(query-page config-atom
{:keys [limit output-fields cursor content-hash-prefix]
:or {limit 100}
:as opts})IMemoryStorePage: one page of the entries matching opts (the query-entries
filter keys plus :content-hash-prefix), {:rows :next-cursor}. :limit is the
page size (default 100, at most enumerate/max-page), :output-fields the
projection, :cursor nil to start or the previous :next-cursor. Never an empty
page for a failure: a collection that cannot answer raises.
IMemoryStorePage: one page of the entries matching `opts` (the query-entries
filter keys plus :content-hash-prefix), `{:rows :next-cursor}`. :limit is the
page size (default 100, at most `enumerate/max-page`), :output-fields the
projection, :cursor nil to start or the previous :next-cursor. Never an empty
page for a failure: a collection that cannot answer raises.(read-outcome rows failed)Successful reads retain their vector API. An incomplete read is an error Result with partial rows as diagnostic data, never a match vector.
Successful reads retain their vector API. An incomplete read is an error Result with partial rows as diagnostic data, never a match vector.
One collection that could not answer a read.
One collection that could not answer a read.
(relocate-entry! config-atom id)(relocate-entry! config-atom id opts)Move entry id from its current collection to the canonical target.
Implements IMemoryStoreWithRouting/relocate-entry!, and with opts
IMemoryStoreRoutingEmbedText/relocate-entry-with!: (:embed-text opts)
is embedded in place of the stored content, and never stored.
Delegates to hive-milvus.relocate.pipeline/relocate-one, which is
the CPPB-layered (Collect -> Promote -> Boundary) implementation.
This wrapper unwraps the pipeline's r/ok / r/err result back into
the legacy raw-map shape callers expect:
{:moved? true :from src :to target :id id} on move {:moved? false :from src :to target :id id} on no-op {:moved? false :from nil :to nil :id id :reason :not-found} when the id resolves to no collection {:moved? false :error <category> :id id :detail err-data} when the pipeline returns r/err for any other reason
Migration path: callers that want railway-tracked errors should
call reloc-pipeline/relocate-one directly instead of this
facade; they get an r/ok / r/err with full error context.
Move entry `id` from its current collection to the canonical target.
Implements `IMemoryStoreWithRouting/relocate-entry!`, and with `opts`
`IMemoryStoreRoutingEmbedText/relocate-entry-with!`: `(:embed-text opts)`
is embedded in place of the stored content, and never stored.
Delegates to `hive-milvus.relocate.pipeline/relocate-one`, which is
the CPPB-layered (Collect -> Promote -> Boundary) implementation.
This wrapper unwraps the pipeline's r/ok / r/err result back into
the legacy raw-map shape callers expect:
{:moved? true :from src :to target :id id} on move
{:moved? false :from src :to target :id id} on no-op
{:moved? false :from nil :to nil :id id :reason :not-found}
when the id resolves to no collection
{:moved? false :error <category> :id id :detail err-data}
when the pipeline returns r/err for any other reason
Migration path: callers that want railway-tracked errors should
call `reloc-pipeline/relocate-one` directly instead of this
facade; they get an r/ok / r/err with full error context.(scan-deadline-ms rows fields)Deref bound for a scalar read of rows rows projecting fields: the base
deadline plus an allowance per row, dearer when content is among them.
Deref bound for a scalar read of `rows` rows projecting `fields`: the base deadline plus an allowance per row, dearer when `content` is among them.
(scan-ids config-atom {:keys [include-expired? timeout-ms consistency]})Every entry id across the collections this store reads, each once; expired
ones only with :include-expired?. :timeout-ms bounds each Milvus call of the
walk (default: scaled to the page). :consistency is :strict (default) or
:snapshot-of-stable-rows, the walk check of
hive-milvus.relocate.enumerate/collect-ids: the latter guarantees every
row present for the whole walk and refuses a shortfall that churn does not
explain. Raises rather than truncating. The result's metadata is
{:consistency :complete? :walks}, where :walks holds one report per
collection (counts before/after, and the tolerance in the stable mode),
and :complete? is true only when every walk was strict.
Every entry id across the collections this store reads, each once; expired
ones only with :include-expired?. :timeout-ms bounds each Milvus call of the
walk (default: scaled to the page). :consistency is :strict (default) or
:snapshot-of-stable-rows, the walk check of
`hive-milvus.relocate.enumerate/collect-ids`: the latter guarantees every
row present for the whole walk and refuses a shortfall that churn does not
explain. Raises rather than truncating. The result's metadata is
{:consistency :complete? :walks}, where :walks holds one report per
collection (counts before/after, and the tolerance in the stable mode),
and :complete? is true only when every walk was strict.(search-context config-atom)The live collaborators a semantic search runs against.
The live collaborators a semantic search runs against.
(search-similar config-atom query-text opts)Semantic search. Returns entries, best first. A target that failed (a
timeout included) makes the result an error Result (see search-with),
never an empty match vector. :output-fields narrows the columns read.
Semantic search. Returns entries, best first. A target that failed (a timeout included) makes the result an error Result (see `search-with`), never an empty match vector. `:output-fields` narrows the columns read.
(search-with ctx query-text opts)Search entries, best first. Any failed target returns an error Result with diagnostic partial rows, never absence disguised as metadata.
Search entries, best first. Any failed target returns an error Result with diagnostic partial rows, never absence disguised as metadata.
(target-collection-for _config-atom entry)Resolve the canonical Milvus collection for entry per current
routing config (per-type → per-dim). Returns the collection name
string. Implements IMemoryStoreWithRouting/target-collection-for.
Resolve the canonical Milvus collection for `entry` per current routing config (per-type → per-dim). Returns the collection name string. Implements `IMemoryStoreWithRouting/target-collection-for`.
(update-entry! config-atom id updates)Update an entry's fields. Routing-aware via the CPPB-layered pipeline — when the merged entry's target collection differs from its current collection, the pipeline relocates it transparently.
Delegates to hive-milvus.relocate.pipeline/relocate-update and answers
its result through update-result: the merged entry (carrying id) on
success, nil when id is unknown, or the update-failure map
({:error .. :reconnecting? transient?}) for downstream errors - each a
value of hive-spi's UpdateResult contract.
Migration path: callers that want railway-tracked errors should
call reloc-pipeline/relocate-update directly instead of this
facade.
Update an entry's fields. Routing-aware via the CPPB-layered
pipeline — when the merged entry's target collection differs from
its current collection, the pipeline relocates it transparently.
Delegates to `hive-milvus.relocate.pipeline/relocate-update` and answers
its result through `update-result`: the merged entry (carrying `id`) on
success, nil when `id` is unknown, or the `update-failure` map
(`{:error .. :reconnecting? transient?}`) for downstream errors - each a
value of hive-spi's UpdateResult contract.
Migration path: callers that want railway-tracked errors should
call `reloc-pipeline/relocate-update` directly instead of this
facade.(update-failure id res)The legacy error map update-entry! answers for a relocate-pipeline
r/err res on id:
{:error category :id id :detail (dissoc res :error) :reconnecting? bool}
:reconnecting? says whether a retry can change the answer, the same
flag resilient's failure map carries, so a queue drain re-queues only
transient failures. Only a :boundary/* effect error (a Milvus call that
threw) or a :milvus/read-incomplete locate (some collection could not
be read, see lookup/locate-collection) can be transient, and only when
a message it carries is a transport drop or timeout (failure/classify).
Routing, embedding and collector errors are permanent. Pure.
The legacy error map `update-entry!` answers for a relocate-pipeline
r/err `res` on `id`:
{:error category :id id :detail (dissoc res :error) :reconnecting? bool}
`:reconnecting?` says whether a retry can change the answer, the same
flag `resilient`'s failure map carries, so a queue drain re-queues only
transient failures. Only a `:boundary/*` effect error (a Milvus call that
threw) or a `:milvus/read-incomplete` locate (some collection could not
be read, see `lookup/locate-collection`) can be transient, and only when
a message it carries is a transport drop or timeout (`failure/classify`).
Routing, embedding and collector errors are permanent. Pure.(update-fields-keep-embedding! config-atom id updates)Update entry fields without re-embedding.
Reads the existing record (including its :embedding vector via
query-scalar), merges updates, and upserts in place via
entry->record-pure with the retrieved vector. Suitable for
metadata-only changes (e.g. :kg-incoming back-edge bookkeeping)
where re-running the embedder on unchanged content is wasted work
— and on 4096d Venice that waste blows past the 30 s memory-write
timeout when an add fans out updates to multiple KG targets.
Returns the merged entry on success, nil when every known collection
answered and none holds id. A collection that failed is not absence
(see locate-entry / located-value): a transient cause reaches the
caller as the legacy failure map from resilient, a fatal cause as the
raised ex-info. Never a nil that reads as 'unknown id'.
Update entry fields without re-embedding. Reads the existing record (including its :embedding vector via query-scalar), merges `updates`, and upserts in place via `entry->record-pure` with the retrieved vector. Suitable for metadata-only changes (e.g. :kg-incoming back-edge bookkeeping) where re-running the embedder on unchanged content is wasted work — and on 4096d Venice that waste blows past the 30 s memory-write timeout when an add fans out updates to multiple KG targets. Returns the merged entry on success, nil when every known collection answered and none holds `id`. A collection that failed is not absence (see `locate-entry` / `located-value`): a transient cause reaches the caller as the legacy failure map from `resilient`, a fatal cause as the raised ex-info. Never a nil that reads as 'unknown id'.
(update-result id res)The hive-spi write-contract value (hive-spi.memory.contract/UpdateResult)
for a relocate-update result res on id. Pure:
r/ok merged -> the merged entry, carrying id as its :id
(an :id inside the updates cannot rename it)
r/err :collector/not-found -> nil, the id is absent
any other r/err -> the update-failure map (:error non-nil)
The hive-spi write-contract value (`hive-spi.memory.contract/UpdateResult`)
for a `relocate-update` result `res` on `id`. Pure:
r/ok merged -> the merged entry, carrying `id` as its :id
(an :id inside the updates cannot rename it)
r/err :collector/not-found -> nil, the id is absent
any other r/err -> the `update-failure` map (:error non-nil)One collection that did not take a write (delete).
One collection that did not take a write (delete).
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |