Every id in a collection.
A drain can read the head repeatedly because moved rows leave the source.
A COPY leaves them, so it needs the full id set up front, and Milvus caps a
query at max-page rows, so a single page is not the collection.
Milvus answers a query with a limit with the LOWEST matching primary keys,
in ascending order (measured on hive_mcp_memory_2560d, 2026-10-29; the Java
SDK's QueryIterator relies on the same contract). The walk is therefore
keyset paging: a full page that comes back ascending moves a cursor to its
last id and the next page asks for id > cursor. Every page strictly
advances the cursor, so the walk ends with no depth bound and reads each
row once.
A full page that is NOT ascending means the server does not keep that
contract; that range is then split at the page's median id, both halves
strictly smaller. A cursor that does not advance, or a range still full
after max-depth splits, raises rather than returning a short answer, and
collect-ids checks the walk against the server's count(*).
Every id in a collection. A drain can read the head repeatedly because moved rows leave the source. A COPY leaves them, so it needs the full id set up front, and Milvus caps a query at `max-page` rows, so a single page is not the collection. Milvus answers a query with a limit with the LOWEST matching primary keys, in ascending order (measured on hive_mcp_memory_2560d, 2026-10-29; the Java SDK's QueryIterator relies on the same contract). The walk is therefore keyset paging: a full page that comes back ascending moves a cursor to its last id and the next page asks for `id > cursor`. Every page strictly advances the cursor, so the walk ends with no depth bound and reads each row once. A full page that is NOT ascending means the server does not keep that contract; that range is then split at the page's median id, both halves strictly smaller. A cursor that does not advance, or a range still full after `max-depth` splits, raises rather than returning a short answer, and `collect-ids` checks the walk against the server's count(*).
(all-ids coll)(all-ids coll base)(all-ids coll base opts)Every id in coll matching the optional base filter expression, as a
vector, checked against count(*). Options: :timeout-ms bounds each Milvus
call (default: scaled to the page), :page is the page size, and
:consistency is :strict (default) or :snapshot-of-stable-rows (see
collect-ids). The walk's report is the result's metadata. Raises rather
than truncating.
Every id in `coll` matching the optional `base` filter expression, as a vector, checked against count(*). Options: `:timeout-ms` bounds each Milvus call (default: scaled to the page), `:page` is the page size, and `:consistency` is :strict (default) or :snapshot-of-stable-rows (see `collect-ids`). The walk's report is the result's metadata. Raises rather than truncating.
(all-ids-with query count-q coll base {:keys [timeout-ms page consistency]})collect-ids over QUERY ((fn [filter-expr limit] future-of-rows)) and
COUNT ((fn [filter-expr timeout-ms] n)). Each page is dereferenced within
:timeout-ms when given, else page-deadline-ms of its size; a count
within :timeout-ms, else budget/read-deadline-cap-ms. :consistency
and :page go to collect-ids. A failure carries :coll, and so does
the report in the result's metadata.
`collect-ids` over QUERY (`(fn [filter-expr limit] future-of-rows)`) and COUNT (`(fn [filter-expr timeout-ms] n)`). Each page is dereferenced within `:timeout-ms` when given, else `page-deadline-ms` of its size; a count within `:timeout-ms`, else `budget/read-deadline-cap-ms`. `:consistency` and `:page` go to `collect-ids`. A failure carries `:coll`, and so does the report in the result's metadata.
(collect-ids fetch
count-fn
base
{:keys [page consistency] :or {consistency :strict}})Every id matching base, read through FETCH ((fn [filter-expr limit] rows)) and checked against COUNT-FN ((fn [filter-expr] n), the server's
count(*)). Returns a vector of distinct ids, in walk order. Its metadata is
the walk's report: :consistency, :complete?, :read, :count-before and
:count-after, plus :count-mid, :tolerance and :shortfall in the stable
mode. Options: :page as in rows-matching, and :consistency (one of
consistency-modes, default :strict).
:strict takes a count before and after the walk. If the count did not change, the walk must have read exactly that many rows. If it moved (live writes), the rows read must lie between the two counts. Anything else raises :enumerate/count-mismatch. :complete? is true.
:snapshot-of-stable-rows takes counts before, mid-walk (at row before/2) and after the walk. It accepts the walk only when BOTH hold:
stable-tolerance: the churn the three counts show, plus 0.1% of
min(before, after), capped at 1% of it. Otherwise it raises
:enumerate/stable-rows-shortfall.
Rule 2 is a gap detector, not a proof. Counts show only net churn, so an
insert behind the cursor and a delete ahead of it cancel out. That is why
the tolerance has a floor. A page that drops more stable rows than the
tolerance is refused, and a smaller drop cannot be told apart from churn.
The report carries every count, the tolerance and the shortfall, and
:complete? is :stable-rows (never true), so the caller sees what was
accepted.Every id matching `base`, read through FETCH (`(fn [filter-expr limit]
rows)`) and checked against COUNT-FN (`(fn [filter-expr] n)`, the server's
count(*)). Returns a vector of distinct ids, in walk order. Its metadata is
the walk's report: :consistency, :complete?, :read, :count-before and
:count-after, plus :count-mid, :tolerance and :shortfall in the stable
mode. Options: `:page` as in `rows-matching`, and `:consistency` (one of
`consistency-modes`, default :strict).
:strict takes a count before and after the walk. If the count did not
change, the walk must have read exactly that many rows. If it moved
(live writes), the rows read must lie between the two counts. Anything
else raises :enumerate/count-mismatch. :complete? is true.
:snapshot-of-stable-rows takes counts before, mid-walk (at row
before/2) and after the walk. It accepts the walk only when BOTH hold:
1. Every id is strictly greater than the one before it. Otherwise it
raises :enumerate/duplicate-id or :enumerate/not-ascending. This is
the keyset proof. Each page is the lowest ids above the cursor, so a
row present for the whole walk lies above some cursor and below the
next page's last id, and is returned. Primary keys are immutable, so
no row moves behind the cursor. A server that answers in another
order (the median-split fallback) gives no such proof and is
refused.
2. read >= min(before, after) - tolerance, where tolerance is
`stable-tolerance`: the churn the three counts show, plus 0.1% of
min(before, after), capped at 1% of it. Otherwise it raises
:enumerate/stable-rows-shortfall.
Rule 2 is a gap detector, not a proof. Counts show only net churn, so an
insert behind the cursor and a delete ahead of it cancel out. That is why
the tolerance has a floor. A page that drops more stable rows than the
tolerance is refused, and a smaller drop cannot be told apart from churn.
The report carries every count, the tolerance and the shortfall, and
:complete? is :stable-rows (never true), so the caller sees what was
accepted.What collect-ids holds a walk to.
:strict (default): the walk must account for the server's count(*). With an unchanged count it read exactly that many rows; with a count that moved during the walk (live writes) the rows read lie between the two counts.
:snapshot-of-stable-rows: the guarantee asked for is that every row present
for the WHOLE walk is returned. Rows inserted or deleted during the walk may
or may not appear. See collect-ids for the acceptance rule.
What `collect-ids` holds a walk to. :strict (default): the walk must account for the server's count(*). With an unchanged count it read exactly that many rows; with a count that moved during the walk (live writes) the rows read lie between the two counts. :snapshot-of-stable-rows: the guarantee asked for is that every row present for the WHOLE walk is returned. Rows inserted or deleted during the walk may or may not appear. See `collect-ids` for the acceptance rule.
Floor of the default deref bound for one enumeration call. A full id page measured ~2 s on an idle hive_mcp_memory_2560d; a walk of ~45 such pages runs while the store serves traffic, so the floor is the read cap, not the 5 s interactive default.
Floor of the default deref bound for one enumeration call. A full id page measured ~2 s on an idle hive_mcp_memory_2560d; a walk of ~45 such pages runs while the store serves traffic, so the floor is the read cap, not the 5 s interactive default.
(page-deadline-ms rows)Default deref bound for an id-only page of rows rows: the base deadline
plus a per-row allowance (a 16384-row strong read measured ~2 s), never
below enumerate-timeout-ms.
Default deref bound for an id-only page of `rows` rows: the base deadline plus a per-row allowance (a 16384-row strong read measured ~2 s), never below `enumerate-timeout-ms`.
(rows-matching fetch filter-expr limit)(rows-matching fetch filter-expr limit {:keys [page]})Up to limit rows satisfying filter-expr, read through FETCH
((fn [filter-expr limit] rows)), which is never asked for more than
:page rows (default and ceiling max-page). Within a page this is one
fetch; above it the rows are walked in ascending id order, stopping once
limit are held. Raises rather than truncating.
Up to `limit` rows satisfying `filter-expr`, read through FETCH (`(fn [filter-expr limit] rows)`), which is never asked for more than `:page` rows (default and ceiling `max-page`). Within a page this is one fetch; above it the rows are walked in ascending id order, stopping once `limit` are held. Raises rather than truncating.
(stable-tolerance before mid after)Rows the walk may fall short of min(before, after) under
:snapshot-of-stable-rows: the churn the three counts show, |mid - before| +
|after - mid|, plus ceil(stable-tolerance-floor x min). The whole is
capped at ceil(stable-tolerance-cap x min).
Rows the walk may fall short of min(`before`, `after`) under :snapshot-of-stable-rows: the churn the three counts show, |mid - before| + |after - mid|, plus ceil(`stable-tolerance-floor` x min). The whole is capped at ceil(`stable-tolerance-cap` x min).
Under :snapshot-of-stable-rows, the tolerance never exceeds this fraction of min(count-before, count-after), however much churn was observed. A walk short by more than that is refused.
Under :snapshot-of-stable-rows, the tolerance never exceeds this fraction of min(count-before, count-after), however much churn was observed. A walk short by more than that is refused.
Under :snapshot-of-stable-rows, the part of the shortfall tolerance that does not depend on churn: this fraction of min(count-before, count-after). It covers churn that the counts cannot see, such as an insert behind the cursor offset by a delete ahead of it.
Under :snapshot-of-stable-rows, the part of the shortfall tolerance that does not depend on churn: this fraction of min(count-before, count-after). It covers churn that the counts cannot see, such as an insert behind the cursor offset by a delete ahead of it.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |