Batch processing: embed entries + upsert to Milvus + persist cursor.
Batch processing: embed entries + upsert to Milvus + persist cursor.
Circuit breaker for Milvus protocol operations.
States: :closed (normal) | :open (fail-fast) | :half-open (probing).
Flow:
check returns {:success? false :error :circuit-open
:retry-after ms :reconnecting? true} until cooldown-ms has elapsed
since opened-at.check transitions to :half-open and allows
the call through as a probe. record-success! closes the breaker
and notifies subscribers; record-failure! reopens it (resets
opened-at=now).Subscribers: callers can subscribe! a 0-arity fn invoked on every
:closed transition (open->closed or half-open->closed). Used by the
queue drain module to flush buffered writes when Milvus is healthy.
Thread-safety: single atom, swap! transitions; listener fns run on the caller's thread that closed the breaker (keep them fast or spawn futures inside).
Time discipline: routes wall-clock through hive-ttracking.clock/now-millis
so tests can pin time deterministically — never call
System/currentTimeMillis directly.
Circuit breaker for Milvus protocol operations.
States: :closed (normal) | :open (fail-fast) | :half-open (probing).
Flow:
- :closed - calls pass through. N consecutive connection failures
(threshold) -> :open with opened-at=now.
- :open - `check` returns {:success? false :error :circuit-open
:retry-after ms :reconnecting? true} until cooldown-ms has elapsed
since opened-at.
- cooldown elapsed - next `check` transitions to :half-open and allows
the call through as a probe. `record-success!` closes the breaker
and notifies subscribers; `record-failure!` reopens it (resets
opened-at=now).
Subscribers: callers can `subscribe!` a 0-arity fn invoked on every
:closed transition (open->closed or half-open->closed). Used by the
queue drain module to flush buffered writes when Milvus is healthy.
Thread-safety: single atom, swap! transitions; listener fns run on
the caller's thread that closed the breaker (keep them fast or spawn
futures inside).
Time discipline: routes wall-clock through `hive-ttracking.clock/now-millis`
so tests can pin time deterministically — never call
`System/currentTimeMillis` directly.Typed config for the collection bounded context, resolved via
hive-di defconfig. Reads [:milvus :collections] block from
~/.config/hive-mcp/config.edn.
Two fields:
:sunset — set of Milvus collection names that are RETIRED.
Locator returns them in known-collections (so reads
still fan out to legacy data) but excludes from
active-collections (so writes never land there).:base-collection-name — Chroma form of the legacy 768-d
collection. Default "hive-mcp-memory" mirrors
hive-milvus.collection.naming/legacy-base-collection.Typed config for the collection bounded context, resolved via
hive-di `defconfig`. Reads `[:milvus :collections]` block from
`~/.config/hive-mcp/config.edn`.
Two fields:
- `:sunset` — set of Milvus collection names that are RETIRED.
Locator returns them in `known-collections` (so reads
still fan out to legacy data) but excludes from
`active-collections` (so writes never land there).
- `:base-collection-name` — Chroma form of the legacy 768-d
collection. Default `"hive-mcp-memory"` mirrors
`hive-milvus.collection.naming/legacy-base-collection`.L3 — ICollectionEnsure impl. Idempotent collection creation with
dim-check.
Wraps the imperative hive-milvus.store.index/ensure-collection!
in the Result railway and adds an in-process dim-registry that
rejects re-ensures requesting a different dimension for the same
collection name, surfacing a dim mismatch at ensure! time rather
than at the first failing upsert.
L3 — `ICollectionEnsure` impl. Idempotent collection creation with dim-check. Wraps the imperative `hive-milvus.store.index/ensure-collection!` in the Result railway and adds an in-process dim-registry that rejects re-ensures requesting a different dimension for the same collection name, surfacing a dim mismatch at `ensure!` time rather than at the first failing upsert.
THE DIMENSION INVARIANT: the width of the vector the embedder produces for a collection MUST equal the width that collection holds.
When it does not, nothing crashes. Milvus rejects an insert with error 1804, which a write path can swallow; and a SEARCH with a wrong-width query vector either errors or — the dangerous case, when another model happens to share the width — returns confident neighbours from a space the query was never embedded into. That is noise wearing the costume of an answer, and it is the shape the 2026-07-12 outage wore.
hive-milvus.store.search.boundary/CollectionEmbedder and
hive-milvus.embedder/embed-for-entry already REFUSE to serve on a mismatch
(:embedder/dim-mismatch) — read path and write path. That is the per-call
half of the invariant, and it is the half that keeps a wrong vector out of
the index.
This namespace is the OTHER half: assert it ONCE, at startup, so a misconfiguration is discovered when the server boots rather than on the first query of the day, silently, per call, forever.
Pure core / effectful shell:
check — pure, one collection.
violations — pure, a set of already-read (collection, expected, actual).
verify — the boundary: reads live embedder dims, calls the pure core.
UNKNOWN IS NOT OK. naming/dim-of returns nil for a collection whose width
is not knowable from its name (anything that is not a memory collection).
Those are reported under :unknown — never folded into :ok. A checker
that reports 'fine' when it means 'I could not tell' is the exact failure
this file exists to prevent.
THE DIMENSION INVARIANT: the width of the vector the embedder produces for a collection MUST equal the width that collection holds. When it does not, nothing crashes. Milvus rejects an insert with error 1804, which a write path can swallow; and a SEARCH with a wrong-width query vector either errors or — the dangerous case, when another model happens to share the width — returns confident neighbours from a space the query was never embedded into. That is noise wearing the costume of an answer, and it is the shape the 2026-07-12 outage wore. `hive-milvus.store.search.boundary/CollectionEmbedder` and `hive-milvus.embedder/embed-for-entry` already REFUSE to serve on a mismatch (`:embedder/dim-mismatch`) — read path and write path. That is the per-call half of the invariant, and it is the half that keeps a wrong vector out of the index. This namespace is the OTHER half: assert it ONCE, at startup, so a misconfiguration is discovered when the server boots rather than on the first query of the day, silently, per call, forever. Pure core / effectful shell: `check` — pure, one collection. `violations` — pure, a set of already-read (collection, expected, actual). `verify` — the boundary: reads live embedder dims, calls the pure core. UNKNOWN IS NOT OK. `naming/dim-of` returns nil for a collection whose width is not knowable from its name (anything that is not a memory collection). Those are reported under `:unknown` — never folded into `:ok`. A checker that reports 'fine' when it means 'I could not tell' is the exact failure this file exists to prevent.
L2 — default ICollectionLocator impl.
Thin wrapper composing the pure naming layer (naming/ref-for-dim)
with the sunset-aware config (config/sunset?). Like the router,
reads config lazily on each call so hot-flips propagate.
The locator is the single source of truth for the
spec→CollectionRef mapping. The write pipeline must NEVER
construct a CollectionRef from a raw type or string — only from a
resolved ProviderSpec via this protocol. That invariant is what
eliminates the bypass paths that fed the 1804 bug.
L2 — default `ICollectionLocator` impl. Thin wrapper composing the pure naming layer (`naming/ref-for-dim`) with the sunset-aware config (`config/sunset?`). Like the router, reads config lazily on each call so hot-flips propagate. The locator is the single source of truth for the spec→`CollectionRef` mapping. The write pipeline must NEVER construct a CollectionRef from a raw type or string — only from a resolved `ProviderSpec` via this protocol. That invariant is what eliminates the bypass paths that fed the 1804 bug.
L1 pure — collection-name derivation.
Two pure functions:
chroma-name : dim → Chroma collection name.
768 stays on the legacy hive-mcp-memory; other
dims get the -<dim>d suffix.milvus-name : Chroma name → Milvus name (underscores not
hyphens). Idempotent — a name already in Milvus
form passes through. Reuses
hive-milvus.collections/collection->milvus-name
so the two definitions cannot drift.Why this lives outside routing.clj: SRP. Routing is about
resolving a ProviderSpec; naming is about projecting a dim to a
string. The L2 locator composes both.
L1 pure — collection-name derivation.
Two pure functions:
- `chroma-name` : dim → Chroma collection name.
768 stays on the legacy `hive-mcp-memory`; other
dims get the `-<dim>d` suffix.
- `milvus-name` : Chroma name → Milvus name (underscores not
hyphens). Idempotent — a name already in Milvus
form passes through. Reuses
`hive-milvus.collections/collection->milvus-name`
so the two definitions cannot drift.
Why this lives outside `routing.clj`: SRP. Routing is about
resolving a `ProviderSpec`; naming is about projecting a dim to a
string. The L2 locator composes both.L0 contract — collection bounded context.
Two narrow protocols (ISP):
ICollectionLocator — spec → CollectionRef. Pure lookup. Knows
sunset state so writes can avoid retired
collections while reads can fan out across
both active + sunset (graceful drain).ICollectionEnsure — create-if-missing on the backend with
mandatory dim-check. Rejects an ensure!
whose ref-dim disagrees with an existing
collection's dim — the precise check that
prevents the silent 1804 schema-mismatch.Reload-safety: defonce-guarded.
L0 contract — collection bounded context.
Two narrow protocols (ISP):
- `ICollectionLocator` — spec → `CollectionRef`. Pure lookup. Knows
sunset state so writes can avoid retired
collections while reads can fan out across
both active + sunset (graceful drain).
- `ICollectionEnsure` — create-if-missing on the backend with
mandatory dim-check. Rejects an `ensure!`
whose ref-dim disagrees with an existing
collection's dim — the precise check that
prevents the silent 1804 schema-mismatch.
Reload-safety: `defonce`-guarded.Multi-collection helpers: filtering, naming.
Multi-collection helpers: filtering, naming.
Typed config for Milvus addon — declared via hive-di defconfig.
Replaces ad-hoc env-var parsing previously scattered through addon.clj. Generates (via the macro):
MilvusConfig — ADT (:config/resolved | :unresolved | :invalid) MilvusConfig-fields — field registry (source of truth) MilvusConfig-schema — Malli closed map schema resolve-MilvusConfig — resolver fn (0/1/2 arity) returning Result
Resolution order (per field):
Optional fields (currently :token) resolve to nil when unset.
Typed config for Milvus addon — declared via hive-di defconfig.
Replaces ad-hoc env-var parsing previously scattered through addon.clj.
Generates (via the macro):
MilvusConfig — ADT (:config/resolved | :unresolved | :invalid)
MilvusConfig-fields — field registry (source of truth)
MilvusConfig-schema — Malli closed map schema
resolve-MilvusConfig — resolver fn (0/1/2 arity) returning Result
Resolution order (per field):
1. Explicit overrides map (e.g. addon manifest config)
2. Environment variable lookup (System/getenv by default)
3. blank->nil normalization ("" → trigger default)
4. Pre-typed default (skips coercion)
5. hive-dsl.coerce on string env values
Optional fields (currently :token) resolve to nil when unset.Persistent resume state for batch migrations.
Persistent resume state for batch migrations.
Detect embedding dimension for Chroma collections.
Detect embedding dimension for Chroma collections.
Consumer-owned embedding port + injection slot.
IEmbedder is the boundary hive-milvus calls for every provider
lookup, routing decision and remote embed. The host injects a
concrete adapter via set-embedder! at boot; a no-op default keeps
the core loadable and testable standalone (no hive-mcp on the path).
Protocol methods are --prefixed and dispatch on an injected
instance; the public fns below deref the slot so call sites stay
free of (current) threading. Method contracts:
-embed-entry entry+collection+content => (r/ok vec) | (r/err :embedder/no-provider {…}) | (r/err :embedder/embed-failed {…}) -embed-text collection+text => vec (may throw) -routing-for-type[+size] memory-type[+content] => {:collection-name str :dimension int :max-tokens int :provider-key kw} | nil -no-embed-type? memory-type => bool -collection-names => seq of chroma-style names -dimension-for-collection collection => int | nil -configured-collection-names => seq of chroma-style names -provider-available-for? collection => bool -collection-backed? collection => bool
Consumer-owned embedding port + injection slot.
`IEmbedder` is the boundary hive-milvus calls for every provider
lookup, routing decision and remote embed. The host injects a
concrete adapter via `set-embedder!` at boot; a no-op default keeps
the core loadable and testable standalone (no hive-mcp on the path).
Protocol methods are `-`-prefixed and dispatch on an injected
instance; the public fns below deref the slot so call sites stay
free of `(current)` threading. Method contracts:
-embed-entry entry+collection+content => (r/ok vec) |
(r/err :embedder/no-provider {…}) |
(r/err :embedder/embed-failed {…})
-embed-text collection+text => vec (may throw)
-routing-for-type[+size] memory-type[+content] =>
{:collection-name str :dimension int
:max-tokens int :provider-key kw} | nil
-no-embed-type? memory-type => bool
-collection-names => seq of chroma-style names
-dimension-for-collection collection => int | nil
-configured-collection-names => seq of chroma-style names
-provider-available-for? collection => bool
-collection-backed? collection => boolBOUNDARY: re-embed an entry's content via the injected embedding port.
Record-shape conversion stays pure in hive-milvus.store.schema; the
provider lookup + remote embed live behind hive-milvus.embed.port.
embed-for-entry policy (port-agnostic):
blank content => (r/ok [])
no-embed type => (r/ok zero-vector) sized to the type's routed dim
otherwise => port/embed-entry, then dim-guarded.
Returns (r/ok vec) or one of :embedder/no-provider :embedder/embed-failed :embedder/dim-mismatch.
BOUNDARY: re-embed an entry's content via the injected embedding port. Record-shape conversion stays pure in `hive-milvus.store.schema`; the provider lookup + remote embed live behind `hive-milvus.embed.port`. `embed-for-entry` policy (port-agnostic): blank content => (r/ok []) no-embed type => (r/ok zero-vector) sized to the type's routed dim otherwise => `port/embed-entry`, then dim-guarded. Returns (r/ok vec) or one of :embedder/no-provider :embedder/embed-failed :embedder/dim-mismatch.
Failure ADT + classifier + legacy-shape translator for hive-milvus.
Pure, no side effects: classifies Throwables into the closed
MilvusFailure ADT and translates ADT values back to the legacy
{:success? false :errors [...] :reconnecting? ...} map shape that
protocol callers expect. Classification delegates to
milvus-clj.client/classify-error (dispatch on the
::client/transport ex-data tag) and walks the full .getCause
chain, not just the top throwable.
Failure ADT + classifier + legacy-shape translator for hive-milvus.
Pure, no side effects: classifies Throwables into the closed
`MilvusFailure` ADT and translates ADT values back to the legacy
`{:success? false :errors [...] :reconnecting? ...}` map shape that
protocol callers expect. Classification delegates to
`milvus-clj.client/classify-error` (dispatch on the
`::client/transport` ex-data tag) and walks the full `.getCause`
chain, not just the top throwable.Chroma SQLite → Milvus migration.
Railway-oriented pipeline: resolve-config → read-source → resolve-cursor → connect-target → migrate-batches!
Each step returns Result (ok/err). Cursor persists after each batch for safe interrupt/resume.
Chroma SQLite → Milvus migration. Railway-oriented pipeline: resolve-config → read-source → resolve-cursor → connect-target → migrate-batches! Each step returns Result (ok/err). Cursor persists after each batch for safe interrupt/resume.
Global in-memory write queue + read fail-soft shim for Milvus outages.
Single JVM-wide singleton — shared across ALL projects and agents, not
per-project. Purpose: when the circuit breaker opens because Milvus is
unreachable, mutating protocol calls are enqueued here instead of
failing; reads return a degraded response carrying actionable recovery
tips. On the circuit's :closed transition, a background future drains
the queue via hive-weave.parallel/bounded-pmap with concurrency 8
and a 15 s per-item timeout. Pre-drain the batch is coalesced by
(op,id) keeping the latest mutation, so a burst of writes to the same
entry collapses to one RPC. Any operation that still fails is pushed
to the tail for the next pass; drain repeats until the queue is empty,
the circuit reopens, or a pass makes zero progress.
Intentionally has no hard dependency on the circuit ns or store.clj —
those wire it in (see enqueue!, drain!, degraded-response,
make-store-dispatch). Keeping the boundary clean lets circuit,
store, and queue evolve independently and lets tests exercise the
queue without standing up Milvus.
Global in-memory write queue + read fail-soft shim for Milvus outages. Single JVM-wide singleton — shared across ALL projects and agents, not per-project. Purpose: when the circuit breaker opens because Milvus is unreachable, mutating protocol calls are enqueued here instead of failing; reads return a degraded response carrying actionable recovery tips. On the circuit's :closed transition, a background future drains the queue via `hive-weave.parallel/bounded-pmap` with concurrency 8 and a 15 s per-item timeout. Pre-drain the batch is coalesced by (op,id) keeping the latest mutation, so a burst of writes to the same entry collapses to one RPC. Any operation that still fails is pushed to the tail for the next pass; drain repeats until the queue is empty, the circuit reopens, or a pass makes zero progress. Intentionally has no hard dependency on the circuit ns or store.clj — those wire it in (see `enqueue!`, `drain!`, `degraded-response`, `make-store-dispatch`). Keeping the boundary clean lets circuit, store, and queue evolve independently and lets tests exercise the queue without standing up Milvus.
Pure: memory entry → Milvus insert record.
Pure: memory entry → Milvus insert record.
Background re-embedding + relocation of memory entries from
non-canonical Milvus collections (e.g. legacy 768-d
hive_mcp_memory) into per-dim collections (hive_mcp_memory_1024d,
hive_mcp_memory_4096d) driven by current type-routing config.
Idempotent + resumable. entries/relocate-entry! is itself a no-op
once an entry is already in its canonical collection, so resume
never duplicates.
Background re-embedding + relocation of memory entries from non-canonical Milvus collections (e.g. legacy 768-d `hive_mcp_memory`) into per-dim collections (`hive_mcp_memory_1024d`, `hive_mcp_memory_4096d`) driven by current type-routing config. Idempotent + resumable. `entries/relocate-entry!` is itself a no-op once an entry is already in its canonical collection, so resume never duplicates.
BOUNDARY: thin Result-tracked wrappers around Milvus side effects
used by the relocation pipeline. Each fn does ONE Milvus call,
wrapped in r/try-effect* so failures stay railway-tracked
instead of escaping as exceptions.
These replace the bare @(milvus/...) + bare try/catch
patterns in hive-milvus.store.entries for the routing-aware
path. Pre-existing entries.clj fns are not modified here (out of
scope per the refactor plan).
BOUNDARY: thin Result-tracked wrappers around Milvus side effects used by the relocation pipeline. Each fn does ONE Milvus call, wrapped in `r/try-effect*` so failures stay railway-tracked instead of escaping as exceptions. These replace the bare `@(milvus/...)` + bare `try`/`catch` patterns in `hive-milvus.store.entries` for the routing-aware path. Pre-existing entries.clj fns are not modified here (out of scope per the refactor plan).
COLLECT: locate an entry's current physical collection and read its
record. Wraps the existing lookup/find-entry-collection +
query/get-entry-by-id chain in Result envelopes so the pipeline
sees them as data.
Imports the leaf hive-milvus.store.lookup ns rather than
hive-milvus.store.entries to keep the relocate pipeline cycle-free
(entries delegates routing-aware ops to the pipeline; the pipeline
must NOT loop back into entries).
COLLECT: locate an entry's current physical collection and read its record. Wraps the existing `lookup/find-entry-collection` + `query/get-entry-by-id` chain in Result envelopes so the pipeline sees them as data. Imports the leaf `hive-milvus.store.lookup` ns rather than `hive-milvus.store.entries` to keep the relocate pipeline cycle-free (entries delegates routing-aware ops to the pipeline; the pipeline must NOT loop back into entries).
Every id in a collection.
A drain can read the head repeatedly because moved rows leave the source.
A COPY leaves them, so it needs the full id set up front — and Milvus caps a
query at max-page rows and returns them in no particular order, so a single
page is not the collection.
The id space is therefore split into disjoint prefix buckets, each fetched whole. A bucket that comes back full may have been truncated, so it is split again. Nothing is ever assumed about ordering, and a bucket that cannot be split further raises rather than returning a short answer.
Every id in a collection. A drain can read the head repeatedly because moved rows leave the source. A COPY leaves them, so it needs the full id set up front — and Milvus caps a query at `max-page` rows and returns them in no particular order, so a single page is not the collection. The id space is therefore split into disjoint prefix buckets, each fetched whole. A bucket that comes back full may have been truncated, so it is split again. Nothing is ever assumed about ordering, and a bucket that cannot be split further raises rather than returning a short answer.
PIPELINE: compose the collectors + promoters + boundaries into the single-entry relocate operation. All-Result; first :err short-circuits.
The flow:
collect-existing-entry ;; COLLECT → compute-target ;; PROMOTE (pure) → classify-relocation-need ;; PROMOTE (pure) → IF :no-op → return r/ok with :moved? false IF :move → ensure-target-collection ;; PROMOTE/BOUNDARY edge → build-target-record ;; PROMOTE (calls embedder) → milvus-write! ;; BOUNDARY → milvus-delete! ;; BOUNDARY → return r/ok with :moved? true
Replaces the body of hive-milvus.store.entries/relocate-entry!
with a Result-tracked, layer-disciplined version.
PIPELINE: compose the collectors + promoters + boundaries into the
single-entry relocate operation. All-Result; first :err short-circuits.
The flow:
collect-existing-entry ;; COLLECT
→ compute-target ;; PROMOTE (pure)
→ classify-relocation-need ;; PROMOTE (pure)
→ IF :no-op → return r/ok with :moved? false
IF :move → ensure-target-collection ;; PROMOTE/BOUNDARY edge
→ build-target-record ;; PROMOTE (calls embedder)
→ milvus-write! ;; BOUNDARY
→ milvus-delete! ;; BOUNDARY
→ return r/ok with :moved? true
Replaces the body of `hive-milvus.store.entries/relocate-entry!`
with a Result-tracked, layer-disciplined version.Pure decisions for a drain pass: which rows to ask for next, and what a finished round means for the loop. Plain data in, plain data out — no client, no clock, no state.
Pure decisions for a drain pass: which rows to ask for next, and what a finished round means for the loop. Plain data in, plain data out — no client, no clock, no state.
PROMOTE: wrap the embedder BOUNDARY call + the pure schema record builder into a Result-tracked promoter. The single caller (the pipeline) gets one r/ok / r/err to thread through.
Why this is a promoter even though it touches the embedder:
embed-for-entry returns Result; from the pipeline's POV the
embedding is just data flowing through the bundle. The actual
I/O (HTTP call to Ollama) is encapsulated in the embedder ns;
here we only orchestrate.
PROMOTE: wrap the embedder BOUNDARY call + the pure schema record builder into a Result-tracked promoter. The single caller (the pipeline) gets one r/ok / r/err to thread through. Why this is a promoter even though it touches the embedder: `embed-for-entry` returns Result; from the pipeline's POV the embedding is just data flowing through the bundle. The actual I/O (HTTP call to Ollama) is encapsulated in the embedder ns; here we only orchestrate.
PROMOTE: pure functions that compute relocation routing decisions from an entry. No I/O, no embedder calls, no Milvus reads — purely data → Result.
Used by hive-milvus.relocate.pipeline/relocate-one to decide
whether an entry needs to move and where it should go before any
BOUNDARY effect fires.
PROMOTE: pure functions that compute relocation routing decisions from an entry. No I/O, no embedder calls, no Milvus reads — purely data → Result. Used by `hive-milvus.relocate.pipeline/relocate-one` to decide whether an entry needs to move and where it should go before any BOUNDARY effect fires.
Where the next ids to relocate come from.
The drain reads the head of the source collection and excludes what it already handled. It relies on relocation removing the row from the source, so the head is always fresh work — it never assumes the store returns rows in any order.
Where the next ids to relocate come from. The drain reads the head of the source collection and excludes what it already handled. It relies on relocation removing the row from the source, so the head is always fresh work — it never assumes the store returns rows in any order.
Liveness probing — Boundary stage of the resilience CPPB pipeline.
Single responsibility: ask the underlying milvus client whether it can actually reach the server, with a tiny TTL cache so a burst of probes amortises to one RPC.
Transport-agnostic: dispatches through milvus-clj.client/ILivenessProbe
(extended by every transport's defrecord). Adding a new transport
requires zero changes here — that is the point of the protocol seam.
Why a separate ns from reconnect/retry: each of probe / reconnect / retry has its own rate of change. The probe surface is the most stable (one cached read); reconnect-loop algorithm and retry budget knobs change more often. Splitting them keeps each ns small and individually testable (mock the protocol; no need to mock a loop).
Time discipline: never call System/currentTimeMillis directly — go
through hive-ttracking.clock/now-millis so tests can pin time.
Liveness probing — Boundary stage of the resilience CPPB pipeline. Single responsibility: ask the underlying milvus client whether it can actually reach the server, with a tiny TTL cache so a burst of probes amortises to one RPC. Transport-agnostic: dispatches through `milvus-clj.client/ILivenessProbe` (extended by every transport's defrecord). Adding a new transport requires zero changes here — that is the point of the protocol seam. Why a separate ns from reconnect/retry: each of probe / reconnect / retry has its own rate of change. The probe surface is the most stable (one cached read); reconnect-loop algorithm and retry budget knobs change more often. Splitting them keeps each ns small and individually testable (mock the protocol; no need to mock a loop). Time discipline: never call `System/currentTimeMillis` directly — go through `hive-ttracking.clock/now-millis` so tests can pin time.
Reconnect-loop ownership — Boundary stage of the resilience CPPB pipeline.
Single responsibility: drive the singleton milvus client back to a
reachable state after a transient failure. The loop's success
criterion is an actual probe round-trip, NOT the post-connect!
atom state — that distinction is the bug fix vs the prior design
where connect! always 'succeeded' (it just sets the atom) and the
loop exited even when the network was still down.
Decoupled from probe.clj (which owns the cache) and retry.clj
(which owns the reactive pipeline) per SRP. This ns owns ONE atom
(reconnect-state) and its loop future.
Time discipline: never call System/currentTimeMillis directly — go
through hive-ttracking.clock/now-millis so tests can pin time.
Reconnect-loop ownership — Boundary stage of the resilience CPPB pipeline. Single responsibility: drive the singleton milvus client back to a reachable state after a transient failure. The loop's success criterion is an actual probe round-trip, NOT the post-`connect!` atom state — that distinction is the bug fix vs the prior design where `connect!` always 'succeeded' (it just sets the atom) and the loop exited even when the network was still down. Decoupled from probe.clj (which owns the cache) and retry.clj (which owns the reactive pipeline) per SRP. This ns owns ONE atom (`reconnect-state`) and its loop future. Time discipline: never call `System/currentTimeMillis` directly — go through `hive-ttracking.clock/now-millis` so tests can pin time.
Reactive try / classify / retry pipeline — Pipeline stage of the resilience CPPB pipeline.
Single responsibility: wrap one RPC body in a fault-tolerant pipeline that classifies failure, kicks the reconnect loop, awaits verified recovery, and retries exactly once. Public surface:
with-auto-reconnect — function form, takes a thunkresilient — macro sugar that combines ensure-live! with
with-auto-reconnect for use inside protocol method bodiesComposes Promote (failure/classify, circuit/check) over Boundary
(probe, reconnect). Stays pure with respect to the singleton
client — never mutates default-client directly; only orchestrates
the Boundary nses that do.
Migration note: this ns replaces hive-milvus.store.health. The old
ns kept circuit, probe, reconnect, classify, retry all in one file
(5 responsibilities). This split means each concern can evolve
independently.
Reactive try / classify / retry pipeline — Pipeline stage of the
resilience CPPB pipeline.
Single responsibility: wrap one RPC body in a fault-tolerant
pipeline that classifies failure, kicks the reconnect loop, awaits
verified recovery, and retries exactly once. Public surface:
- `with-auto-reconnect` — function form, takes a thunk
- `resilient` — macro sugar that combines `ensure-live!` with
`with-auto-reconnect` for use inside protocol method bodies
Composes Promote (`failure/classify`, `circuit/check`) over Boundary
(`probe`, `reconnect`). Stays pure with respect to the singleton
client — never mutates `default-client` directly; only orchestrates
the Boundary nses that do.
Migration note: this ns replaces `hive-milvus.store.health`. The old
ns kept circuit, probe, reconnect, classify, retry all in one file
(5 responsibilities). This split means each concern can evolve
independently.Proactive periodic liveness scheduler — Boundary stage of the resilience CPPB pipeline.
Single responsibility: own a timer that fires a health check on a fixed
cadence, so a connection that dies while the store is IDLE (no traffic to
trigger the reactive retry path or the on-demand ensure-live! gate)
still heals. This closes the gap where auto-heal was 100% traffic-driven:
after a network blip with no in-flight RPC, nothing re-probed and the dead
client sat dead until the next operation happened to touch it.
Why this ns owns ONLY the timer (SRP + DRY):
retry/ensure-live!
(reads cached liveness, kicks the reconnect loop only when-not it is
already running). Re-implementing it here would duplicate the guard and
risk calling reconnect/kick! mid-heal, which drops the fresh client on
every tick. So the tick DELEGATES the policy and keeps only the
scheduling (Boundary) concern.Lifecycle: started from store.lifecycle/connect! once the config-atom
exists; stopped from store.lifecycle/disconnect! before teardown, so the
timer can't kick a heal loop right after the operator closed the store.
Time discipline: never call System/currentTimeMillis directly — go
through hive-ttracking.clock/now-millis (mirrors probe/reconnect).
Proactive periodic liveness scheduler — Boundary stage of the resilience CPPB pipeline. Single responsibility: own a timer that fires a health check on a fixed cadence, so a connection that dies while the store is IDLE (no traffic to trigger the reactive `retry` path or the on-demand `ensure-live!` gate) still heals. This closes the gap where auto-heal was 100% traffic-driven: after a network blip with no in-flight RPC, nothing re-probed and the dead client sat dead until the next operation happened to touch it. Why this ns owns ONLY the timer (SRP + DRY): - The health *decision* — 'is it dead, and if so start a heal loop WITHOUT disturbing one already in flight' — already lives in `retry/ensure-live!` (reads cached liveness, kicks the reconnect loop only `when-not` it is already running). Re-implementing it here would duplicate the guard and risk calling `reconnect/kick!` mid-heal, which drops the fresh client on every tick. So the tick DELEGATES the policy and keeps only the scheduling (Boundary) concern. - Complements, does not replace: reactive retry (on failure) + preemptive ensure-live! (at op start) stay as-is; this adds the idle heartbeat that neither covered. Lifecycle: started from `store.lifecycle/connect!` once the config-atom exists; stopped from `store.lifecycle/disconnect!` before teardown, so the timer can't kick a heal loop right after the operator closed the store. Time discipline: never call `System/currentTimeMillis` directly — go through `hive-ttracking.clock/now-millis` (mirrors probe/reconnect).
Chroma SQLite reader — pure JDBC access, no domain logic.
Chroma SQLite reader — pure JDBC access, no domain logic.
L0 contract — storage bounded context.
Two narrow protocols (ISP):
IVectorWrite — upsert one or more entries to a CollectionRef.IVectorRead — search across one or more CollectionRefs with
fan-out semantics (read may union results from
active + sunset collections during drain).Both protocols depend only on ICollectionLocator (via the ref) —
they have zero coupling to embedder or router internals. The write
pipeline (in storage/write_*.clj) is responsible for resolving
spec → ref BEFORE calling upsert!; this layer never derives a
collection from a raw type.
Reload-safety: defonce-guarded.
L0 contract — storage bounded context.
Two narrow protocols (ISP):
- `IVectorWrite` — upsert one or more entries to a `CollectionRef`.
- `IVectorRead` — search across one or more `CollectionRef`s with
fan-out semantics (read may union results from
active + sunset collections during drain).
Both protocols depend only on `ICollectionLocator` (via the ref) —
they have zero coupling to embedder or router internals. The write
pipeline (in `storage/write_*.clj`) is responsible for resolving
spec → ref BEFORE calling `upsert!`; this layer never derives a
collection from a raw type.
Reload-safety: `defonce`-guarded.Milvus implementation of IMemoryStore protocol.
Standalone addon project. Wraps milvus-clj.api into protocol methods.
Milvus requires explicit embeddings on insert and search; those are
produced through the injected hive-milvus.embed.port boundary.
DDD: Repository pattern — MilvusMemoryStore is the Milvus aggregate adapter.
This namespace is the façade over four cohesive submodules:
hive-milvus.store.schema — entry<->record, filter expressions
hive-milvus.store.index — collection loading + scalar indexes
hive-milvus.store.query — single-entry read / read-modify-write
hive-milvus.resilience.retry — reactive retry pipeline + resilient
hive-milvus.resilience.probe — cached liveness query
hive-milvus.resilience.reconnect — background heal loop, probe-verified
The old hive-milvus.store.health was a 5-responsibility monolith
(circuit, probe, reconnect-loop, retry, classify); split into the
three resilience.* nses per SRP. store.health now exists as a
deprecation shim that re-exports the new locations so external
callers don't break.
The defrecord MilvusMemoryStore + create-store live here because
the protocol methods can't be split across files.
Milvus implementation of IMemoryStore protocol. Standalone addon project. Wraps milvus-clj.api into protocol methods. Milvus requires explicit embeddings on insert and search; those are produced through the injected `hive-milvus.embed.port` boundary. DDD: Repository pattern — MilvusMemoryStore is the Milvus aggregate adapter. This namespace is the façade over four cohesive submodules: hive-milvus.store.schema — entry<->record, filter expressions hive-milvus.store.index — collection loading + scalar indexes hive-milvus.store.query — single-entry read / read-modify-write hive-milvus.resilience.retry — reactive retry pipeline + `resilient` hive-milvus.resilience.probe — cached liveness query hive-milvus.resilience.reconnect — background heal loop, probe-verified The old `hive-milvus.store.health` was a 5-responsibility monolith (circuit, probe, reconnect-loop, retry, classify); split into the three `resilience.*` nses per SRP. `store.health` now exists as a deprecation shim that re-exports the new locations so external callers don't break. The `defrecord MilvusMemoryStore` + `create-store` live here because the protocol methods can't be split across files.
Analytics protocol helpers for MilvusMemoryStore.
Analytics protocol helpers for MilvusMemoryStore.
Batch protocol helpers for MilvusMemoryStore.
Batch protocol helpers for MilvusMemoryStore.
Bounded deref for milvus-clj RPC futures. Replaces bare @(milvus/...);
on timeout throws a :milvus/timeout-tagged ex-info.
Bounded deref for milvus-clj RPC futures. Replaces bare `@(milvus/...)`; on timeout throws a `:milvus/timeout`-tagged ex-info.
Core entry CRUD, query, search, expiry, and status helpers.
Core entry CRUD, query, search, expiry, and status helpers.
DEPRECATED — kept only as a re-export shim for the public surface that
pre-resilience-split consumers (store/batch.clj, store/entries.clj,
store/lifecycle.clj, store/analytics.clj, store/staleness.clj)
already :refer from this ns.
New code MUST require directly from one of:
hive-milvus.resilience.probe — cached liveness query
hive-milvus.resilience.reconnect — heal-loop + verified reconnect
hive-milvus.resilience.retry — reactive retry + resilient macro
Why split: SRP. The old monolith owned 5 concerns (circuit, probe,
loop, retry, classify) in one 290-line file; each had its own rate
of change and test surface. The split makes adding a transport (e.g.
future WebSocket) require zero changes to the resilience layer —
only an extend-type for milvus-clj.client/ILivenessProbe.
The bug fix that motivated the split: the old loop's success
criterion was post-connect! atom state, not an actual probe
round-trip. After a network flip the atom would be set fresh while
the underlying transport was still unreachable, so the loop exited
prematurely and operators had to restart the JVM. The new
reconnect/try-reconnect-and-verify! requires both connect! AND
a successful probe! to declare recovery.
This shim re-exports the public surface so old consumers compile unchanged. Will be deleted in a future PR after consumers migrate.
DEPRECATED — kept only as a re-export shim for the public surface that pre-resilience-split consumers (`store/batch.clj`, `store/entries.clj`, `store/lifecycle.clj`, `store/analytics.clj`, `store/staleness.clj`) already `:refer` from this ns. New code MUST require directly from one of: hive-milvus.resilience.probe — cached liveness query hive-milvus.resilience.reconnect — heal-loop + verified reconnect hive-milvus.resilience.retry — reactive retry + `resilient` macro Why split: SRP. The old monolith owned 5 concerns (circuit, probe, loop, retry, classify) in one 290-line file; each had its own rate of change and test surface. The split makes adding a transport (e.g. future WebSocket) require zero changes to the resilience layer — only an `extend-type` for `milvus-clj.client/ILivenessProbe`. The bug fix that motivated the split: the old loop's success criterion was post-`connect!` atom state, not an actual probe round-trip. After a network flip the atom would be set fresh while the underlying transport was still unreachable, so the loop exited prematurely and operators had to restart the JVM. The new `reconnect/try-reconnect-and-verify!` requires both `connect!` AND a successful `probe!` to declare recovery. This shim re-exports the public surface so old consumers compile unchanged. Will be deleted in a future PR after consumers migrate.
Collection loading and scalar-index management for MilvusMemoryStore.
Owns two process-wide memoization atoms:
loaded-collections: collections this process has load-collection'd.indexed-collections: collections we've probed + installed INVERTED
scalar indexes on.Both are defonce so hot-reload doesn't desync with milvus-clj's own
singleton client state.
Collection loading and scalar-index management for MilvusMemoryStore. Owns two process-wide memoization atoms: - `loaded-collections`: collections this process has `load-collection`'d. - `indexed-collections`: collections we've probed + installed INVERTED scalar indexes on. Both are `defonce` so hot-reload doesn't desync with milvus-clj's own singleton client state.
Connection lifecycle helpers for MilvusMemoryStore.
Connection lifecycle helpers for MilvusMemoryStore.
Leaf ns for cross-collection id lookup. Extracted out of
hive-milvus.store.entries so the relocate/ pipeline can call
into it without creating a cycle (entries → pipeline → collectors
→ entries). No upward dependencies — only routing + query.
Leaf ns for cross-collection id lookup. Extracted out of `hive-milvus.store.entries` so the `relocate/` pipeline can call into it without creating a cycle (entries → pipeline → collectors → entries). No upward dependencies — only `routing` + `query`.
Single-entry read + read-merge-upsert helpers.
These are the primitives every protocol method composes with when it needs to fetch a row by id or mutate it through a read-modify-write cycle. They don't own any state — the collection name + id are passed through, and the resilient-retry wrapper is applied by the caller.
Single-entry read + read-merge-upsert helpers. These are the primitives every protocol method composes with when it needs to fetch a row by id or mutate it through a read-modify-write cycle. They don't own any state — the collection name + id are passed through, and the resilient-retry wrapper is applied by the caller.
Per-type collection routing for Milvus.
Bridges the injected embedding port's type→provider routing
(port/routing-for-type) to Milvus collection naming (underscores,
no hyphens). Each provider dimension lives in its own Milvus collection
so dual-run migrations (e.g. nomic 768-d → qwen3-embedding:0.6b 1024-d)
coexist without schema conflict — Milvus collections have immutable
dimension at create time.
Collection-name conventions:
hive_mcp_memory (legacy name preserved)hive_mcp_memory_<dim>dReads without a type opt fan out across all known collections; writes
are routed to the entry's type's collection and auto-create on first
use via index/ensure-collection!.
Per-type collection routing for Milvus. Bridges the injected embedding port's type→provider routing (`port/routing-for-type`) to Milvus collection naming (underscores, no hyphens). Each provider dimension lives in its own Milvus collection so dual-run migrations (e.g. nomic 768-d → qwen3-embedding:0.6b 1024-d) coexist without schema conflict — Milvus collections have immutable dimension at create time. Collection-name conventions: - Default 768-d collection: `hive_mcp_memory` (legacy name preserved) - Per-dim collections: `hive_mcp_memory_<dim>d` Reads without a type opt fan out across all known collections; writes are routed to the entry's type's collection and auto-create on first use via `index/ensure-collection!`.
Entry <-> Milvus record conversion and filter expression builders.
Pure functions only — no side effects, no Milvus RPCs. Extracted from hive-milvus.store so the schema/translation layer can be understood (and tested) independently of the connection + resilience machinery.
Entry <-> Milvus record conversion and filter expression builders. Pure functions only — no side effects, no Milvus RPCs. Extracted from hive-milvus.store so the schema/translation layer can be understood (and tested) independently of the connection + resilience machinery.
The effects a search performs: embedding the query, and asking a collection for its nearest rows. Both are ports; both return Result.
The effects a search performs: embedding the query, and asking a collection for its nearest rows. Both are ports; both return Result.
Merge hits from different embedding spaces. Pure.
Merge hits from different embedding spaces. Pure.
Compose a semantic search from its ports: resolve targets, embed the query into each, search each, fuse the rankings.
Compose a semantic search from its ports: resolve targets, embed the query into each, search each, fuse the rankings.
Which collections a search runs against, paired with the embedding space each is written in.
Which collections a search runs against, paired with the embedding space each is written in.
Staleness protocol helpers for MilvusMemoryStore.
Staleness protocol helpers for MilvusMemoryStore.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |