Liveness probing — Boundary stage of the resilience CPPB pipeline.
Single responsibility: ask the underlying milvus client whether it can actually reach the server, with a tiny TTL cache so a burst of probes amortises to one RPC.
Transport-agnostic: dispatches through milvus-clj.client/ILivenessProbe
(extended by every transport's defrecord). Adding a new transport
requires zero changes here — that is the point of the protocol seam.
Why a separate ns from reconnect/retry: each of probe / reconnect / retry has its own rate of change. The probe surface is the most stable (one cached read); reconnect-loop algorithm and retry budget knobs change more often. Splitting them keeps each ns small and individually testable (mock the protocol; no need to mock a loop).
Time discipline: never call System/currentTimeMillis directly — go
through hive-ttracking.clock/now-millis so tests can pin time.
Liveness probing — Boundary stage of the resilience CPPB pipeline. Single responsibility: ask the underlying milvus client whether it can actually reach the server, with a tiny TTL cache so a burst of probes amortises to one RPC. Transport-agnostic: dispatches through `milvus-clj.client/ILivenessProbe` (extended by every transport's defrecord). Adding a new transport requires zero changes here — that is the point of the protocol seam. Why a separate ns from reconnect/retry: each of probe / reconnect / retry has its own rate of change. The probe surface is the most stable (one cached read); reconnect-loop algorithm and retry budget knobs change more often. Splitting them keeps each ns small and individually testable (mock the protocol; no need to mock a loop). Time discipline: never call `System/currentTimeMillis` directly — go through `hive-ttracking.clock/now-millis` so tests can pin time.
Reconnect-loop ownership — Boundary stage of the resilience CPPB pipeline.
Single responsibility: drive the singleton milvus client back to a
reachable state after a transient failure. The loop's success
criterion is an actual probe round-trip, NOT the post-connect!
atom state — that distinction is the bug fix vs the prior design
where connect! always 'succeeded' (it just sets the atom) and the
loop exited even when the network was still down.
Decoupled from probe.clj (which owns the cache) and retry.clj
(which owns the reactive pipeline) per SRP. This ns owns ONE atom
(reconnect-state) and its loop future.
Time discipline: never call System/currentTimeMillis directly — go
through hive-ttracking.clock/now-millis so tests can pin time.
Reconnect-loop ownership — Boundary stage of the resilience CPPB pipeline. Single responsibility: drive the singleton milvus client back to a reachable state after a transient failure. The loop's success criterion is an actual probe round-trip, NOT the post-`connect!` atom state — that distinction is the bug fix vs the prior design where `connect!` always 'succeeded' (it just sets the atom) and the loop exited even when the network was still down. Decoupled from probe.clj (which owns the cache) and retry.clj (which owns the reactive pipeline) per SRP. This ns owns ONE atom (`reconnect-state`) and its loop future. Time discipline: never call `System/currentTimeMillis` directly — go through `hive-ttracking.clock/now-millis` so tests can pin time.
Reactive try / classify / retry pipeline — Pipeline stage of the resilience CPPB pipeline.
Single responsibility: wrap one RPC body in a fault-tolerant pipeline that classifies failure, kicks the reconnect loop, awaits verified recovery, and retries exactly once. Public surface:
with-auto-reconnect — function form, takes a thunkresilient — macro sugar that combines ensure-live! with
with-auto-reconnect for use inside protocol method bodiesComposes Promote (failure/classify, circuit/check) over Boundary
(probe, reconnect). Stays pure with respect to the singleton
client — never mutates default-client directly; only orchestrates
the Boundary nses that do.
Migration note: this ns replaces hive-milvus.store.health. The old
ns kept circuit, probe, reconnect, classify, retry all in one file
(5 responsibilities). This split means each concern can evolve
independently.
Reactive try / classify / retry pipeline — Pipeline stage of the
resilience CPPB pipeline.
Single responsibility: wrap one RPC body in a fault-tolerant
pipeline that classifies failure, kicks the reconnect loop, awaits
verified recovery, and retries exactly once. Public surface:
- `with-auto-reconnect` — function form, takes a thunk
- `resilient` — macro sugar that combines `ensure-live!` with
`with-auto-reconnect` for use inside protocol method bodies
Composes Promote (`failure/classify`, `circuit/check`) over Boundary
(`probe`, `reconnect`). Stays pure with respect to the singleton
client — never mutates `default-client` directly; only orchestrates
the Boundary nses that do.
Migration note: this ns replaces `hive-milvus.store.health`. The old
ns kept circuit, probe, reconnect, classify, retry all in one file
(5 responsibilities). This split means each concern can evolve
independently.Proactive periodic liveness scheduler — Boundary stage of the resilience CPPB pipeline.
Single responsibility: own a timer that fires a health check on a fixed
cadence, so a connection that dies while the store is IDLE (no traffic to
trigger the reactive retry path or the on-demand ensure-live! gate)
still heals. This closes the gap where auto-heal was 100% traffic-driven:
after a network blip with no in-flight RPC, nothing re-probed and the dead
client sat dead until the next operation happened to touch it.
Why this ns owns ONLY the timer (SRP + DRY):
retry/ensure-live!
(reads cached liveness, kicks the reconnect loop only when-not it is
already running). Re-implementing it here would duplicate the guard and
risk calling reconnect/kick! mid-heal, which drops the fresh client on
every tick. So the tick DELEGATES the policy and keeps only the
scheduling (Boundary) concern.Lifecycle: started from store.lifecycle/connect! once the config-atom
exists; stopped from store.lifecycle/disconnect! before teardown, so the
timer can't kick a heal loop right after the operator closed the store.
Time discipline: never call System/currentTimeMillis directly — go
through hive-ttracking.clock/now-millis (mirrors probe/reconnect).
Proactive periodic liveness scheduler — Boundary stage of the resilience CPPB pipeline. Single responsibility: own a timer that fires a health check on a fixed cadence, so a connection that dies while the store is IDLE (no traffic to trigger the reactive `retry` path or the on-demand `ensure-live!` gate) still heals. This closes the gap where auto-heal was 100% traffic-driven: after a network blip with no in-flight RPC, nothing re-probed and the dead client sat dead until the next operation happened to touch it. Why this ns owns ONLY the timer (SRP + DRY): - The health *decision* — 'is it dead, and if so start a heal loop WITHOUT disturbing one already in flight' — already lives in `retry/ensure-live!` (reads cached liveness, kicks the reconnect loop only `when-not` it is already running). Re-implementing it here would duplicate the guard and risk calling `reconnect/kick!` mid-heal, which drops the fresh client on every tick. So the tick DELEGATES the policy and keeps only the scheduling (Boundary) concern. - Complements, does not replace: reactive retry (on failure) + preemptive ensure-live! (at op start) stay as-is; this adds the idle heartbeat that neither covered. Lifecycle: started from `store.lifecycle/connect!` once the config-atom exists; stopped from `store.lifecycle/disconnect!` before teardown, so the timer can't kick a heal loop right after the operator closed the store. Time discipline: never call `System/currentTimeMillis` directly — go through `hive-ttracking.clock/now-millis` (mirrors probe/reconnect).
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |