hive-milvus.resilience.retry — io.github.hive-agi/hive-milvus 0.4.4

Reactive try / classify / retry pipeline — Pipeline stage of the resilience CPPB pipeline.

Single responsibility: wrap one RPC body in a fault-tolerant pipeline that classifies failure, kicks the reconnect loop, awaits verified recovery, and retries exactly once. Public surface:

• with-auto-reconnect — function form, takes a thunk
• resilient — macro sugar that combines ensure-live! with with-auto-reconnect for use inside protocol method bodies

Composes Promote (failure/classify, circuit/check) over Boundary (probe, reconnect). Stays pure with respect to the singleton client — never mutates default-client directly; only orchestrates the Boundary nses that do.

Migration note: this ns replaces hive-milvus.store.health. The old ns kept circuit, probe, reconnect, classify, retry all in one file (5 responsibilities). This split means each concern can evolve independently.

attempt-call clj

(attempt-call f)

Run f under a try-effect that tags any Throwable as a raw :milvus/call error carrying the original exception. No classification yet — classification happens in r/map-err so the pipeline stays linear.

Uses bare try/catch (not try-effect*) because the carried payload is the Throwable itself, not just its message — classify-err downstream needs the live exception to dispatch on transport class.

classify-err clj

(classify-err err)

Promote stage — pure. Lift a raw {:throwable e} error into a MilvusFailure ADT. Re-throws non-connection failures so callers outside the resilient retry path keep seeing exceptions for genuine programming errors (the original contract of with-auto-reconnect).

ensure-live! clj

(ensure-live! config-atom)

Preemptive liveness gate. Called at the top of every resilient body.

Reads probe/alive? (cached). If the cache says dead AND the reconnect loop isn't already running, kicks the loop preemptively so healing starts before the caller's RPC fails.

Does NOT block waiting for recovery — with-auto-reconnect still owns the reactive retry budget. This just shrinks the window between 'connection died' and 'reconnect loop running'.

Returns true if probe says alive at the end of the call, false if still dead (the caller's RPC will then hit the reactive path).

resilient clj macro

(resilient config-atom & body)

Wrap body in with-auto-reconnect: on transient gRPC/HTTP/connection failure (UNAVAILABLE, DEADLINE_EXCEEDED, IO timeout, "Keepalive failed", etc.) the background reconnect loop is kicked, we block briefly for it to verify a fresh client via probe round-trip, and body is re-executed once. Transparent to callers — they either see the successful retry result or a graceful {:success? false :reconnecting? true} map after the budget is spent.

Before executing body, calls ensure-live! to proactively detect a dead client and start the reconnect loop early. The reactive with-auto-reconnect path still handles any failure that slips through.

Intended for use inside MilvusMemoryStore protocol method bodies where config-atom is captured from the defrecord.

retry-once clj

(retry-once config-atom f budget-ms)

Pipeline stage — composes Promote (classify) over Boundary (reconnect/kick!, reconnect/await!).

On a transient failure: kick the background heal loop, block up to budget-ms for verified recovery (probe round-trip), then retry f one time. On recovery failure or second-attempt failure we carry the failure forward through the Result chain. Never throws for transient paths — fatal exceptions already unwound in classify-err.

with-auto-reconnect clj

(with-auto-reconnect config-atom f)

(with-auto-reconnect config-atom f budget-ms)

Execute f with transparent reconnect-and-retry on transient connection failures. Flat hive-dsl.result pipeline:

attempt -> classify -> (maybe) retry -> legacy-shape

Semantics:

• Circuit breaker gated first: if :open and in cooldown we return the breaker's fail map without running f.
• Fatal exceptions (non-transient) are re-thrown — callers outside this helper keep their existing contract.
• Transient failures kick the background reconnect loop, block up to budget-ms for VERIFIED recovery (probe round-trip), then retry f exactly once.
• Final failure (second attempt, or reconnect timeout) is translated back to the legacy {:success? false :errors [...] :reconnecting? ...} map so protocol callers continue to see the shape they expect.

Why inline reactive retry: tailscale userspace netstack (and any NAT-style intermediary) can drop an idle gRPC flow between RPCs. Returning the raw failure to the caller forces every caller to implement retry, which they don't.